Why hregionserver is not started when hbase is started

When we configured hbase today, we found that the regionserver of several machines could not start. The following error is reported:

 

At sun. Reflect. nativeconstructoraccessorimpl. newinstance0 (native method)
At sun. Reflect. nativeconstruct%cessorimpl. newinstance (nativeconstruct%cessorimpl. Java: 39)
At sun. Reflect. delegatingconstruct%cessorimpl. newinstance (delegatingconstruct%cessorimpl. Java: 27)
At java. Lang. Reflect. constructor. newinstance (constructor. Java: 513)
At org. Apache. hadoop. IPC. RemoteException. instantiateexception (RemoteException. Java: 95)
At org. Apache. hadoop. IPC. RemoteException. unwrapremoteexception (RemoteException. Java: 79)
At org. Apache. hadoop. hbase. regionserver. hregionserver. reportforduty (hregionserver. Java: 1506)
At org. Apache. hadoop. hbase. regionserver. hregionserver. tryreportforduty (hregionserver. Java: 1470)
At org. Apache. hadoop. hbase. regionserver. hregionserver. Run (hregionserver. Java: 563)
At java. Lang. thread. Run (thread. Java: 662)
Caused by: Org. apache. hadoop. IPC. remoteException: Org. apache. hadoop. hbase. clockoutofsyncexception: Server S3, 60020,1301097875246 has been rejected; reported time is too far out of sync with master. time Difference of 41450 MS> MAX allowed of 30000 ms
At org. Apache. hadoop. hbase. master. servermanager. checkclockskew (servermanager. Java: 181)
At org. Apache. hadoop. hbase. master. servermanager. regionserverstartup (servermanager. Java: 129)
At org. Apache. hadoop. hbase. master. hmaster. regionserverstartup (hmaster. Java: 613)
At sun. Reflect. nativemethodaccessorimpl. invoke0 (native method)
At sun. Reflect. nativemethodaccessorimpl. Invoke (nativemethodaccessorimpl. Java: 39)
At sun. Reflect. delegatingmethodaccessorimpl. Invoke (delegatingmethodaccessorimpl. Java: 25)
At java. Lang. Reflect. method. Invoke (method. Java: 597)
At org. Apache. hadoop. hbase. IPC. hbaserpc $ server. Call (hbaserpc. Java: 570)
At org. Apache. hadoop. hbase. IPC. hbaseserver $ handler. Run (hbaseserver. Java: 1039)

At org. Apache. hadoop. hbase. IPC. hbaseclient. Call (hbaseclient. Java: 771)
At org. Apache. hadoop. hbase. IPC. hbaserpc $ invoker. Invoke (hbaserpc. Java: 257)
At $ proxy5.regionserverstartup (unknown source)

At org. Apache. hadoop. hbase. regionserver. hregionserver. reportforduty (hregionserver. Java: 1502)

 

The error indicates that the time difference between the node time and the master time is greater than 30000 ms, that is, the service cannot be started in 30 seconds. Hadoop has never encountered this problem before, so hbase has the problem of system time synchronization, and the error must be less than 30 s.

 

I checked it on the Internet. to configure Server time synchronization, NTP is required. It is best to connect to the time server on the Internet. It is troublesome to configure the Intranet, so you can manually change the time and start it again.

 

Another solution is to modify the maximum time error of hbase.

Add configuration hbase. master. maxclockskew 180000 time difference of regionserver from Master in hbase-site.xml