Avoid regionServer downtime and regionserver downtime

Avoid regionServer downtime and regionserver downtime

Because regionserver management information is mainly recorded in zookeeper, regionserver's downtime judgment is based on session expired. OK

So what are the causes for the session expired of regionserver and Zookeeper?

1. Poor network. 2. Java full GC, which blocks all threads. If the time is long, the session expired will also occur.Solution: 1. Extend the timeout time of Zookeeper. 2. Set "hbase. regionserver. restart. on. zk. expire" to true.In this case, when ZooKeeper session expired is encountered, regionserver will select restart instead of abort. The specific configuration is to add <property> <name> zookeeper in the hbase-site.xml. session. timeout </name> <value> 90000 </value> <description> ZooKeeper session timeout. HBase passes this to the zk quorum as suggested maximum time for asession. see your client sends a requested timeout, the server responds with thetimeout that it can give the client. the current implementationrequires that the timeout be a minimum of 2 times the tickTime (as set in the server configuration) and a maximum of 20 timesthe tickTime. "Set the zk ticktime with hbase. zookeeper. property. tickTime. in milliseconds. </description> </property> <name> hbase. regionserver. restart. on. zk. expire </name> <value> true </value> <description> Zookeeper session expired will force regionserver exit. enable this will make the regionserver restart. </description> </property> 3. To avoid the impact of java full GC suspend thread on Zookeeper heartbeat, we also need to configure the hbase-env.sh. Set the jvm memory collection algorithm:

     -XX:+CMSParallelRemarkEnabled。

As follows:

export HBASE_OPTS="-Xms16g -Xmx16g -Xmn2g -Xss200k -XX:MaxNewSize=2g -XX:SurvivorRatio=2 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC  -XX:+CMSParallelRemarkEnabled   -XX:+UseFastAccessorMethods  -XX:+UseParNewGC -XX:MaxPermSize=300m -XX:MaxTenuringThreshold=5  -XX:GCTimeRatio=19 -XX:ParallelGCThreads=10 -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:-UseGCOverheadLimit "

Finally, start regionserver:Command:Hbase-daemon.sh start regionserver

Enable balanceCommand:Balance_switch true