At intervals, hbase reads will pause for 10 s and solve the problem. hbase10s

At intervals, hbase reads will pause for 10 s and solve the problem. hbase10s

Cause: some time ago, because the heapsize of the region server was set to 16 GB, the block cache size was changed to 16 GB * 0.4 = 3.2 GB, check the log and the following logs will appear in the jvm at intervals:

2015-03-24 16:09:27,405 WARN org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 15959msGC pool 'ParNew' had collection(s): count=1 time=213msGC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=16158ms--2015-03-24 16:28:42,048 WARN org.apache.hadoop.hbase.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 20125msGC pool 'ParNew' had collection(s): count=1 time=337msGC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=20206ms 

It can be concluded that the block cache has a cms gc, up to 10 s, or even 20 s,

Currently, BlockCache uses a HashMap to maintain the Block ing between the Block Key and the Block. The strict LRU algorithm is used to remove the Block. The capacity is specified during initialization, when the usage reaches 85%, the proportion of block to 75% is eliminated.
Advantage: The HashMap provided by jvm is used directly to manage the Cache, which is simple and reliable. The JVM will help you reclaim the memory occupied by the obsolete BlOCK.
Disadvantages:
1. When a Block is cached to be eliminated, it is basically promoted from the New area to the Old area along with the location in the Heap.
2. After the blocks promoted to the Old area are eliminated, the CMS eventually recycles the garbage, resulting in Heap fragments. The old area becomes larger, leading to a long cms time.
3. due to the fragmentation problem, the FullGC fails to be promoted during GC. Our online system has a frequency of FullGC for one day based on different business characteristics, for one week, there are both in the first half of January. For high frequencies, FullGC can be manually triggered in the middle of the night in O & M.
4. If the cache speed is faster than the elimination speed, unfortunately, the current Code has the OOM risk (this can be avoided by modifying the Code)

Solution:
Hbase-0.94 later provided BucketCache, on the one hand reduce region server heap size to 8 gb, reduce the probability of cms occurrence and reduce time, on the other hand, you can use BucketCache to Increase the level 2 cache. The off heap cache or ssd cache can be used for Level 2 caching.
The configuration of CDH5.3.0 is as follows: add
XX: MaxDirectMemorySize = 16G

Configure the BucketCache size in region server hbase-site:

<property>  <name>hbase.bucketcache.ioengine</name>  <value>offheap</value></property><property>  <name>hbase.bucketcache.percentage.in.combinedcache</name>  <value>0.8</value></property><property>  <name>hbase.bucketcache.size</name>  <value>16384</value></property>

The read cache is increased: the number of cached blocks is doubled, and the total hit rate is increased by more than 60%.

Reference:
Http://www.slideshare.net/bijugs/h-base-performance
Http://zh.hortonworks.com/blog/hbase-blockcache-101/
Http://zjushch.iteye.com/blog/1751387
Http://blog.csdn.net/bluishglc/article/details/21516067
Http://punishzhou.iteye.com/blog/1277141