How to solve the problem of region Server compact process consuming large amount of network export bandwidth in HBase

After HBase version 0.92, the Region server's compact process is divided into small compaction and large compaction two, depending on the size of the file to be merged, This may result in the compact consuming too much network export bandwidth when the cluster writes large amounts. This article will describe in detail the process of troubleshooting and the solution of this problem during cluster usage.

1. Finding problems

HBase cluster (version 0.94.0) running process, found that 5 region server network export bandwidth is often maintained above 100mb/s, close to the limit of the network card, and region server's machine load load is also high, peak time to reach the 30~ 50.

2. Troubleshooting Issues

1, the actual operation of the cluster, observed the region Server service side of the network card, the average per write flow of about 60mb/s (at this time the written volume is already very large); read out the flow of 90mb/s, sometimes even break 100mb/s (note: Each machine is gigabit NIC);

2, observe the actual amount of written data in 5w per second TPs around, a single record average size of 1KB, will probably occupy 50mb/s around the network card entrance bandwidth request, and observed the phenomenon is consistent;

3, the observation of the number of 6w QPS per second or so, a single record average size of 1KB, will probably occupy 60mb/s about the network card export bandwidth requests, strange is actually observed to have close to or even more than 100MB/S network export bandwidth requests, more than the 40mb/s of the network export bandwidth;

4, after the analysis and troubleshooting to determine the cause of the above process, may be the HBase service end due to the large number of writes to trigger compaction process, and compaction is required to read the HBase data, so occupy a considerable part of the network export bandwidth;

More Wonderful content: http://www.bianceng.cnhttp://www.bianceng.cn/database/extra/

5, combined with the relevant source org/apache/hadoop/hbase/regionserver/ Compactsplitthread.java analysis, decided to make changes to the HBase cluster configuration (see the next section), the main purpose is to reduce the occurrence of compaction;

6, the next, observed that region server network utilization significantly reduced, the general import and export bandwidth can be maintained under 70MB/S.

3. Problem solving

HBase 0.92 has been added to the configuration options on the compact, the compact is divided into small compaction and large compaction two thread pool execution (by default, each has 1 threads, the specific source code see: org/apache/ hadoop/hbase/regionserver/ Compactsplitthread.java), because the compact process needs to read data from the HBase cluster, the actual operation caused the compact to occupy a large amount of network export traffic, the solution for selective shutdown small Compaction or large compaction. The following two modes of change can be resolved:

1) Programme I.

(1) Modify hbase.regionserver.thread.compaction.throttle for a very large value (such as 50GB), forcing all compact to become small compaction, reduce the compact pressure;

(2) The number of small compaction and large compaction threads is set to 1, reducing the pressure of the compact (not configured, the system will initialize it by default to 1).

Operation Steps:

Prepare Hbase-site.xml files, add or modify the following options:

<property>
    <name>hbase.regionserver.thread.compaction.throttle</name>
    <value> 53687091200</value>
</property>
<property>
    <name> hbase.regionserver.thread.compaction.small</name>
    <value>1</value>
</property>
<property>
    <name>hbase.regionserver.thread.compaction.large</name>
    <value >1</value>
</property>

Restart the cluster for the configuration to take effect.

2) Programme II

The number of small compaction threads is set to 0, which closes the small compaction, leaving only large compaction, which reduces the pressure on the compact.

Operation Steps:

Prepare Hbase-site.xml files, add or modify the following options:

<property>
    <name>hbase.regionserver.thread.compaction.small</name>
    <value>0 </value>
</property>

Restart the cluster for the configuration to take effect.

Author: great Circle those things

URL: http://www.cnblogs.com/panfeng412/archive/2012/10/17/ How-to-resolve-hbase-region-server-compaction-taking-up-too-much-network-bandwidth.html