HBase solves the problem that RegionServerCompact occupies a large amount of outbound bandwidth.

HBase solves the problem that RegionServerCompact occupies a large amount of network egress bandwidth. author: those in the big circle | the article can be reproduced. please mark the original source and author information in the form of hyperlinks. URL: recommended: after HBase 0.92, the RegionServer Compact process is divided into smal based on the file size to be merged.

HBase solves the problem that the Region Server Compact process occupies a large amount of network egress bandwidth. author: those in the grand circle | the article can be reproduced. please mark the original source and author information in the form of hyperlinks. URL: recommended: after HBase 0.92, the Compact process of the Region Server is divided into smal based on the file size to be merged.

HBase solves the problem that the Region Server Compact process occupies a large amount of outbound bandwidth.

Author: those things in the big circle | the article can be reproduced. please mark the original source and author information in the form of hyperlinks

URL:

Recommendation: HBase authoritative guide

After HBase 0.92, the Compact process of the Region Server is divided into small compaction and large compaction based on the file size to be merged, as a result, Compact occupies an excessive amount of network egress bandwidth when the cluster has a large write volume. This article describes in detail the troubleshooting process and solutions for this problem during cluster usage.

1. Problems found

When the HBase cluster (version 0.94.0) is running, it is found that the network egress bandwidth of five Region servers is usually above 100 MB/s. the Hong Kong VM is close to the network adapter limit; at the same time, the load on the Region Server is also high, and the load can reach 30 ~ during peak hours ~ 50.

2. troubleshoot

1. during the actual operation of the cluster, the NIC of the Region Server is observed. The average write traffic per Server is about 60 MB/s (the write traffic is already large). The read traffic is 90 MB/s, sometimes it even exceeded 100 MB/s (note: Each machine is a gigabit Nic );

2. observe that the actual write data volume is about 5 million tps per second. The average size of a single record is 1 KB, which may take about 50 MB/s of Nic entry bandwidth requests, it is consistent with the observed phenomenon;

3. check that the query volume is about 6 million qps per second. The average size of a single record is 1 KB, which may take about 60 MB/s of the network card egress bandwidth requests, it is strange that the number of outgoing bandwidth requests close to or even exceeds 100 MB/s is observed, and the outgoing bandwidth is about 40 MB/s;

4. after analysis and troubleshooting, determine the cause of the above process. it may be that the HBase server triggers the compaction process frequently due to the large write volume, and compaction needs to read HBase data, therefore, it occupies a considerable amount of outbound network bandwidth;

5. Combine the relevant source code org/apache/hadoop/hbase/regionserver/CompactSplitThread. java Analysis decides to make changes to the HBase cluster configuration (see the next section for details). The main purpose is to reduce the occurrence of compaction;

6. Next, we can see that the network utilization of the Region Server is significantly reduced, and the import and export bandwidth can be maintained below 70 MB/s.

3. solve the problem

After HBase 0.92, the compact configuration option is added. compact is divided into two thread pools: small compaction and large compaction for execution. (by default, each thread has its own virtual host. For more information about the source code, see: org/apache/hadoop/hbase/regionserver/CompactSplitThread. java). because the compact process needs to read data from the HBase cluster, compact occupies a large amount of outbound network traffic during actual operation. the solution is to disable small compaction or large compaction selectively. There are two possible change methods:

1) solution 1

(1) modify hbase. regionserver. thread. compaction. throttle is a large value (such as 50 GB). The Hong Kong Virtual host forcibly changes all compact into small compaction, reducing the compact pressure;

(2) set the number of small compaction and large compaction threads to 1 to reduce the compact pressure (if not configured, the system will initialize it to 1 by default ).

Procedure:

Prepare the hbase-site.xml file and add or modify the following options:

Hbase. regionserver. thread. compaction. throttle53687091200hbase. regionserver. thread. compaction. small1hbase. regionserver. thread. compaction. large1

Restart the cluster to make the configuration take effect.

2) solution 2

Set the number of small compaction threads to 0 to disable small compaction. only large compaction is left, which can reduce the compact pressure.

Procedure:

Prepare the hbase-site.xml file and add or modify the following options:

Hbase. regionserver. thread. compaction. small0

Restart the cluster to make the configuration take effect.

Posted on