Deep anatomy of HBase load balancing and performance metrics

Depth Analysis HBase load balancing and performance metrics

in distributed systems, load balancing is a very important function, HBase uses the number of region to load balance, that is, through hbase.master.loadbalancer.class implementation of a custom load balancing algorithm. Below you will analyze HBase load balancer's related content and performance metrics.

content

The HBase system load balancer is a cyclical operation that distributes the region evenly across the regionserver by load balancing, using the Hbase.balancer.period property to control the time interval for load balancing, which defaults to 5 minutes. Triggering a load-balancing operation is conditional, but does not trigger a load-balancing operation if the following conditions occur:

L Load Balancer automatic operation balance_switch off, i.e.: Balance_switch false;

L HBase Master node is initializing operation;

L The RIT is being executed in the HBase cluster, that is, region is migrating;

L hbase cluster is processing off-line regionserver;

Load Balancing Algorithm

When HBase performs load balancing operations, how to determine whether the number of region on each regionserver node is balanced, here is the following steps to determine:

L Calculate the interval range of the equilibrium value, through the total region number and the number of Regionserver nodes, calculate the average number of regions, and then calculate the minimum and maximum value;

L Traverse The Regionserver node that exceeds the region's maximum value, and migrate the value on that node until the region of the node is less than or equal to the maximum value.

L Traverse The Regionserver node below the region minimum, allocating region of the cluster to these regionserver until the region is greater than or equal to the minimum value;

L responsible for the above operation, until the number of the region on all regionserver in the cluster is between the minimum and maximum, the cluster will arrive at the load balancer, then, even if the balanced command is executed manually again, the hbase underlying logic will perform the ignore operation.

algorithm Flow Instance

the following author through the actual application scenario to give you an analysis The implementation process of the HBase load balancing algorithm. For example, if we currently have a 5-node HBase cluster with Master and Regionserver, of which 2 master and 3 Regionserver are composed, the number of region on each regionserver:

Figure 1

before performing a load balancing operation, first calculate the total Region number, the total number of region in the cluster in the current instance is 175+56+99=330, and then calculates the region mean value that each regionserver needs to accommodate, and calculates the result:

Average (110) = Total region number (Regionservers)/Total (3)

Calculate the minimum and maximum values to determine Does the HBase cluster require load balancing operations to calculate the formula:

# hbase.regions.slop weight value, default is 0.2
Minimum Value = Math.floor (average * (1-0.2))
Maximum Value = Math.ceil (average * (1+0.2))

HBase cluster If you determine that the minimum region number in each regionserver is greater than the calculated minimum, and that the maximum region number is less than the maximum, this is a direct return that does not trigger a load balancer operation. Based on the region number given in the example, the minimum value of region is calculated to be 88 and the maximum region is 120.

because the instance The region number of RegionServer2 is 56, less than the minimum region number 88, and the region number of RegionServer1 is 175, which is greater than the maximum region number of 120, so load balancing operation is required.

The HBase system provides administrator commands to operate load balancing, with the following commands:

# Use the HBase shell command to enter the HBase console and turn on automatic load balancing
HBase (Main):001:0> Balance_switch true

Balance_switch command The underlying implementation of BALANCE_SWITCH.RB and admin.rb File source code:

Figure 2

This command output is the previous load Balancer balancer switch settings, and then look at the balance_switch command processing to implement the source code:

Figure 3

at this HBase Load Balancer Automatic operation is turned on, but what if we need to balance the cluster's region number immediately? Here HBase also provides administrative commands, implemented by the Balancer command, which operate the command:

HBase (main):001:0> balancer

Balancer Command Implementation View BALANCER.RB and admin.rb file Source:

Figure 4

Figure 5

This command generates a load balancer plan to perform the load balancing operation of the cluster by invoking the Balancecluster () method of the Load Balancer balancer, and the master implements load balancing the underlying source code:

Figure 6-1

Figure 6-2

However, each time a manual execution, the number of the balance does not necessarily meet the requirements, then we can encapsulate the command, with a script execution, implementation code:

Figure 7

This script executes 20 times by default and can customize the number of executions by entering an integer parameter.

when When the HBase cluster checks all the Regionserver on the region, the load balancing operation for the cluster is completed. If the requirement is not met, you can execute the above script again until all the region numbers are between the minimum and maximum values. When all regionserver in the HBase cluster have completed load balancing, the number of region on each regionserver in the instance is distributed:

Figure 8

at this time each The number of region on the Regionserver node is within the minimum and maximum range, and the area on each regionserver node of the HBase cluster is processed in a balanced state.

Performance Index

The HBase system has a very important performance indicator, which is the delay of the cluster processing request. The HBase system provides a tool class for reacting to the time spent in the cluster processing requests:

Org.apache.hadoop.hbase.tool.Canary

This type of primary user checks The time-consuming state of the hbase system. If you do not know how to use the Help command to view the specific usage, manipulate the command:

HBase Org.apache.hadoop.hbase.tool.canary-help

(1) View the time-consuming situation for each region in each table in the cluster

HBase org.apache.hadoop.hbase.tool.Canary

(2) viewing The time-consuming situation of each region in the money table, using space splitting between multiple tables

# View Money tables and person tables
HBase org.apache.hadoop.hbase.tool.Canary Money Person

(3) View The time-consuming situation of each regionserver

HBase Org.apache.hadoop.hbase.tool.canary-regionserver DN1

usually we pay more attention to each of the Regionserver the time-consuming situation of the node, encapsulate the command, and then print the time-consuming situation of each regionserver in the cluster, the script implementation:

Figure 9

Summarize

Maintenance HBase clusters, such as restarting a certain number of regionserver nodes, may send a region imbalance, if you turn on automatic equalization, you need to immediately make the current cluster on the other regionserver on the region in a balanced state, Then you can use the manual equalization operation. In addition, the time-consuming situation of each regionserver in HBase cluster can reflect the health status of the current cluster.

Deep anatomy of HBase load balancing and performance metrics