Hadoop Balancer HBase Balancer

Hadoop Equalizer

While Hadoop is running, its datanode blocks become increasingly unbalanced, and unbalanced clusters can cause some datanode to be relatively busy.

The Hadoop equalizer is a daemon. It re-allocates blocks, moving blocks from busy datanode to relatively idle datanode. At the same time, adhere to the replica policy, the replicas scattered to different racks, to reduce the data corruption rate.

Cluster equalization Criteria: the utilization rate of each datanode is very close to the usage of the cluster, and the gap does not exceed the given threshold.

Datanode usage: The ratio between the amount of space used on the node and the total space;

Cluster usage: The ratio of the space used in the cluster to the total amount of space in the cluster.

In order to reduce the cluster load and avoid interfering with other users, the equalizer runs in the background and limits the bandwidth of the replicated data between the different nodes, the default value is 1mb/s.

The Hadoop equalizer is turned off by default, the boot needs to execute the%start-balancer.sh, and the cluster equalization will stop automatically.

HBase Load Balancing

The EQ equalizer for HBase is executed five minutes by default, set by the Hbase.balancer.period property. After the equalizer runs, it attempts to evenly distribute the region to all region servers.

First identify a region allocation plan, and then start moving region.

The switch class of the equalizer controls the on and off state.

The equalizer for HBase is turned on by default and runs periodically.

Here is an explanation of the official website:

Hadoop Balancer

Over time, the distribution of blocks across datanodes can become unbalanced. An unbalanced cluster can affect locality for MapReduce, and it puts a greater strain on the highly utilized datanodes It ' s best avoided.

The Balancer program is a Hadoop daemon this redistributes blocks by moving them from overutilized datanodes to u Nderutilized Datanodes, while adhering to the block replica placement policy This makes data loss unlikely by placing bloc K Replicas on different racks ( see Replica Placement). It moves blocks until the cluster is deemed to being balanced, which means that the utilization of every datanode (ratio of U Sed space on the "node to total capacity of the node" differs from the utilization of the cluster (ratio of used space on t He cluster to total capacity of the cluster) by no more than a given threshold percentage. You can start the balancer with:

% start-balancer.sh

the -threshold argument Specifies the threshold percentage that defines what it means for the cluster to be balanced. The flag is optional; If omitted, the threshold is 10%. At any one time, only one balancer is running on the cluster.

The balancer runs until the cluster is balanced, it cannot move any more blocks, or it loses contact with the namenode. It produces a logfile in the standard log directory, where it writes a line for every iteration of redistribution Carries out. The output from a short run on a small cluster (slightly reformatted to fit the page):

time stamp      iteration #  bytes already moved  , ..... LEFT TO MOVE &NBSP, ..... Being moved 
 mar 18, 2009 5:23:42 pm  0                  0 KB         219.21 MB       150.29 MB 
 Mar 18,  2009 5:27:14 PM  1             195.24 MB         22.45 MB        150.29 MB 
 The cluster is balanced. exiting ... 
 Balancing took 6.072933333333333 minutes 
  

The balancer is designed to run in the background without unduly taxing the cluster or interfering with other clients Using the cluster. It limits the bandwidth that it uses to copy a block from one node to another. The default is a modest 1 Mb/s, but that can be changed dfs.datanode.balance.bandwidthPerSec  by setting the property in Hdfs-site.xml, S Pecified in bytes.

HBase Load Balancing

the Master has a built-in feature, called the  balancer . By default, the balancer runs every five minutes, and it was configured by the hbase.balancer.period  property. Once The balancer is started, it'll attempt to equal out the number of assigned regions per region server so that they a Re within one region of the average number per server. The call first determines a new  assignment Plan , which describes which regions should is moved where. Then it starts the process of moving the regions by calling the unassign () method of the administrative API iteratively.

The balancer has a upper limit on how long it's allowed to run, which is configured using the hbase.balancer.max.balancing Property Anddefaults to half of the balancer period value, or both and a half minutes.

You can control the balancer by means of the balancer switch: either use the shell ' s balance_switch command to toggle the Balancer status between enabled and disabled, or use the API method to do the balanceSwitch() same. W Hen you disable the balancer, it no longer runs as expected.

The balancer can be explicitly started using the shell ' s Balancer command, or using thebalancer()API method. The time-controlled invocation mentioned previously calls this method implicitly. It would determine if there is any work to being done and returntrueIf that's the case . The return value offalsemeans that it is not able to run the balancer, because either it is switched off, there is no work to is done (all is balanced), or something else was prohibiting the process. One example in transition list ( seeMain Page): If there is a region currently in transition, the balancer would be skipped.

Instead of relying on the balancer to do it work properly, you can use the move command and API method to assign Regions to other servers. This is useful if you want to control where the regions of a particular table is assigned. See region hotspotting for an example.

Hadoop Balancer HBase Balancer