Hadoop cluster balance tools in a detailed

Hadoop's balance tools are typically used to balance the file block distribution in each datanode in a Hadoop cluster while on-line Hadoop cluster operations. To avoid the problem of a high percentage of datanode disk usage (which is also likely to cause the node to have higher CPU utilization than other servers).

1) usage of the Hadoop balance tool:
[HTML] view plain copy to start:bin/start-balancer.sh [-threshold <threshold>] Example:bin/start-bal ancer.sh start the balancer with a default threshold of 10% Bin/start-balancer.sh-threshold 5 start the balancer W ith a threshold of 5% to stop:bin/stop-balancer.sh 2) several parameters that affect the Hadoop balance tool:

-threshold default settings: 10, Parameter value range: 0-100, parameter meaning: To determine whether the cluster is balanced by the target parameters, each Datanode storage utilization and total cluster storage utilization of the difference should be less than this threshold, theoretically, the smaller the parameter setting, the more balanced the whole cluster, However, in the online environment, when the Hadoop cluster is balance, the data is written and deleted concurrently, so it is possible that the set balance parameter value cannot be reached.

Dfs.balance.bandwidthPerSec default setting: 1048576 (1 m/s), parameter meaning: Set the bandwidth that the balance tool can occupy in the run, the excessive setting may cause the mapred to run slowly

3) Other features of the Hadoop balance tool:

Balance tool during the run, the iteration moves the file block from the high-utilization datanode to the low-usage datanode, and the amount of data moved in each iteration does not exceed the lesser of the following two values: 10G or specify threshold * capacity, and no more than 20 minutes per iteration. At the end of each iteration, the Balance tool updates the file block distribution of the Datanode. The following is a description of the official documentation in English:
[Python]View plain Copy the tool moves blocks from highly utilized datanodes to poorly utilized datanodes iteratively. In each iteration a datanode moves or receives no further than the lesser of 10G bytes or the threshold fraction of its cap Acity. Each iteration runs no more than minutes. At the end of each iteration, the balancer obtains updated datanodes information from the Namenode.