131th: Hadoop Cluster Management tool Equalizer Balancer practical Detailed learning notes

Section 131 :Hadoop Cluster management tool equalizer Balancer The actual combat detailed study notes

Why do I need a equalizer?

As the cluster runs, the block on each data storage node in HDFs may be distributed more and more unevenly, resulting in reduced MapReduce locality when running the job .

One of the essence of distributed computing: data does not move code. Reducing the impact of local performance is a cause, and does not make full use of the resources of the cluster, because the task is calculated to focus on the partial Datanode , which is more prone to failure.

balancer block from busy datanode move to busy datanode Span style= "font-family: the song Body;" >, reassign block and in the production environment there will be Span style= "font-family:arial;" >rack balancer will adhere to repalication placement policy, Span style= "font-family:arial;" >block distributed to different racks, reducing the likelihood of data corruption.

Balancer itself constantly moves blocks until it reaches equilibrium, what is equilibrium? The usage of each datanode (percentage of space and space capacity used by the current node), and cluster utilization (percentage of the cluster's used space and cluster space capacity), If the node usage is close to the cluster usage (not exceeding a certain threshold), it is considered to be balanced. How to set this threshold:threshold parameter.

The default threshold value is 10%.

Because balancer consumes resources while working, there is only one balancer in the cluster when working .

Balancer creates a log record of each block redistribution process in the standard log . To reduce cluster load,balancer is designed to run in the background, and the bandwidth to replicate data between different nodes is also limited, and the default is 1mb/s(configurable)

Start-balancer.sh can be run.

Balancer is very important in the production environment.

above content is Liaoliang teacher Dt hadoop The first-class combat classic "section 131 Talk about the study notes.
Liaoliang: " flink docker , android technical Chinese evangelist. spark Dean and chief expert of Asia Pacific Research Institute, dt android Span style= "font-family: the song Body;" > Soft and hard integrated source-level experts, English pronunciation magician, fitness enthusiasts.

Public account:Dt_spark

Contact email [email protected]

Tel:18610086859

qq:1740415547

Number:18610086859

Sina Weibo:ilovepains

Liaoliang's first Chinese Dream: Free for the whole society to cultivate thousands of outstanding big Data practitioners!

Can be donated by Liaoliang Teacher's number 18610086859 to send red envelopes, now released Liaoliang free video complete the following:

1, "Big Data sleepless Night:Spark kernel decryption (total )":http://pan.baidu.com/s/1eQsHZAq

2, "Hadoop in- depth Combat classic" Http://pan.baidu.com/s/1mgpfRPu

3 spark Pure combat Public Welfare Forum "   http://pan.baidu.com/s/1jGpNGwu 
4 Span style= "font-family: the song Body;" >, " scala The classic of the practical,"   http://pan.baidu.com/s/1sjDWG25 
5 docker   http ://pan.baidu.com/s/1ktpl8uf 
6 spark Asia Pacific Research Institute spark   http://pan.baidu.com/s/1i30Ewsd 

7,Spark Combat Master Road All six stages video:http://edu.51cto.com/pack/view/id-144.html

8, "Big Data Spark Enterprise-level combat" purchase http://item.jd.com/11622851.html

Section 131 video website address:

51CTO

Http://edu.51cto.com/lesson/id-78585.html

131th: Hadoop Cluster Management tool Equalizer Balancer practical Detailed learning notes