Add and delete nodes in Hadoop

Add, delete, and delete nodes in Hadoop. 1. Modify host as normal datanode. Add namenode ip2. modify namenode configuration file confslaves add ip or host3. on the machine of the new node, start the service [root @ slave-004hadoop] #. binhadoop-daemon.shstartdatanode [root @ slav

Add, delete, and delete nodes in Hadoop. 1. Modify host as normal datanode. Add the ip address of namenode 2. modify the configuration file conf/slaves of namenode to add the ip address or host of the new node. 3. on the machine of the new node, start the service [root @ slave-004 hadoop] #. /bin/hadoop-daemon.sh start datanode [root @ slav

Add and delete nodes in Hadoop

Add Node

1. Modify host

It is the same as normal datanode. Add namenode ip Address

2. modify the configuration file conf/slaves of namenode.

Add the ip address or host of the new node

3. Start the service on the machine of the new node.

[Root @ slave-004 hadoop] #./bin/hadoop-daemon.sh start datanode

[Root @ slave-004 hadoop] #./bin/hadoop-daemon.sh start tasktracker

4. Balanced block

[Root @ slave-004 hadoop] #./bin/start-balancer.sh

1) if not, the cluster will store new data on the new node, which will reduce the efficiency of mapred.

2) set the balance threshold. The default value is 10%. The lower the value, the more balanced the nodes, but the longer the consumption time.

[Root @ slave-004 hadoop] #./bin/start-balancer.sh-threshold 5

3) set the balance bandwidth. The default value is only 1 M/s.

1

2 Dfs. balance. bandwidthPerSec

3 1048576

4

5 Specifies the maximum amount of bandwidth that each datanode

6 can utilize for the balancing purpose in term

7 the number of bytes per second.

8

9

Note:

1. Make sure that the firewall of slave is disabled;

2. make sure that the new Server Load balancer ip address has been added to the master and other Server Load balancer/etc/hosts, otherwise, add the master and other slave ip addresses to the new slave/etc/hosts.

Delete a node

1. cluster configuration

Modify the conf/hdfs-site.xml File

1

2 Dfs. hosts. exclude

3 /Data/soft/hadoop/conf/excludes

4 Names a file that contains a list of hosts that are

5 not permitted to connect to the namenode. The full pathname of

6 file must be specified. If the value is empty, no hosts are

7 excluded.

8

2. Confirm the machine to be dismounted.

Dfs. hosts. exclude defines the file content as one line for each machine to be deprecated. This will prevent them from connecting to Namenode. For example:

Slave-003

Slave-004

3. Force reload Configuration

[Root @ master hadoop] #./bin/hadoop dfsadmin-refreshNodes

It moves Block blocks in the background.

4. Close the node

After the operation is completed, the unmounted machines can be safely shut down.

[Root @ master hadoop] #./bin/hadoop dfsadmin-report

You can view the nodes connected to the current cluster.

Executing Decommission will show:

Decommission Status: Decommission in progress

After the execution is completed, the following displays:

Decommission Status: Decommissioned

5. Edit the excludes file again.

Once the machine is dismounted, they can be removed from the excludes file.

Log on to the machine to be dismounted and you will find that the DataNode process is gone, but the TaskTracker still exists. You need to handle it manually.