Hadoop Practice 101: Adding machines and removing machines in a Hadoop cluster

Whether you are adding machines and removing machines in a Hadoop cluster, there is no downtime and the entire service is uninterrupted.

Before this operation, the cluster of Hadoop is as follows:

The machine condition for HDFs is as follows:

The machine condition of Mr is as follows:

Adding Machines

In the master machine of the cluster, modify the $hadoop_home/conf/slaves file to add the hostname of the new Machine (HP3) that needs to be added to the cluster:

Hp3
Hp2
Dell1
Dell2
Dell3
Dell4

Then execute the following command in the master machine:

$HADOOP _home/bin/start-all.sh

When this is done, the new machine is added to the cluster.

The HDFs cluster adds a new machine:

There is also a new machine in the Mr Cluster:

Remove Machine Unsafe Way

Since the Hadoop cluster itself has good fault tolerance, the machine can be shut down directly to achieve the purpose of removing the machine. However, if a one-time operation of more than 3 machines, it is possible to cause some data loss, so it is not recommended to operate this way. a safe way

In the master machine of the cluster, create a new file: $HADOOP _home/conf/nn-excluded-list, specify the machine hostname (HP3) to be deleted in this file:

Hp3

Then, modify the master machine's configuration file: $HADOOP _home/conf/hdfs-site.xml, add the following:

<property>

<name>dfs.hosts.exclude</name>

<value>conf/nn-excluded-list</value>

</property>

Finally, execute the following command in the master machine:

$HADOOP _home/bin/hadoop Dfsadmin-refreshnodes

Once this is done, you can see in the HDFs cluster that the HP3 machine is already in the decommission in progress State:

But the hp3 machine in the Mr Machine is still in operation:

We need to wait a certain amount of time and wait until the decommission operation in Hp3 Datanode is complete, then close all Hadoop processes to the HP3 machine.

This completes the entire operation of removing the machine from the cluster.

For more articles on Hadoop, refer to: http://www.cnblogs.com/gpcuster/tag/Hadoop/

Transfer from http://www.cnblogs.com/gpcuster/archive/2011/04/12/2013411.html