Hadoop practice 101: add and delete machines in a hadoop Cluster

ArticleDirectory

• Insecure
• Secure Mode

No downtime is required for adding or deleting machines in the hadoop cluster, and the entire service is not interrupted.

Before this operation, the hadoop cluster is as follows:

HDFS machines are as follows:

The MR machine is as follows:

Add Machine

On the master machine of the cluster, modify the $ hadoop_home/CONF/slaves file and add the Host Name of the new machine (hp3) to be added to the cluster:

Hp3
Hp2
Dell1
Dell2
Dell3
Dell4

Then run the following command on the master machine:

$ Hadoop_home/bin/start-all.sh

After this operation is completed, the new machine is added to the cluster.

A new machine is added to the HDFS cluster:

A machine is also added to the Mr Cluster:

Unsafe ways to delete machines

Because the hadoop cluster has good fault tolerance, you can directly shut down the corresponding machine to remove the machine. However, if more than three machines are operated at a time, some data may be lost. Therefore, this method is not recommended.

Secure Mode

Create a new file $ hadoop_home/CONF/nn-excluded-list on the master machine of the cluster. In this file, specify the Host Name (hp3) to be deleted ):

Hp3

Then, modify the configuration file for the master machine: $ hadoop_home/CONF/hdfs-site.xml and add the following:

<Property>

<Name> DFS. Hosts. Exclude </Name>

<Value> CONF/nn-excluded-list </value>

</Property>

Finally, run the following command on the master machine:

$ Hadoop_home/bin/hadoop dfsadmin-refreshnodes

After this operation, you can see in the HDFS cluster that the hp3 machine is already in the decommission in progress status:

However, hp3 is still running on Mr:

We need to wait for a certain amount of time. After the decommission operation of datanode in hp3 is completed, we can close all hadoop processes in hp3.

In this way, the whole operation of deleting machines from the cluster is completed.

For more articles about hadoop, see: http://www.cnblogs.com/gpcuster/tag/Hadoop/