Hadoop upload file times wrong: could only being replicated to 0 nodes instead of minreplication (=1) ....

Problem

Upload file to Hadoop exception, error message is as follows:

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /home/input/qn_log.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.

Solve

1. View the process of the problem node:

Datanode process does not start

2. View Hadoop datanode.log Information

2018-08-17 05:48:58,076 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/usr/local/hadoop2.7/dfs/data/java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop2.7/dfs/data: namenode clusterID = CID-e1a65f22-f0f6-4423-8c2b-03edd2f30766; datanode clusterID = CID-647259e5-0250-4676-8327-a09f8ccd38a7

Error message for, Namenode Clusterid and Datanode clusterid inconsistent!

The following were:

namenode clusterID = CID-e1a65f22-f0f6-4423-8c2b-03edd2f30766datanode clusterID = CID-647259e5-0250-4676-8327-a09f8ccd38a7

In retrospect, the reason for this problem is that I re-formatted HDFS after restarting the Docker container, resulting in inconsistent versions .

3, Solve:

Method: Change the version of Datanode to the same as Namenode

(1) Modify the Dfs/data/current/version file, change the value of Clusterid to the value of Clusterid with Namenode
(2) Restart the cluster, note, do not perform namenode format, as follows:

$HADOOP_HOME/sbin/start-dfs.sh$HADOOP_HOME/sbin/start-yarn.sh

Check the node process again.

The Datanode process started up!

Try uploading again, OK.

Hadoop upload file times wrong: could only being replicated to 0 nodes instead of minreplication (=1) ....