Namenode and datanode cannot be started. Error: fsnamesystem initialization failed. datanode. datanode: incompatible namespaceids

Question 1:

Namenode cannot be started. Check the log. The error message is as follows:

Org. Apache. hadoop. HDFS. server. namenode. fsnamesystem: fsnamesystem initialization failed.

Java. Io. ioexception: namenode is not formatted.
At org. Apache. hadoop. HDFS. server. namenode. fsimage. recovertransitionread (fsimage. Java: 317)
At org. Apache. hadoop. HDFS. server. namenode. fsdirectory. loadfsimage (fsdirectory. Java: 87)
At org. Apache. hadoop. HDFS. server. namenode. fsnamesystem. initialize (fsnamesystem. Java: 311)
At org. Apache. hadoop. HDFS. server. namenode. fsnamesystem. <init> (fsnamesystem. Java: 292)
At org. Apache. hadoop. HDFS. server. namenode. namenode. initialize (namenode. Java: 201)
At org. Apache. hadoop. HDFS. server. namenode. namenode. <init> (namenode. Java: 279)
At org. Apache. hadoop. HDFS. server. namenode. namenode. createnamenode (namenode. Java: 956)
At org. Apache. hadoop. HDFS. server. namenode. namenode. Main (namenode. Java: 965)
00:34:55, 813 info org. Apache. hadoop. IPC. SERVER: Stopping server on 9000
00:34:55, 814 error org. Apache. hadoop. HDFS. server. namenode. namenode: Java. Io. ioexception: namenode is not formatted.
At org. Apache. hadoop. HDFS. server. namenode. fsimage. recovertransitionread (fsimage. Java: 317)
At org. Apache. hadoop. HDFS. server. namenode. fsdirectory. loadfsimage (fsdirectory. Java: 87)
At org. Apache. hadoop. HDFS. server. namenode. fsnamesystem. initialize (fsnamesystem. Java: 311)
At org. Apache. hadoop. HDFS. server. namenode. fsnamesystem. <init> (fsnamesystem. Java: 292)
At org. Apache. hadoop. HDFS. server. namenode. namenode. initialize (namenode. Java: 201)
At org. Apache. hadoop. HDFS. server. namenode. namenode. <init> (namenode. Java: 279)
At org. Apache. hadoop. HDFS. server. namenode. namenode. createnamenode (namenode. Java: 956)
At org. Apache. hadoop. HDFS. server. namenode. namenode. Main (namenode. Java: 965)

Cause: HDFS is not initialized after the tmp directory in the configuration file is modified. As a result, namenode is not initialized when hadoop is started.

Solution: Delete the DFS and mapred folders in the hadoop/tmp directory, and format the hadoop

Rm-rf dfs/

Rm-RF mapred/

Bin/./hadoop namenode-format

OK, but sometimes it will cause problem 2.

 

Question 2:

Datanode cannot be started

View the following log errors

 

Error Org. apache. hadoop. HDFS. server. datanode. datanode: Java. io. ioexception: incompatible namespaceids in/home/admin/Joe. wangh/hadoop/data/DFS. data. dir: namenode namespaceid = 898136669; datanode namespaceid = 2127444065
At org. Apache. hadoop. HDFS. server. datanode. datastorage. dotransition (datastorage. Java: 233)
At org. Apache. hadoop. HDFS. server. datanode. datastorage. recovertransitionread (datastorage. Java: 148)
At org. Apache. hadoop. HDFS. server. datanode. datanode. startdatanode (datanode. Java: 288)
At org. Apache. hadoop. HDFS. server. datanode. datanode. <init> (datanode. Java: 206)
At org. Apache. hadoop. HDFS. server. datanode. datanode. makeinstance (datanode. Java: 1239)
At org. Apache. hadoop. HDFS. server. datanode. datanode. instantiatedatanode (datanode. Java: 1194)
At org. Apache. hadoop. HDFS. server. datanode. datanode. createdatanode (datanode. Java: 1202)
At org. Apache. hadoop. HDFS. server. datanode. datanode. Main (datanode. Java: 1324)

 

The error message "inconsistent namespaceids" is displayed.

Cause: A New namenodeid is created for each namenode format. The tmp/dfs/data contains the ID of the last format. The namenode format clears the data of namenode, however, the data under datanode is not cleared, so the namespaceid on the namenode node is inconsistent with the namespaceid on the datanode node. Startup failed.

Workaround 1: Start from scratch

I can testify that the following steps solve this error, but the side effects won't make you happy (Me neither). The crude workaround I have found is:

1. Stop the Cluster

2. delete the data directory on the problematic datanode: the directory is specified by DFS. data. dir in CONF/hdfs-site.xml; if you followed this tutorial, the relevant directory is/usr/local/hadoop-datastore/hadoop-hadoop/dfs/Data

3. reformat the namenode (Note: All HDFS data is lost during this process !)

4. Restart the Cluster

When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be OK during the initial setup/testing), you might give the second approach a try.

Workaround 2: updating namespaceid of problematic datanodes

Big thanks to Jared stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. this workaround is "minimally invasive" as you only have to edit one file on the problematic datanodes:

1. Stop the datanode

2. Edit the value of namespaceid in <DFS. Data. dir>/current/version to match the value of the current namenode

3. Restart the datanode

If you followed the instructions in my tutorials, the full path of the relevant file is/usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/version (Background: DFS. data. DIR is by default set to $ {hadoop. TMP. dir}/dfs/data, and we set hadoop. TMP. dir to/usr/local/hadoop-datastore/hadoop-hadoop ).

If you wonder how the contents of version look like, here's one of mine:

# Contents of <DFS. Data. dir>/current/version

Namespace id = 393514426

Storageid = DS-1706792599-10.10.10.1-50010-1204306713481

Ctime= 1215607609074

Storagetype = data_node

Layoutversion =-13

We adopt Method 1:

(1) Stop the cluster service

(2) Delete the data directory on the problematic datanode node, which is the DFS configured in the hdfs-site.xml file. data. dir Directory, which on this machine is/var/lib/hadoop-0.20/Cache/HDFS/dfs/data/(note: at that time, we performed this step on all datanode and namenode nodes. In case it fails to be deleted, you can save a copy of the data directory first ).

(3) format namenode.

(4) restart the cluster.

Solve the problem.

One side effect of this method is that all data on HDFS is lost. If you do not recommend this method when storing important data on HDFS, you can try the second method in the provided URL.

namenode and datanode cannot be started. Error: fsnamesystem initialization failed. datanode. datanode: incompatible namespaceids