Common startup errors for Hadoop:hadoop (new issues will continue to be updated)

Recently in learning Hadoop, encountered some common problems, now to sum up, and you share, will continue to update the other problems encountered. When using Hadoop to find a problem, first use the JPS command to see if the starting node is correct, and then view the log file.

Original article, reproduced please indicate the source: http://blog.csdn.net/jessonlv/article/details/18221299

In general, we will meet several errors:

1, set host name error

Looking at the log, you will find the following error:

ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:java.net.UnknownHostException:
Invalid hostname for Server:master

This is due to an error in the host name setting, please check the configuration file for the host name settings, correct.
First use the hostname command to see if the host name is correct;
Then use the More/etc/sysconfig/network command to see if the host name is logged in the file;
Finally, use the more/etc/hosts command to see if the IP and host name mappings are set.

2, IP settings error

Look at the log and find the following error

ERROR Org.apache.hadoop.hdfs.server.namenode.NameNode:java.net.BindException:Problem
binding to book0/ 192.168.1.100:9000:cannot Assign requested Address

This is caused by an incorrect IP address setting, check that the IP settings for the host are consistent with the IP settings of the profile.
Use the more/etc/hosts command to see whether the IP and hostname mappings are set.

3, hostname can not be underlined, can not start with the number

Hostname with underscores can also cause startup failures.

After startup is complete, be sure to use the JPS command to see if all 5 nodes started successfully. If a node does not have a reboot
, please check the corresponding log log. The default directory for the log is $hadoop_home/logs.
Here's a screenshot of my log:

File naming is regular, "hadoop-[the current username]-[node name]-[host name].log" We only look at the file at the end of log. If the Namenode node is not started, view the Hadoop-root-namenode-book0.log file.

4. Perform Hadoop formatting multiple times

Phenomenon: The following error occurred in Hadoop-root-datanode-master.log:

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:java.io.IOException:
incompatible namespaceids in

Reason: Each namenode format will re-create a Namenodeid, and the Dfs.data.dir parameter configuration directory contains the ID created by the last format, and the ID in the directory configured by the Dfs.name.dir parameter is inconsistent.
The Namenode format clears the data under Namenode, but does not empty the data under Datanode, causing the startup to fail, and all you have to do is empty the Dfs.data.dir parameter configuration directory before each fotmat. Reformat the HDFs command.

5, the firewall is not closed

The following error occurred while uploading files from the local to the HDFs file system:

The most likely problem is that the firewall does not shut down, causing the node Datanode to communicate with the node Namenode failed. You can use the Service iptables Status command to turn off the firewall. After this shutdown, the firewall may reboot when the operating system is restarted, and the automatic restart function can be turned off. Use Chkconfig iptables off.

6. Error caused by Safe mode

The error prompts are as follows:

Org.apache.hadoop.dfs.SafeModeException:Cannot Delete ..., Name node is in
Safe mode

When the Distributed file system is started, there is a safe mode at the beginning, and when the Distributed file system is in Safe mode, the contents of the file system are not allowed to be modified or deleted until Safe mode is finished. The main purpose of security mode is to check the validity of the data blocks on each datanode when the system is started, and to copy or delete some pieces of data according to the necessary policies. The runtime can also enter Safe Mode by command. In practice, when the system starts to modify and delete files there will be Safe mode does not allow modification of error prompts, only need to wait for a while. If you are in a hurry, you can perform the Hadoop dfsadmin–safemode leave command to turn off safe mode.

Summary: The above is a few days to learn Hadoop, encountered a few problems, the following learning process will certainly meet a variety of problems, I will write down, and find solutions, continue to update this article.

In addition, I will be the learning process and do the notes and share with you.

Finally, recently found that a lot of forums grilled my article, let me very distressed, do not know what the majority of students have solutions to help support the idea, thank you all ~

2014/1/14 Update:

Today, in configuring the Hadoop configuration file, I made a small omission, please look at the following:

When I was configuring the hadoop-env.sh,

The contents of the Nineth line of this file were originally:

#export java_home=/usr/lib/jdk-1.6.0/

We need to change this to our own JDK directory, our directory is/USE/LOCAL/JDK
I change the time, only the front of the # number removed, only jdk-1.6.0 this name changed, did not change Lib to local, when the start of Hadoop caused by the following error:

Can not find JDK, hehe, should not be careless, recorded, a long memory ....