Common hadoop exceptions and Handling Methods

Exception 1:

Java. Lang. Exception: Java. Lang. outofmemoryerror: Java heap Space

At org. Apache. hadoop. mapred. localjobrunner $ job. Run (Localjobrunner. Java: 400)

Caused by: Java. Lang. outofmemoryerror: javaheap Space

Solution: To solve this problem, you need to adjust the hadoop runtime memory. If the client uses myeclipse, check if eclipse is fully running.

Exception 2:

Error org. Apache. hadoop. HDFS. server. datanode. datanode: incompatible namespaceids... datanode namespaceid... the problem is that namespaceid on namenode is inconsistent with namespaceid on datanode. Cause: each time namenodeformat creates a new namenodeid, and TMP/dfs/data contains the ID in the last format. namenode format clears the data in namenode, however, the data under datanode is not cleared, so the namespaceid on the namenode node is inconsistent with the namespaceid on the datanode node. Startup failed.

Solution: refer to the Web site http://blog.csdn.net/wh62592855/archive/2010/07/21/5752199.aspx to give two solutions, we use the first solution:

(1) Stop the cluster service

(2) Delete the data directory on the problematic datanode node, which is the DFS configured in the hdfs-site.xml file. data. dir Directory, which on this machine is/var/lib/hadoop-0.20/Cache/HDFS/dfs/data/(note: at that time, we performed this step on all datanode and namenode nodes. In case it fails to be deleted, you can save a copy of the data directory first ).

(3) format namenode.

(4) restart the cluster.

Solve the problem. One side effect of this method is that all data on HDFS is lost. If you do not recommend this method when storing important data on HDFS, you can try the second method in the provided URL.

Exception 3:

Exception in thread "Main"Org. Apache. hadoop. Security. accesscontrolexception:
Org. Apache. hadoop. Security. accesscontrolexception: Permission denied: User = hadoop/172.25.18.183, access = execute, inode = "hadoop": hadoop: supergroup: RW-r --

The problem is that you do not have the access permission.

Supplemented and revised later