Could not find output/file.out to the configured local directories

Reprint: http://blog.pureisle.net/archives/1785.html

12. If you encounter the following error:

FAILED java.lang.IllegalArgumentException:java.net.URISyntaxException:Relative path in absolute uri:***

There are characters that are not allowed in the URI, such as a colon: a file name character that is not allowed by the operating system. Detailed grep matching can be viewed based on the portion of the hint (the asterisk section). The elimination will be solved.

13, encountered Tasktracker can not start, Tasktracker log the following error:

ERROR Org.apache.hadoop.mapred.TaskTracker:Can not start Task Tracker because Java.net.BindException:Address already in Use * * * *

Is that the port is occupied or a corresponding process has been started, you stop the cluster first and then use Ps-aux | grep hadoop command to look at the associated Hadoop process and kill the daemon-related daemons.

14, encountered Datanode can not start the situation, Datanode log the following error:

ERROR Org.apache.hadoop.hdfs.server.datanode.DataNode:java.io.IOException:No Locks available

There are many on the web that need format, personal feeling is nonsense, because I add the same node, add other nodes no problem, only one has a problem, so the problem must not be in the format, and if the format after the HDFS data will be all lost, I will not try this method. In detail, the Apache mailing list has this section:

No locks available can mean that for you are trying to use Hadoop on a filesystem that does not support file level locking. Are you trying to run your name node storage in NFS spaces?

This refers to the case of file-level locks, using the

$/ETC/INIT.D/NFS Status

command to view the network file system, all is turned off. In addition, you can view the type of file system using the df-th or Mount command, and the result is indeed an NFS file system problem. You cannot use the network file system that hangs on, because it seems to be read-only, even if it is not read-only, as mentioned above, it does not support file-level locks.
Finally, you can try to add file-level locks to NFS. I am here to modify the Dfs.data.dir, do not use NFS finished.

XV, Datanode died, and unable to start the process, log reported the following error:

2012-06-04 10:31:34,915 INFO org.apache.hadoop.hdfs.server.common.Storage:Cannot access Storage directory/data5/ Hadoop_data
2012-06-04 10:31:34,915 INFO org.apache.hadoop.hdfs.server.common.Storage:Storage Directory/data5/hadoop_data does Not exist.
2012-06-04 10:31:35,033 ERROR org.apache.hadoop.hdfs.server.datanode.datanode:org.apache.hadoop.util.diskchecker$ Diskerrorexception:invalid value for Volsfailed:2, volumes tolerated:0

This question of mine, the reason is that the disk of the node has become read-only (only read) mode, a search on the internet found that this is quite a lot, the Linux machine's hard drive is set to read and write (Read/write) way, but occasionally found automatically become read only ), there are a number of reasons for this, possible problems: File system error kernel-related hardware driver bug FW firmware class problem disk bad drive backplane failure hard drive cable fault HBA card failure RAID card failure inode resource exhaustion

Solution: Reboot the server (command reboot) re-mount HDD fsck try to repair the replacement hard drive

Attach a detailed solution (not to try, look very reliable).

16, JPS can not see the Hadoop, hbase daemon daemon, but Hadoop, hbase are working properly, Web pages can be accessed, with Ps-ef|grep Java can also see the boot of the Java process.

The reason for this is that when the Java program is started, the default (note is the default) creates a new file in the/tmp/hsperfdata_username directory with the ID of the process, and stores information about the JVM running in that file. The username is the current username, and the/tmp/hsperfdata_username directory holds all the Java process information that the user has started. For Windows machine/tmp, use Windows to store temporary file directories instead. And JPS, Jconsole, JVISUALVM and other tools of the data source is this file (/tmp/hsperfdata_username/pid). So when the file does not exist or cannot be read, JPS cannot view the process number, Jconsole cannot monitor, and so on.

So if your TMP has regular emptying, disk full, write permissions, and so on, will cause JPS can not see the situation of the Java process, you need to resolve the cache can be set to save location. The parameter for setting the file location is-djava.io.tmpdir

Other than that:
The/tmp/hsperfdata_username/pid file is purged after the corresponding Java process exits. If the Java process does not exit normally (such as kill-9), the PID file is retained until a Java command is executed or commands (such as JPS, Javac, Jstat) that load the JVM program are purged of all unwanted PID files.

17, the implementation of the MapReduce task when the error is as follows:

2012-06-21 10:50:43,290 WARN org.mortbay.log:/mapoutput:org.apache.hadoop.util.diskchecker$diskerrorexception: Could not find tasktracker/hadoop/jobcache/job_201206191809_0004/attempt_201206191809_0004_m_000006_0/output/ File.out.index in any of the configured local directories
2012-06-21 10:50:45,592 WARN org.apache.hadoop.mapred.TaskTracker:getMapOutput (attempt_201206191809_0004_m_000006 _0,0) failed:org.apache.hadoop.util.diskchecker$diskerrorexception:could not find Tasktracker/ha.doop/jobcache/job _201206191809_0004/attempt_201206191809_0004_m_000006_0/output/file.out.index in any of the configured local Directories

Although it is two warn, but also affect the efficiency of the job, so try to solve the error because the job can not find the middle output file. The following checks are required:

A, configure the Mapred.local.dir properties
b, df-h see if there is enough space under the cache path
C, free to see if the memory space is enough
D, make sure the cache path can write permissions
E, check for disk corruption

The Namenode circulation error is as follows:

2012-08-21 09:20:24,486 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem:Cannot Roll Edit log, edits.new files already exists in all healthy directories:
/data/work/hdfs/name/current/edits.new
/backup/current/edits.new
2012-08-21 09:20:25,357 ERROR org.apache.hadoop.security.UserGroupInformation:PriviledgedActionException as:hadoop Cause:java.net.ConnectException:Connection refused
2012-08-21 09:20:25,357 ERROR org.apache.hadoop.security.UserGroupInformation:PriviledgedActionException as:hadoop Cause:java.net.ConnectException:Connection refused
2012-08-21 09:20:25,359 WARN Org.mortbay.log:/getimage:java.io.ioexception:getimage failed. Java.net.ConnectException:Connection refused

Secondarynamenode also has related errors.
The reason for the search is:

With 1.0.2, only one checkpoint process was executed at a time. When the namenode gets a overlapping checkpointing request, it checks edit.new in its storage directories. If all of them have this file, Namenode concludes the previous checkpoint process isn't done yet and prints the warning m Essage you ' ve seen.

In this case, if you make sure that the Edits.new file is not used before the wrong operation of the file, then you can delete, after testing whether there is such a problem.
Also, make sure that the Namenode hdfs-site.xml configuration has the following:

<property>
<name>dfs.secondary.http.address</name>
<value>0.0.0.0:50090</value>
</property>

Modify the above 0.0.0.0 for you to deploy the Secondarynamenode hostname

Secondarynamenode's hdfs-site.xml has the following items:

<property>
<name>dfs.http.address</name>
<value>0.0.0.0:50070</value>
</property>

Modify the above 0.0.0.0 for you to deploy the Namenode hostname