Pseudo-distributed environment for Windows configuration hadoop-1.1.0 (cont.)

Pseudo-distributed environment for Windows configuration hadoop-1.1.0 (cont.) Blog Category: Bigdata windowshadoop

In the previous article, we introduced a workaround for common problems.

However, when I reload the system, and once again install Cygwin and hadoop-1 in the previous article (http://winseclone.iteye.com/blog/1734737), I find that the pseudo distributed environment always complains when it uses mapred. (I forgot, but I didn't seem to have seen that.) As a gift to install Win8! ）。

Suspected a lot of things, configuration problems, Hadoop.tmp.dir, hadoop-1.1.0 replaced hadoop-1.0.0 and so on.

The error log is as follows:

$ hhadoop fs-rmr/test/output; Hhadoop jar Hadoop-examples-1.0.0.jar wordcount/test/input/test/outputdeleted hdfs://winse:9000/test/output13/03/ 22:46:07 INFO input. Fileinputformat:total input paths to PROCESS:113/03/23 22:46:08 INFO mapred. Jobclient:running job:job_201303232144_000213/03/23 22:46:09 INFO mapred. Jobclient:map vs reduce 0%13/03/23 22:46:16 INFO mapred. Jobclient:task id:attempt_201303232144_0002_m_000002_0, Status:FAILEDjava.lang.Throwable:Child Error at Org.apache.hadoop.mapred.TaskRunner.run (taskrunner.java:272) caused by:java.io.IOException:Task process exit with Nonzero status Of-1. At Org.apache.hadoop.mapred.TaskRunner.run (taskrunner.java:259) 13/03/23 22:46:16 WARN mapred. Jobclient:error Reading Task outputhttp://winse:50060/tasklog?plaintext=true&attemptid=attempt_201303232144_ 0002_M_000002_0&FILTER=STDOUT13/03/23 22:46:16 WARN mapred. Jobclient:error Reading Task outputhttp://winse:50060/tasklog?plaintext=true&attemptid=attempt_201303232144_0002_M_000002_0&FILTER=STDERR13/03/23 22:46:22 INFO mapred. Jobclient:task Id:attempt_201303232144_0002_m_000002_1, status:failed

After modification of the original, continuous modification, add sysout printing, is the final find the wrong place of the program!

Org.apache.hadoop.mapred.DefaultTaskController.java #launchTask
Org.apache.hadoop.mapred.JvmManager.java #runChild
Org.apache.hadoop.mapred.TaskRunner.java #launchJvmAndWait

Org.apache.hadoop.fs.FileUtil.java #checkReturnValue
Org.apache.hadoop.fs.RawLocalFileSystem.java #setPermission #mkdirs

Found in the Org.apache.hadoop.fs.RawLocalFileSystem.mkdirs (path) method, the path method for establishing a file is checked attempt_201303232144_0002_m_000001_ 0 if the folder will fail!

View in Cygwin:

Winseliu@winse ~/hadoop/logs/userlogs/job_201303232144_0002$ ll Total dosage 9lrwxrwxrwx 1 Winseliu None 89 March 22:46 Attempt_ 201303232144_0002_m_000001_0->/cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_ 0002_M_000001_0LRWXRWXRWX 1 Winseliu None 89 March 22:46 attempt_201303232144_0002_m_000001_1->/cluster/mapred/ LOCAL/USERLOGS/JOB_201303232144_0002/ATTEMPT_201303232144_0002_M_000001_1LRWXRWXRWX 1 Winseliu None 89 March 23 22:46 Attempt_201303232144_0002_m_000001_2->/cluster/mapred/local/userlogs/job_201303232144_0002/attempt_ 201303232144_0002_M_000001_2LRWXRWXRWX 1 Winseliu None 89 March 22:46 attempt_201303232144_0002_m_000001_3->/ CLUSTER/MAPRED/LOCAL/USERLOGS/JOB_201303232144_0002/ATTEMPT_201303232144_0002_M_000001_3LRWXRWXRWX 1 Winseliu None 89 March 22:46 attempt_201303232144_0002_m_000002_0->/cluster/mapred/local/userlogs/job_201303232144_0002/ ATTEMPT_201303232144_0002_M_000002_0LRWXRWXRWX 1 Winseliu None 89 March 22:46 attempt_201303232144_0002_m_000002_1->/cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000002_ 1LRWXRWXRWX 1 Winseliu None 89 March 22:46 attempt_201303232144_0002_m_000002_2->/cluster/mapred/local/userlogs/job _201303232144_0002/ATTEMPT_201303232144_0002_M_000002_2LRWXRWXRWX 1 Winseliu None 89 March 22:46 attempt_201303232144_ 0002_m_000002_3->/cluster/mapred/local/userlogs/job_201303232144_0002/attempt_201303232144_0002_m_000002_3- Rwxr-xr-x 1 Winseliu None 404 March 22:46 Job-acls.xml

For Linux, these are references to another folder, which itself should also be folders! But the Windows JDK doesn't know these things!

public boolean mkdirs (path F) throws IOException {Path parent = f.getparent (); File p2f = PathToFile (f); return (parent = NULL | | mkdirs (parent)) && (P2f.mkdir () | | p2f.isdirectory ()); }

So in the Judgment P2f.isdirectory () returns false, then throws the IOException, finally exits the map child's program in 1 state!

Use the Org.apache.hadoop.mapred.TaskRunner.prepareLogFiles (Taskattemptid, Boolean) method to set the position of the output log output. The final execution will output sysout and Syserr to the log file in the shell command. Userlogs's parent directory is configured with Hadoop.log.dir System Properties!

Mapred. Defaulttaskcontroller.launchtask ()

|--mapred. Tasklog.buildcommandline ()

Temporary Solution:

Locate the Hadoop.log.dir to the real mapred log directory (mapred.local.dir: ${hadoop.tmp.dir}/mapred/local)!

Export Hadoop_log_dir=/cluster/mapred/local

Map Windows/cluster to Cygwin (Linux)/cluster:

Winseliu@winse ~/hadoop$ ll/cygdrive/c | grep clusterdrwxr-xr-x+ 1 winseliu None 0 March 00:08 clusterwinseliu@winse ~/hadoop$ ll/| grep clusterlrwxrwxrwx 1 Winseliu None 19 March 09:39 cluster->/cygdrive/c/cluster

However, when running the example of WordCount, it is still not normal! When viewing Tasktracker's log, a string was found to be an integer numberformatexception exception!

Modify the Org.apache.hadoop.mapred.JvmManager.JvmManagerForType.JvmRunner.kill () method. Add Pidstr to an empty string check!

String pidstr = Jvmidtopid.get (Jvmid); if (pidstr!= null &&!pidstr.isempty ()) {

Then, finally see finish! The results were also seen in the/test/output/part-r-00000.

other simplified processing, that is, configuration file:

alias startcluster= "~/hadoop/bin/start-all.sh" Alias Stopcluster= ~/hadoop/bin/stop-all.sh; ~/hadoop/bin/stop-all.sh "Alias hhadoop=" ~/hadoop/bin/hadoop "

Winseliu@winse ~$ ll | grep hadooplrwxrwxrwx 1 Winseliu None 12 March 10:44 Hadoop-> hadoop-1.0.0drwx------+ 1 Winseliu None 0 March 24 00:06 hadoop-1.0.0

<!--Core-site.xml--><configuration><property><name>fs.default.name</name>< Value>hdfs://winse:9000</value></property><property><name>hadoop.tmp.dir</name ><value>/cluster</value></property></configuration><!--Hdfs-site.xml-->< configuration><property><name>dfs.replication</name><value>1</value></ Property><property> <name>dfs.permissions</name> <value>false</value></ Property><property> <name>dfs.permissions.supergroup</name> <value>None</value> </property><property><name>dfs.safemode.extension</name><value>1000</value ></property></configuration><!--Mapred-site.xml--><configuration><property> <name>mapred.job.tracker</name><value>winse:9001</value></property></ Configuration>

For more information about viewing a startup process, look through task Manager to see:

Size: 56.6 KB View picture attachments