Strange problems caused by mapred streaming Script output running status

Today, when debugging the mapred streaming script, a very strange problem occurs: The mapred script will always fail randomly after being called by tasktracker. The error message is as follows:

java.lang.RuntimeException: java.lang.NullPointerExceptionat org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:95)at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:518)at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419)at org.apache.hadoop.mapred.Child$4.run(Child.java:259)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)at org.apache.hadoop.mapred.Child.main(Child.java:253)Caused by: java.lang.NullPointerExceptionat org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:522)at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:454)

From the error message, we can know that the error is not caused by a mapred script error, but an internal error occurs in tasktracker when tasktracker initializes the mapred script.

At first, I thought it was an error caused by insufficient system resources on the cluster server. But after checking other running tasks on the cluster, I did not find the same problem as above, other running tasks do not throw the preceding error. Now it must be my mapred script. Considering the redis I use in the mapred script, my redis interface package was modified on the basis of redis-py (redis-py does not support python2.4). I suspect that the above problem is caused by the redis-py package, as a result, the code of using the redis module in the Code is blocked and mapred is run again. The same problem still occurs.

No way, I had to carefully compare the difference between the normal mapred script and the problematic mapred script, and found that compared with the normal mapred script, the current Code has an additional code to report the mapred script running status to jobtarcker:

sys.stderr.write("reporter:status:output\n")

Although in theory this statement should not cause the above strange problems, but it is still a dead horse to try, shield the code in the mapred script, re-run the mapred script, a miracle has occurred, everything is normal, no error occurs. If you re-Add the redis operation code and run the mapred script again, there is still no error. It seems that the culprit of the problem is the code that reports the status to jobtracker:

sys.stderr.write("reporter:status:output\n")

The above error should be a small bug in tasktracker. Do you know if someone submitted this bug? Our cluster will be upgraded to hadoop1.0.3 soon, and we will try again on hadoop1.0.