An error occurred while debugging the hadoop program in eclipse.

Mapreduce debugging on a single machine recently Program At that time, because Code With Chinese characters in it, I changed the eclipse encoding from default to utf8 to GBK, and then found that the code can run to the program and cannot run now.

Java. Io. ioexception: expecting a line not the end of stream
At org. Apache. hadoop. fs. df. parseexecresult (DF. Java: 109) 

The error message is basically the same as the following. For the solution, refer to the following description.

Http://hi.baidu.com/amdkings/blog/item/b589a5f56c1ddae17609d78f.html

 

The system is a ubuntu-10.04
IDE uses eclipse
At this time, there was an incredible phenomenon. There was no problem in running the nutch in the terminal. However, once it is switched to eclipse for debugging, nutch cannot run and the following error is reported:
Java. Io. ioexception: expecting a line not the end of stream
At org. Apache. hadoop. fs. df. parseexecresult (DF. Java: 109)
At org. Apache. hadoop. util. Shell. runcommand (shells. Java: 179)
At org. Apache. hadoop. util. Shell. Run (shell. Java: 134)
At org. Apache. hadoop. fs. df. getavailable (DF. Java: 73)
At org. Apache. hadoop. fs. localdirallocator $ allocatorpercontext. getlocalpathforwrite (localdirallocator. Java: 321)
At org. Apache. hadoop. fs. localdirallocator. getlocalpathforwrite (localdirallocator. Java: 124)
At org. Apache. hadoop. mapred. mapoutputfile. getspillfileforwrite (mapoutputfile. Java: 107)
At org. Apache. hadoop. mapred. maptask $ mapoutputbuffer. sortandspill (maptask. Java: 930)
At org. Apache. hadoop. mapred. maptask $ mapoutputbuffer. Flush (maptask. Java: 842)
At org. Apache. hadoop. mapred. maptask. Run (maptask. Java: 343)
At org. Apache. hadoop. mapred. localjobrunner $ job. Run (localjobrunner. Java: 138)

After checking the information on the Internet, this problem is basically encountered in windows, because there is no DF command in windows. However, I happen on Ubuntu. These two situations are not the same thing.

I tracked the code to Org. apache. hadoop. FS. DF. parseexecresult: Observe the input parameter bufferedreader lines of this function. The content displayed in lines is garbled, but it is indeed the result after DF is correctly executed. At this time, I understand that it was originally the result of two lines of DF. Due to garbled characters, it was changed to one line, which caused the above error.

Solution: Modify debugconfigurations and set console encoding on the commen option page to UTF-8. the previous value is default-inherited. Because I set the project default encoding to GBK, the default value is GBK.

At this point, the DF command itself runs normally without errors. Unlike some people in South Korea and Japan on the Internet who say that permission issues cause DF to fail, however, after the result is output to the eclipse console, the result deteriorated due to the encoding settings of the console, resulting in an exception in parseexecresult.

It took a whole day to read more than 20 technical blogs in Chinese, Japanese, Korean, and English languages. I didn't expect it to be a problem caused by modifying the project code without authorization. By the way, after cygwin was installed, I had a smooth sailing experience in windows, but I did not have this problem.

However, I still want to give some suggestions for the development of the hadoop project. If possible, I 'd better re-design the DF and DF-related classes to automatically identify the encoding format of the command results, this will not cause errors.

The project was initially developed in windows and used eclispe.