Hadoop: Java. Lang. outofmemoryerror: Unable to create new Native thread

Recently running hadoopProgram, Encountered two problems:

 

1. outofmemoryerror in hadoop

Error: Unable to create new Native thread error initializing attempt_2011100000003_0013_r_000000_0: Java. lang. outofmemoryerror: Unable to create new Native thread at java. lang. thread. start0 (native method) at java. lang. thread. start (thread. java: 614) at java. lang. unixprocess $ 1.run( unixprocess. java: 157) at java. security. accesscontroller. doprivileged (native method) at java. lang. unixprocess. (unixprocess. java: 119) at java. lang. processimpl. start (processimpl. java: 81) at java. lang. processbuilder. start (processbuilder. java: 468) at Org. apache. hadoop. util. shell. runcommand (shell. java: 149) at Org. apache. hadoop. util. shell. run (shell. java: 134) at Org. apache. hadoop. FS. DF. getavailable (DF. java: 73) at Org. apache. hadoop. FS. localdirallocator $ allocatorpercontext. getlocalpathforwrite (localdirallocator. java: 329) at Org. apache. hadoop. FS. localdirallocator. getlocalpathforwrite (localdirallocator. java: 124) at Org. apache. hadoop. mapred. tasktracker. localizejob (tasktracker. java: 750) at Org. apache. hadoop. mapred. tasktracker. startnewtask (tasktracker. java: 1664) at Org. apache. hadoop. mapred. tasktracker. access $1200 (tasktracker. java: 97) at Org. apache. hadoop. mapred. tasktracker $ tasklauncher. run (tasktracker. java: 1629)

when you have this kind of erros when runnning hadoop jobs, there might be a numer of possible reasons thanks to the feeble Implementation of hadoop. one possible reason is because in your mapreduce programs you open too much processes exceeding the default setting of your OS, for example, the default number is 1024 (you can check this number by executing 'ulimit-U '). A perfect example of using processing processes wocould be such a case, in which you want control the output file name based on key-value pair in the reduce stage. to solve this problem, you need to modify some configuration files to raise up the maximum process number you can use, which can be done by editing/etc/security/limits. conf. simply adding the following two lines to the llimits. conf to set the 100000 as the maximum number of processs in your system for user hadoop.

Hadoop soft nproc 100000
Hadoop hard nproc 100000

Other useful resources about OOM in hadoop can be found in the following links:
The dark side of hadoop; nosql; Dealing with outofmemoryerror-in-hadoop;

2. task id: attempt_200912131946_0001_m_000000_0, status: failed too ‑fetch-failures

Reduce taskThe first phase after startup isShuffle, ForwardMapEndFetchData. Each timeFetchBecauseConnectTimeout,ReadTimeout,ChecksumErrors and other causes.Reduce taskFor eachMapA counter is set to recordFetchTheMapNumber of failures in the output. Notification will be sent when the number of failures reaches a certain threshold.Jobtracker fetchTheMapThe output operation failed too many times and printed as follows:Log:

Failed to fetch map-output from attempt_201105261254_102769_m_001802_0 even after max_fetch_retries_per_map retries... reporting to the jobtracker

Blog: http://blog.csdn.net/liangliyin/article/details/6455713

I think it is a communication problem between machine nodes.

 

There are some good posts below:

Http://www.360doc.com/content/11/0323/23/23378_104035203.shtml