Description of the problem: the program is executed on hadoop and can be executed normally after the program runs. Everything seems to be normal, but after a period of time, the program begins to block until the program times out and exits (as follows ).
14/08/19 21:17:51 INFO mapred.JobClient: map 99% reduce 71%14/08/19 21:17:54 INFO mapred.JobClient: map 99% reduce 75%14/08/19 21:17:57 INFO mapred.JobClient: map 99% reduce 79%14/08/19 21:18:00 INFO mapred.JobClient: map 99% reduce 83%14/08/19 21:18:03 INFO mapred.JobClient: map 99% reduce 87%14/08/19 21:18:06 INFO mapred.JobClient: map 99% reduce 91%
This problem occurs because of program exceptions that cause task execution to fail. However, hadoop does not exit or restart the task.
Exception 1: Program glass Error
attempt_201408192045_0002_m_000196_2: [2014-08-19 21:16:44 WARN ] [main] (org.apache.hadoop.mapred.Child:291) - Error running childattempt_201408192045_0002_m_000196_2: java.io.IOException: Index: 0, Size: 0attempt_201408192045_0002_m_000196_2: at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:61)attempt_201408192045_0002_m_000196_2: at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:1)attempt_201408192045_0002_m_000196_2: at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)attempt_201408192045_0002_m_000196_2: at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)attempt_201408192045_0002_m_000196_2: at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)attempt_201408192045_0002_m_000196_2: at org.apache.hadoop.mapred.Child$4.run(Child.java:270)attempt_201408192045_0002_m_000196_2: at java.security.AccessController.doPrivileged(Native Method)attempt_201408192045_0002_m_000196_2: at javax.security.auth.Subject.doAs(Subject.java:416)attempt_201408192045_0002_m_000196_2: at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)attempt_201408192045_0002_m_000196_2: at org.apache.hadoop.mapred.Child.main(Child.java:264)attempt_201408192045_0002_m_000196_2: Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0attempt_201408192045_0002_m_000196_2: at java.util.ArrayList.rangeCheck(ArrayList.java:571)attempt_201408192045_0002_m_000196_2: at java.util.ArrayList.get(ArrayList.java:349)attempt_201408192045_0002_m_000196_2: at com.ict.wxparser.parser.WXParser.getMsgContent(WXParser.java:188)attempt_201408192045_0002_m_000196_2: at com.ict.wxparser.parser.WXParser.parseLine(WXParser.java:137)attempt_201408192045_0002_m_000196_2: at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:57)attempt_201408192045_0002_m_000196_2: ... 9 moreattempt_201408192045_0002_m_000196_2: [2014-08-19 21:16:44 INFO ] [main] (org.apache.hadoop.mapred.Task:956) - Runnning cleanup for the task14/08/19 21:17:18 INFO mapred.JobClient: Task Id : attempt_201408192045_0002_m_000196_3, Status : FAILEDjava.io.IOException: Index: 0, Size: 0 at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:61) at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:1) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264)Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck(ArrayList.java:571) at java.util.ArrayList.get(ArrayList.java:349) at com.ict.wxparser.parser.WXParser.getMsgContent(WXParser.java:188) at com.ict.wxparser.parser.WXParser.parseLine(WXParser.java:137) at com.ict.hadoop.WXExtraction$Map.map(WXExtrac
The key to solving this problem is to modify the code so that program tasks can be executed normally.
Exception 2: org. Apache. hadoop. mapred. Child: Error running child: Java. Lang. outofmemoryerror: Unable to create new Native thread
This problem indicates that the program memory has exceeded. At this time, an overflow exception is thrown and the program execution fails.
Solution:
1. Increase the hadoop_heapsize value in the hadoop-env.sh
2. Increase the value of mapred. Child. java. opts in the mapred-site.xml (default: 200 m)
<property><name>mapred.child.java.opts</name><value>-Xmx2048m</value></property>
3. Reduce the values of mapred. tasktracker. Map. Tasks. maximumde and mapred. tasktracker. Reduce. Tasks. Maximum in the mapred-site.xml
<property><name>mapred.tasktracker.map.tasks.maximum</name><value>15</value></property>
Hadoop development problems: reduce getting stuck