Hadoop development problems: reduce getting stuck

Source: Internet
Author: User

Description of the problem: the program is executed on hadoop and can be executed normally after the program runs. Everything seems to be normal, but after a period of time, the program begins to block until the program times out and exits (as follows ).

14/08/19 21:17:51 INFO mapred.JobClient: map 99% reduce 71%14/08/19 21:17:54 INFO mapred.JobClient: map 99% reduce 75%14/08/19 21:17:57 INFO mapred.JobClient: map 99% reduce 79%14/08/19 21:18:00 INFO mapred.JobClient: map 99% reduce 83%14/08/19 21:18:03 INFO mapred.JobClient: map 99% reduce 87%14/08/19 21:18:06 INFO mapred.JobClient: map 99% reduce 91%

This problem occurs because of program exceptions that cause task execution to fail. However, hadoop does not exit or restart the task.

Exception 1: Program glass Error

attempt_201408192045_0002_m_000196_2: [2014-08-19 21:16:44 WARN ] [main] (org.apache.hadoop.mapred.Child:291) - Error running childattempt_201408192045_0002_m_000196_2: java.io.IOException: Index: 0, Size: 0attempt_201408192045_0002_m_000196_2:   at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:61)attempt_201408192045_0002_m_000196_2:   at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:1)attempt_201408192045_0002_m_000196_2:   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)attempt_201408192045_0002_m_000196_2:   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)attempt_201408192045_0002_m_000196_2:   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)attempt_201408192045_0002_m_000196_2:   at org.apache.hadoop.mapred.Child$4.run(Child.java:270)attempt_201408192045_0002_m_000196_2:   at java.security.AccessController.doPrivileged(Native Method)attempt_201408192045_0002_m_000196_2:   at javax.security.auth.Subject.doAs(Subject.java:416)attempt_201408192045_0002_m_000196_2:   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)attempt_201408192045_0002_m_000196_2:   at org.apache.hadoop.mapred.Child.main(Child.java:264)attempt_201408192045_0002_m_000196_2: Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0attempt_201408192045_0002_m_000196_2:   at java.util.ArrayList.rangeCheck(ArrayList.java:571)attempt_201408192045_0002_m_000196_2:   at java.util.ArrayList.get(ArrayList.java:349)attempt_201408192045_0002_m_000196_2:   at com.ict.wxparser.parser.WXParser.getMsgContent(WXParser.java:188)attempt_201408192045_0002_m_000196_2:   at com.ict.wxparser.parser.WXParser.parseLine(WXParser.java:137)attempt_201408192045_0002_m_000196_2:   at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:57)attempt_201408192045_0002_m_000196_2:   ... 9 moreattempt_201408192045_0002_m_000196_2: [2014-08-19 21:16:44 INFO ] [main] (org.apache.hadoop.mapred.Task:956) - Runnning cleanup for the task14/08/19 21:17:18 INFO mapred.JobClient: Task Id : attempt_201408192045_0002_m_000196_3, Status : FAILEDjava.io.IOException: Index: 0, Size: 0        at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:61)        at com.ict.hadoop.WXExtraction$Map.map(WXExtraction.java:1)        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)        at org.apache.hadoop.mapred.Child$4.run(Child.java:270)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:416)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)        at org.apache.hadoop.mapred.Child.main(Child.java:264)Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0        at java.util.ArrayList.rangeCheck(ArrayList.java:571)        at java.util.ArrayList.get(ArrayList.java:349)        at com.ict.wxparser.parser.WXParser.getMsgContent(WXParser.java:188)        at com.ict.wxparser.parser.WXParser.parseLine(WXParser.java:137)        at com.ict.hadoop.WXExtraction$Map.map(WXExtrac

The key to solving this problem is to modify the code so that program tasks can be executed normally.

Exception 2: org. Apache. hadoop. mapred. Child: Error running child: Java. Lang. outofmemoryerror: Unable to create new Native thread

This problem indicates that the program memory has exceeded. At this time, an overflow exception is thrown and the program execution fails.

Solution:

1. Increase the hadoop_heapsize value in the hadoop-env.sh

2. Increase the value of mapred. Child. java. opts in the mapred-site.xml (default: 200 m)

<property><name>mapred.child.java.opts</name><value>-Xmx2048m</value></property>

3. Reduce the values of mapred. tasktracker. Map. Tasks. maximumde and mapred. tasktracker. Reduce. Tasks. Maximum in the mapred-site.xml

<property><name>mapred.tasktracker.map.tasks.maximum</name><value>15</value></property>

 

Hadoop development problems: reduce getting stuck

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.