NM concurrentmodificationexception crash problems

Recently, the NM on the line has a crash problem, check the error log:

2014-06-19 00:01:22,308  fatalorg.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.resourcelocalizationservice:  error: shutting downjava.util.concurrentmodificationexception         at java.util.linkedlist$listitr.checkforcomodification (linkedlist.java:761)          at java.util.linkedlist$listitr.next (linkedlist.java:696)          at  Org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource.toString ( localizedresource.java:120)         at java.lang.string.valueof ( string.java:2826)         at java.lang.stringbuilder.append ( stringbuilder.java:115)         at  Org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.resourcelocalizationservice$publiclocAlizer.run (resourcelocalizationservice.java:656) 2014-06-19 00:01:22,308 info  Org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.resourcelocalizationservice: public  cache exiting2014-06-19 00:03:40,685 INFO  org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.resourcelocalizationservice:  Downloading public rsrc:{ hdfs://bipcluster/tmp/hive-hdfs/hive_2014-06-19_00-05-51_049_ 5891972191087895437/-mr-10004/a1495555-b0dc-4356-8b68-1c881012e123, 1403107405580, file, null  }2014-06-19 00:03:40,685 fatal org.apache.hadoop.yarn.event.asyncdispatcher: error  in dispatcher threadjava.util.concurrent.RejectedExecutionException         at java.util.concurrent.threadpoolexecutor$abortpolicy.rejectedexecution ( threadpoolexecutor.java:1768)         at java.util.concurRent. Threadpoolexecutor.reject (threadpoolexecutor.java:767)         at  Java.util.concurrent.ThreadPoolExecutor.execute (threadpoolexecutor.java:658)          at java.util.concurrent.executorcompletionservice.submit (ExecutorCompletionService.java :         at  org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.resourcelocalizationservice$ Publiclocalizer.addresource (resourcelocalizationservice.java:618)          at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.resourcelocalizationservice $LocalizerTracker. Handle (resourcelocalizationservice.java:514)          at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.resourcelocalizationservice$ Localizertracker.handle (resourcelocalizationservice.java:456)         at org.apache.hadoop.yarn.event.asyncdispatcher.dispatch ( asyncdispatcher.java:128)         at  Org.apache.hadoop.yarn.event.asyncdispatcher$1.run (asyncdispatcher.java:77)          at java.lang.thread.run (thread.java:662) 2014-06-19 00:03:40,685 info  Org.apache.hadoop.yarn.event.asyncdispatcher: exiting, bbye.

is a multi-threaded concurrency update problem when doing resource localize cause NM to exit abnormally
This is a Bug,bug ID:
https://issues.apache.org/jira/browse/YARN-573
Bug Description:

SHARED data structures in public Localizer and Private Localizer is not Thread safe. PublicLocalizer1) pending accessed by AddResource (part of the event handling) and run method (as a part of publiclocalizer.ru N ()). Privatelocalizer (Localizerrunner?) 1) pending accessed by AddResource (part of event handling) and Findnextresource (I.remove ()). Also Update method should is fixed. It too is sharing pending list.

There are two threads that control resource localize
Publiclocalizer and Localizerrunner, a download to control the public file, a download to control the private file, both will operate Pending,fix method is to increase synchronization, This bug has been fixed in cdh5.2.0 's yarn.
For exceptions that trigger Java.util.ConcurrentModificationException, refer to:

http://examples.javacodegeeks.com/java-basics/exceptions/ java-util-concurrentmodificationexception-how-to-handle-concurrent-modification-exception/

This article is from the "Food and Light Blog" blog, please make sure to keep this source http://caiguangguang.blog.51cto.com/1652935/1587265

NM concurrentmodificationexception crash problems