[Hadoop]-Container [XXXX] is running beyond physical/virtual memory limits.

When running MapReduce, there are times when the exception message indicates that physical memory or virtual memory exceeds the limit, by default: virtual memory is 2.1 times times the physical memory. The exception information is similar to the following:

 for container_1449820132317_0013_01_000012: |-PID PPID pgrpid sessid cmd_name user_mode_time (Millis) SYSTEM_ Time (Millis) vmem_usage (BYTES) rssmem_usage (PAGES) full_cmd_line | 13044 13026 13026 13026 (Java) 4479 494 1696595968 271 631/home/hadoop/cdh5.2.4/jdk1.7.0_79/bin/java-djava.net.preferipv4stack=true -xxx

We can see that the exception information is a hint of physical memory exceeding the limit, but by looking at the code we find that the memory we end up using is not up to 1G, only 500m-. (For specific reasons) we finally found that the parameters that affected this code execution were:

Parameters	Default value	Describe
Yarn.scheduler.minimum-allocation-mb	1024	Minimum JVM configuration per container request, in Unit M. If the requested memory is less than this value, it is reset to that value.
Yarn.scheduler.maximum-allocation-mb	8192	Maximum JVM configuration per container request, in Unit M. If it is greater than this value, it is reset.
Yarn.nodemanager.resource.memory-mb	8192	Each NodeManager node prepares the highest memory configuration, Unit m
Yarn.nodemanager.vmem-pmem-ratio	2.1	The ratio between virtual memory and physical memory, which can be limits if you are prompted to do so.
Yarn.nodemanager.pmem-check-enabled	True	If the physical memory limit comparison is set to False, no size comparison is made
Yarn.nodemanager.vmem-check-enabled	False	Whether to make a virtual memory limit comparison.
Mapreduce.map.memory.mb	1024	Map memory request size, Unit m
Mapreduce.reduce.memory.mb	1024	Reduce memory request size, Unit m
Mapred.child.java.opts	-xmx200	Map/reduce Execute parameter settings, you can parameter Hadoop Mapreduce ERROR:GC overhead limit exceeded

There are several ways to resolve this exception:

The first type:

Set the yarn.nodemanager.pmem-check-enabled and yarn.nodemanager.vmem-check-enabled directly to False, then you can prevent the exception information from being generated.

The second type:

If the exception information indicates that virtual memory is not enough, then you can change the Yarn.nodemanager.vmem-pmem-ratio parameter, you can also avoid the generation of abnormal information.

The third type:

Modify the MapReduce parameter and the settings are modified as follows:

MAPREDUCE.MAP.MEMORY.MB = (twice) * YARN.SCHEDULER.MINIMUM-ALLOCATION-MB

MAPREDUCE.REDUCE.MEMORY.MB = (1~4 times) * YARN.SCHEDULER.MINIMUM-ALLOCATION-MB

1. mapred.child.java.opts =-XMXTM (t number is less than the set value of map and reduce)

2. MAPREDUCE.MAP.JAVA.OPTS=-XMX (<MAPREDUCE.MAP.MEMORY.MB) m

Mapreduce.reduce.java.opts=-xmx (<MAPREDUCE.REDUCE.MEMORY.MB) m

Summary: The JVM heap size given by the final run parameter must be less than the memory size of the map and reduce specified by the parameter, preferably 70% or less.

Hadoop source code involves the place:

1. org.apache.hadoop.mapred.MapReduceChildJVM.getChildJavaOpts

2. Org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl.MonitoringThread.run () <398-465 line > (for Memory limit judgment)

3. Org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.LaunchTransition.transition ( ) <647-658 line > (Physical memory and virtual memory size limit calculation and assignment)

Physical memory size is actually a size value of MAPREDUCE.MAP.MEMORY.MB and MAPREDUCE.REDUCE.MEMORY.MB.

[Hadoop]-Container [XXXX] is running beyond physical/virtual memory limits.