Mrv2 memory monitoring Solution

An error occurred while running a hive job online:

    Container [pid=28474,containerID=container_1411897705890_0181_01_000012] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.5 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1411897705890_0181_01_000012 :        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE        |- 28474 19508 28474 28474 (bash) 0 0 9416704 309 /bin/bash -c /usr/java/jdk1.7.0_67/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx1024m -Djava.io.tmpdir=/data/yarn/local/usercache/hadoop/appcache/application_1411897705890_0181/container_1411897705890_0181_01_000012/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data/yarn/logs/application_1411897705890_0181/container_1411897705890_0181_01_000012 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.10.11.161 32875 attempt_1411897705890_0181_r_000000_3 12 1>/data/yarn/logs/application_1411897705890_0181/container_1411897705890_0181_01_000012/stdout 2>/data/yarn/logs/application_1411897705890_0181/container_1411897705890_0181_01_000012/stderr          |- 28481 28474 28474 28474 (java) 2356 397 1630285824 264098 /usr/java/jdk1.7.0_67/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024m -Djava.io.tmpdir=/data/yarn/local/usercache/hadoop/appcache/application_1411897705890_0181/container_1411897705890_0181_01_000012/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data/yarn/logs/application_1411897705890_0181/container_1411897705890_0181_01_000012 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.10.11.161 32875 attempt_1411897705890_0181_r_000000_3 12

According to the exception analysis, the memory usage exceeds the limit. containersmonitorimpl causes the process to kill and check the JVM memory recovery.

[GC [PSYoungGen: 241753K->16036K(306176K)] 241753K->16116K(1005568K), 0.0362550 secs] [Times: user=0.31 sys=0.05, real=0.04 secs] [GC [PSYoungGen: 210741K->4826K(306176K)] 210821K->282228K(1005568K), 0.0996080 secs] [Times: user=1.58 sys=0.15, real=0.10 secs] [GC [PSYoungGen: 194630K->4762K(306176K)] 472032K->624439K(1005568K), 0.1418910 secs] [Times: user=2.30 sys=0.14, real=0.14 secs] [Full GC [PSYoungGen: 4762K->0K(306176K)] [ParOldGen: 619677K->359650K(699392K)] 624439K->359650K(1005568K) [PSPermGen: 21635K->21622K(43520K)], 0.1742260 secs] [Times: user=0.82 sys=0.07, real=0.17 secs] [GC-- [PSYoungGen: 192581K->192581K(306176K)] 655085K->833707K(1005568K), 0.0634170 secs] [Times: user=1.08 sys=0.00, real=0.06 secs] [Full GC [PSYoungGen: 192581K->0K(306176K)] [ParOldGen: 641125K->640707K(699392K)] 833707K->640707K(1005568K) [PSPermGen: 21674K->21674K(49152K)], 0.0663990 secs] [Times: user=0.65 sys=0.05, real=0.07 secs] [Full GC [PSYoungGen: 262656K->0K(306176K)] [ParOldGen: 640709K->8142K(699392K)] 903365K->8142K(1005568K) [PSPermGen: 24649K->24647K(49152K)], 0.0662210 secs] [Times: user=0.37 sys=0.00, real=0.07 secs] [GC [PSYoungGen: 262656K->15936K(327680K)] 270798K->24078K(1027072K), 0.0175890 secs] [Times: user=0.14 sys=0.14, real=0.02 secs] Heap PSYoungGen      total 327680K, used 201250K [0x00000000eaa80000, 0x0000000100000000, 0x0000000100000000)  eden space 284160K, 65% used [0x00000000eaa80000,0x00000000f5f78b18,0x00000000fc000000)  from space 43520K, 36% used [0x00000000fd580000,0x00000000fe510010,0x0000000100000000)  to   space 22016K, 0% used [0x00000000fc000000,0x00000000fc000000,0x00000000fd580000) ParOldGen       total 699392K, used 8142K [0x00000000bff80000, 0x00000000eaa80000, 0x00000000eaa80000)  object space 699392K, 1% used [0x00000000bff80000,0x00000000c07738d8,0x00000000eaa80000) PSPermGen       total 49152K, used 24726K [0x00000000bad80000, 0x00000000bdd80000, 0x00000000bff80000)  object space 49152K, 50% used [0x00000000bad80000,0x00000000bc5a5908,0x00000000bdd80000)

There is no obvious Memory leakage or memory overflow. We should start with the heap memory. Because the object life cycle of Mr job is generally short, we try to increase the size of the new generation, allows more objects to be recycled in the new generation to improve the efficiency of collection and adjust the parameters

-Xms1024m -Xmx1024m -Xmn600m

Solve the problem. This is a problem where the physical memory exceeds the limit. Another problem is that the task is killed because the virtual memory exceeds the limit.

Parameter yarn. nodemanager. vmem-pmem-ratio indicates the virtual memory volume corresponding to the total physical memory per unit. The default value is 2.1, indicating that each 1 MB physical memory is used, A maximum of MB of virtual memory can be used. To solve the virtual memory problem, you can increase this parameter or optimize the JVM memory collection.

Finally, let's talk about the containersmonitorimpl monitoring policy. It stores the PID of each container, and the internal monitoringthread scans the running container process tree at intervals. Nodemanager reads the/proc/<pid>/STAT file to construct a process tree with the container process as the root. It monitors the memory usage of the Process tree to limit the memory usage of the task.

Private class monitoringthread extends thread {public void run () {While (true) {// obtain the process tree resourcecalculatorprocesstree ptree = ptinfo. getprocesstree (); ptree. updateprocesstree (); // obtain the memory usage of the container process tree long currentvmemusage = ptree. getcumulativevmem (); long currentpmemusage = ptree. getcumulativerssmem (); // get the memory usage of processes older than 1 in the process tree long curmemusageofagedprocesses = ptree. getcumulativevmem (1); long currssmemusageofagedprocesses = ptree. getcumulativerssmem (1); long vmemlimit = ptinfo. getvmemlimit (); long pmemlimit = ptinfo. getpmemlimit (); Boolean ismemoryoverlimit = false; string MSG = ""; // if all processes in a container process tree (older than 0) if the total memory exceeds two times of the set maximum or the total memory size of processes older than 1 exceeds the set maximum, the container is killed if (isvmemcheckenabled () & isprocesstreeoverlimit (containerid. tostring (), currentvmemusage, quota, vmemlimit) {MSG = formaterrormessage ("virtual", currentvmemusage, vmemlimit, currentpmemusage, pmemlimit, PID, containerid, ptree); Limit = true ;} else if (ispmemcheckenabled () & isprocesstreeoverlimit (containerid. tostring (), currentpmemusage, limit, pmemlimit) {MSG = formaterrormessage ("physical", currentvmemusage, vmemlimit, currentpmemusage, pmemlimit, PID, containerid, ptree ); ismemoryoverlimit = true ;}}}

In some cases, not the heap memory overflow of a JVM process can cause the task to be killed. You need to adjust the corresponding parameters, and the heap memory is not as large as possible, it is also important to adjust the proportion of each generation.

This article is from the "Lotso blog" blog, please be sure to keep this source http://lotso.blog.51cto.com/3681673/1567548

Mrv2 memory monitoring Solution