I. Description of the phenomenon
The recent discovery of VIRT virtual memory usage for the online machine Java 7 (OPENJDK) process has reached 50g+, as follows:
PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND 3130 Tomcat 0
9128m 1.4g 6544 S 1.3 9.0 84:14.67 java 31480 tomcat 0 7244m 983m 5528
s 0.3 6.2 34:36.95 Java 22142 tomcat 0 6787m 537m 13m S 0.7 3.4
6:26.70 Java 22097 Tomcat 0 7049m 515m 11m S 0.3 3.2 4:52.39 java 19 root 0 600m 10m 1560 S 0.0 0.1 2:50.42 salt-minion
Guess by phenomenon:
1. There may be insufficient memory, using more swap memory;
2. Java JDK version is caused;
3. Virtual machine pseudo memory resource due to the possibility of insufficient physical host memory;
Second, the problem of troubleshooting
2.1 Whether using XMS
The first thought, of course, is to use Java's-xmx to limit the use of heaps. However, with top view, there is actually not much physical memory in use.
2.2 Viewing analytics with Vmstate
[Email protected] ~]# vmstat-a-S Mprocs -----------memory---------- ---swap-- -----io---- -- system-- -----CPU-----R B swpd free inact active si so bi bo In CS us sy ID wa st0 0 bayi 1604 340 3381 0 0 1 1 6 0 0 0 0
From the above information, can be seen in fact, the actual use of resources is very little;
2.3 Troubleshooting Virtual Machine resource consumption
After viewing VMware, it is found that the above virtual machine resource usage is not abnormal.
Third, determine the cause of the problem
After consulting the data, it was discovered by accident that this is a normal phenomenon, why is it normal? Take a look at the analysis below.
GLIBC introduced in version 2.10 of the arena new feature was caused. The glibc of CentOS 6/7 is mostly 2.12/2.17, so there will be this problem. This feature assigns a local arena to each thread to speed up the execution of multithreading. Java programs are less likely to call glibc to manage memory because of their own maintenance of heap usage. What's worse is that Java 8 starts using Metaspace to replace the permanent generation, and the meta-space is stored in the operating system's local memory, which is a lot of threads, each thread has to use a bit of space, each thread is assigned a arena, each 64MB, will cause a huge virtual address is allocated.
Summarized as follows:
- Virt High is due to the allocation of too much address space.
- Generally do not care too much virt too high, because you have more space to use.
- If you really need to control the use of Virt, set the environment variable Malloc_arena_max, such as the Hadoop recommended value of 4, because yarn uses virt values to monitor resource usage.
Reference Link:https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_ may_show_excessive_virtual_memory_usage?lang=en
Research on the problem of the Java process occupying VIRT virtual memory super high