Research on the problem of the Java process occupying VIRT virtual memory super high

I. Description of the phenomenon

The recent discovery of VIRT virtual memory usage for the online machine Java 7 (OPENJDK) process has reached 50g+, as follows:

PID   USER PR NI     VIRT RES    SHR     S%cpu%MEM  time+  COMMAND 3130  Tomcat  0
   9128m 1.4g   6544    S  1.3 9.0 84:14.67 java 31480 tomcat  0    7244m 983m   5528
   s 0.3 6.2  34:36.95 Java 22142 tomcat  0    6787m   537m   13m     S 0.7 3.4 
   
    6:26.70 Java 22097 Tomcat  0    7049m   515m   11m   S  0.3 3.2  4:52.39 java 19  root  0    600m    10m    1560  S  0.0 0.1  2:50.42 salt-minion
   

Guess by phenomenon:

1. There may be insufficient memory, using more swap memory;

2. Java JDK version is caused;

3. Virtual machine pseudo memory resource due to the possibility of insufficient physical host memory;

Second, the problem of troubleshooting

2.1 Whether using XMS

The first thought, of course, is to use Java's-xmx to limit the use of heaps. However, with top view, there is actually not much physical memory in use.

2.2 Viewing analytics with Vmstate

[Email protected] ~]# vmstat-a-S Mprocs      -----------memory----------    ---swap--     -----io----    -- system--     -----CPU-----R  B swpd   free   inact  active     si   so        bi     bo       In     CS       us sy ID  wa st0 0  bayi     1604   340    3381       0    0         1       1       6       0   0  0  0

From the above information, can be seen in fact, the actual use of resources is very little;

2.3 Troubleshooting Virtual Machine resource consumption

After viewing VMware, it is found that the above virtual machine resource usage is not abnormal.

Third, determine the cause of the problem

After consulting the data, it was discovered by accident that this is a normal phenomenon, why is it normal? Take a look at the analysis below.

GLIBC introduced in version 2.10 of the arena new feature was caused. The glibc of CentOS 6/7 is mostly 2.12/2.17, so there will be this problem. This feature assigns a local arena to each thread to speed up the execution of multithreading. Java programs are less likely to call glibc to manage memory because of their own maintenance of heap usage. What's worse is that Java 8 starts using Metaspace to replace the permanent generation, and the meta-space is stored in the operating system's local memory, which is a lot of threads, each thread has to use a bit of space, each thread is assigned a arena, each 64MB, will cause a huge virtual address is allocated.

Summarized as follows:

• Virt High is due to the allocation of too much address space.
• Generally do not care too much virt too high, because you have more space to use.
• If you really need to control the use of Virt, set the environment variable Malloc_arena_max, such as the Hadoop recommended value of 4, because yarn uses virt values to monitor resource usage.

Reference Link:https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_ may_show_excessive_virtual_memory_usage?lang=en

　

Research on the problem of the Java process occupying VIRT virtual memory super high