Research on the high VIRT virtual memory occupied by Java processes, javavirt

Research on the high VIRT virtual memory occupied by Java processes, javavirt

 

1. symptom

Recently, we found that the VIRT virtual memory usage of java 8 processes on the online machine reaches 50 GB +, as shown in:

void *mem = mmap(0, 4ul * 1024ul * 1024ul * 1024ul * 1024ul, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);

When mmap is used and the MAP_NORESERVE flag is set, the actual physical memory and swap space are not required. Therefore, the above Code can be seen in the top that 4096 gb virt virtual memory is used, which of course is impossible. It only indicates that 4096 GB of address space is used.

4. why are so many address spaces used?

Why does the Java program use so many address spaces? Use "pmap-x" to view the information:

...

00007ff638021000   65404       0       0 -----    [ anon ]00007ff63c000000     132      36      36 rw---    [ anon ]00007ff63c021000   65404       0       0 -----    [ anon ]00007ff640000000     132      28      28 rw---    [ anon ]00007ff640021000   65404       0       0 -----    [ anon ]00007ff644000000     132       8       8 rw---    [ anon ]00007ff644021000   65404       0       0 -----    [ anon ]00007ff648000000     184     184     184 rw---    [ anon ]00007ff64802e000   65352       0       0 -----    [ anon ]00007ff64c000000     132     100     100 rw---    [ anon ]00007ff64c021000   65404       0       0 -----    [ anon ]00007ff650000000     132      56      56 rw---    [ anon ]00007ff650021000   65404       0       0 -----    [ anon ]00007ff654000000     132      16      16 rw---    [ anon ]00007ff654021000   65404       0       0 -----    [ anon ]

…

We found many strange 64 MB memory ing, which was caused by the new arena function introduced by glibc 2.10. The glibc of CentOS 6/7 is mostly 2.12/2.17, so this problem will occur. This function assigns a local arena to each thread to accelerate multi-thread execution.

The mmap () call used in arena. c In glibc is similar to the previous sample code:

    p2 = (char *)mmap(aligned_heap_area, HEAP_MAX_SIZE, PROT_NONE,                          MAP_NORESERVE | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0)

Then, only a small part of the address is mapped to the physical memory:

    mprotect(p2, size, PROT_READ | PROT_WRITE)

Therefore, in a multi-threaded program, a considerable number of 64 MB arena will be allocated. This can be controlled using the environment variable MALLOC_ARENA_MAX. In a 64-bit system, the default value is 128.

5. Java particularity

Java programs maintain their own usage of the heap, resulting in fewer calls to glibc to manage the memory. Worse, Java 8 began to use metaspace's original space to replace the permanent generation, while the metadata is stored in the local memory of the operating system. There are more threads, and each thread needs to use a little metadata, each thread is allocated a arena, and each thread is 64 MB, which will cause a huge virtual address to be allocated.

6. Conclusion

Summary:

• VIRT is high because too much address space is allocated.
• Generally, you don't need to worry too much about VIRT because you have 16 EB space to use.
• If you really need to control the use of VIRT, set the environment variable MALLOC_ARENA_MAX. For example, the recommended value of hadoop is 4 because YARN uses the VIRT value to monitor resource usage.