Research on the high VIRT virtual memory occupied by Java processes, javavirt
1. symptom
Recently, we found that the VIRT virtual memory usage of java 8 processes on the online machine reaches 50 GB +, as shown in:
void *mem = mmap(0, 4ul * 1024ul * 1024ul * 1024ul * 1024ul, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
When mmap is used and the MAP_NORESERVE flag is set, the actual physical memory and swap space are not required. Therefore, the above Code can be seen in the top that 4096 gb virt virtual memory is used, which of course is impossible. It only indicates that 4096 GB of address space is used.
4. why are so many address spaces used?
Why does the Java program use so many address spaces? Use "pmap-x" to view the information:
...
00007ff638021000 65404 0 0 ----- [ anon ]00007ff63c000000 132 36 36 rw--- [ anon ]00007ff63c021000 65404 0 0 ----- [ anon ]00007ff640000000 132 28 28 rw--- [ anon ]00007ff640021000 65404 0 0 ----- [ anon ]00007ff644000000 132 8 8 rw--- [ anon ]00007ff644021000 65404 0 0 ----- [ anon ]00007ff648000000 184 184 184 rw--- [ anon ]00007ff64802e000 65352 0 0 ----- [ anon ]00007ff64c000000 132 100 100 rw--- [ anon ]00007ff64c021000 65404 0 0 ----- [ anon ]00007ff650000000 132 56 56 rw--- [ anon ]00007ff650021000 65404 0 0 ----- [ anon ]00007ff654000000 132 16 16 rw--- [ anon ]00007ff654021000 65404 0 0 ----- [ anon ]
…
We found many strange 64 MB memory ing, which was caused by the new arena function introduced by glibc 2.10. The glibc of CentOS 6/7 is mostly 2.12/2.17, so this problem will occur. This function assigns a local arena to each thread to accelerate multi-thread execution.
The mmap () call used in arena. c In glibc is similar to the previous sample code:
p2 = (char *)mmap(aligned_heap_area, HEAP_MAX_SIZE, PROT_NONE, MAP_NORESERVE | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0)
Then, only a small part of the address is mapped to the physical memory:
mprotect(p2, size, PROT_READ | PROT_WRITE)
Therefore, in a multi-threaded program, a considerable number of 64 MB arena will be allocated. This can be controlled using the environment variable MALLOC_ARENA_MAX. In a 64-bit system, the default value is 128.
5. Java particularity
Java programs maintain their own usage of the heap, resulting in fewer calls to glibc to manage the memory. Worse, Java 8 began to use metaspace's original space to replace the permanent generation, while the metadata is stored in the local memory of the operating system. There are more threads, and each thread needs to use a little metadata, each thread is allocated a arena, and each thread is 64 MB, which will cause a huge virtual address to be allocated.
6. Conclusion
Summary:
VIRT is high because too much address space is allocated.
Generally, you don't need to worry too much about VIRT because you have 16 EB space to use.
If you really need to control the use of VIRT, set the environment variable MALLOC_ARENA_MAX. For example, the recommended value of hadoop is 4 because YARN uses the VIRT value to monitor resource usage.