The metaspace of the FULLGC of JDK8

Objective:

As a result of the recently written program after running for a period of time after the high CPU, and then not to be used to troubleshoot, and finally to locate due to metaspace caused FULLGC, constantly fullgc and occupy a large number of CPUs resulting in the program eventually unusable. The following is the process of analysis and summary, easy to later temperature, but also hope to be able to meet the same problems of students some reference.

Memory allocation of a JVM:

Eden Survivor1 Survivor2 tenured
Tenured contains perm jdk<=7

JVM Memory Young area diagram. png

GC types are divided into: minor GC and major GC, Major 10 times times slower than minor at least

The GC, which occurs in the young (mostly survivor) area, is called the minor GC
The GC that occurs in the old (tenured) area is called the major GC

1. Description of the problem

Jstat-gcutil 26819
S0 S1 E O M CCS ygc ygct FGC fgct GCT
0.00 43.75 0.00 42.22 67.19 50.93 4955 30.970 4890 3505.049 3536.020
You can see that the value of M (Metaspace usage) is 67.19,metaspace usage of 67.19;o for 42.22,old zone usage is 42.22

top-h-P 26819
26821 AppDev 0 6864m 1.2g 13m R 87.6 7.5 53:40.18 java
26822 AppDev 0 6864m 1.2g 13m R 87.6 7.5 53:41.40 java
26823 AppDev 0 6864m 1.2g 13m R 87.6 7.5 53:43.64 java
26824 AppDev 0 6864m 1.2g 13m R 85.6 7.5 53:41.59 java
26825 AppDev 0 6864m 1.2g 13m R 85.6 7.5 53:43.82 java
26826 AppDev 0 6864m 1.2g 13m R 85.6 7.5 53:40.47 java
26827 AppDev 0 6864m 1.2g 13m R 85.6 7.5 53:45.05 java
26828 AppDev 0 6864m 1.2g 13m R 83.6 7.5 53:39.08 java
You can see that 26821 to 26828 of CPU usage is high, and 26821 is converted to 16 binary to 68C5

jstack 26819 > 26819.text
Vim 26819.text then search 68c5-68cc
GC task thread#0 (PARALLELGC) os_prio=0 tid=0x00007f0aa401e000 nid=0x68c5 runnable

GC task thread#1 (PARALLELGC) os_prio=0 tid=0x00007f0aa4020000 nid=0x68c6 runnable

GC task Thread#2 (PARALLELGC) os_prio=0 tid=0x00007f0aa4021800 nid=0x68c7 runnable

GC task Thread#3 (PARALLELGC) os_prio=0 tid=0x00007f0aa4023800 nid=0x68c8 runnable

GC task Thread#4 (PARALLELGC) os_prio=0 tid=0x00007f0aa4025800 nid=0x68c9 runnable

GC task thread#5 (PARALLELGC) os_prio=0 tid=0x00007f0aa4027000 nid=0x68ca runnable

GC task Thread#6 (PARALLELGC) os_prio=0 tid=0x00007f0aa4029000 NID=0X68CB runnable

GC task Thread#7 (PARALLELGC) os_prio=0 tid=0x00007f0aa402a800 nid=0x68cc runnable

It can be found that threads that are consistent with full GC are executing, consuming high CPU resources, and consistently, indicating that full GC conditions have been reached but the memory can not be reclaimed to consume a large amount of CPU, which makes the program unusable.

View the startup configuration parameters as follows:
-xms1000m-xmx1000m-xx:maxnewsize=256m-xx:threadstacksize=256-xx:metaspacesize=38m-xx:maxmetaspacesize=380m
The logic of the parser, the program loads a lot of jars into memory, the program is a public service, many colleagues upload jars, and the program loads the jar into the ClassLoader for analysis and preservation.

2. Problem Analysis:

Based on the triggering conditions of the jdk8 Metaspace FULLGC, the initial metaspacesize is 38m means that the first GC (based on the features of JDK 8) when the first loaded class reaches 38m G1 and CMS will be well-collected in the Metaspace area (usually with full GC). ), and the JVM dynamically adjusts the size of the metaspacesize (after the GC is adjusted).

Jdk8:metaspace
In JDK 8, classes metadata are now stored in the native heap
And this space is called Metaspace. There is some new flags added for
Metaspace in JDK 8:
-xx:metaspacesize=<nnn>
Where <NNN> is the initial amount of space (the initial
High-water-mark) allocated for class metadata (in bytes) the May induce a
Garbage collection to unload classes. The amount is approximate. After the
High-water-mark is first reached, the next High-water-mark was managed by
The garbage collector
-xx:maxmetaspacesize=<nnn>
Where <NNN> is the maximum amount of space to being allocated for class
Metadata (in bytes). This flag can is used to limit the amount of space
Allocated for class metadata. This value is approximate. By default there
is no limit set.
-xx:minmetaspacefreeratio=<nnn>
Where <NNN> is the minimum percentage of class metadata capacity
Avoid a increase in the amount of space
(High-water-mark) allocated for class metadata, that'll induce a garbage
Collection.
-xx:maxmetaspacefreeratio=<nnn>
Where <NNN> is the maximum percentage of class metadata capacity
Free after a GC to avoid a reduction in the amount of space
(High-water-mark) allocated for class metadata, that'll induce a garbage
Collection.
By default class
Metadata allocation is only limited by the amount of available native memory. We
Can use the new option maxmetaspacesize to limit the amount of native memory
Used for the class metadata. It is analogous (similar) to MaxPermSize. A garbage collection is induced to collect the dead classloaders
and classes when the class metadata usage reaches Metaspacesize (12Mbytes on
The 32bit client VM and 16Mbytes on the 32bit server VM with larger sizes on
The 64bit VMs). Set metaspacesize to a higher value to delay the induced
Garbage collections. After an induced garbage collection, the class metadata usage
needed to induce the next garbage collection could be increased.

According to this paragraph you can know:
1. The GC is triggered when metadata usage reaches metaspacesize (the default metaspacesize is 20.8m on server 64);
2.xx:minmetaspacefreeratio is used to avoid triggering a GC when the next application's idle metadata is larger than the temporarily owned idle metadata, for example, when the Metaspacesize's usage size reaches the initial value of 6m for the first set, At this time the expansion (previously done minmetaspaceexpansion and maxmetaspaceexpansion expansion, but still failed), and then after the GC, because the recovery of the memory is very small, and then calculate ((to commit memory)/ (to commit memory + already commmited memory) ==40%, (to commit memory + already commmited memory) is greater than metaspacesize then will try to do the expansion, that is, increase the threshold of triggering METASPACEGC, However, this increment is at least minmetaspaceexpansion to do, otherwise it will not increase the threshold value, this parameter is mainly to avoid triggering METASPACEGC threshold and the GC after the committed of the amount of memory is closer, so this threshold value ( Metaspacesize) is expanded to minimize the chance of the next GC.
3. The same-xx:maxmetaspacefreeratio (default 70) is used to avoid the next application of the idle metadata is very small, much less than the current free memory thereby causing the GC. The main function is to reduce unnecessary memory footprint.

jdk8 Metaspace-raised FULLGC:
Jdk8 using Metaspace instead of the perm,metaspace memory, The maximum size that is used by default is the size of the system memory, and of course you can use-xx:maxmetaspacesize to set the maximum size, which is the same as the previous Max perm size. At the same time, when we set the-xx:maxmetaspacesize parameter, we can also implement the problem of Oom caused by Max Perm.
We can achieve the famed OOM error by setting the Maxmetaspacesize argument to JVM and running of the sample program Prov IDed.
Metaspacesize Default Initial Size:
Metaspacesize (12Mbytes on the 32bit client VMs and 16Mbytes on the 32bit server VMS with larger Sizes on the 64bit VMs). The
can set the initial size we need through-xx:metaspacesize, and setting a large point can increase the time it takes to reach the full GC for a first time.

PS: Below is the process of adjusting the next parameter reboot, which is not the same as the process ID above.
JSTAT-GC 1706
s0c s1c s0u s1u EC EU OC OU MC MU ccsc ccsu ygc ygct FGC fgct GCT
31744.0 32768.0 0.0 21603.6 195584.0 192805.8 761856.0 384823.3 467712.0 309814.3 65536.0 36929.1 101 2.887 3 1.224 4.112
Analysis: MC is already commited memory, MU is currently used memory. Here is a question is whether MC is Metaspace has been a total use of memory, because this value has reached the maxmetaspacesize, and why MU is not the same as the MC I guess is due to the fragmentation of memory, here have to know the classmate can tell me under. 3 FULLGC were performed when the Maxmetaspacesize was reached. However, due to the continuous application of memory, constantly FULLGC,FULLGC can not reclaim memory, the FULLGC frequency increases a lot. In the next top-h-P 1706 View CPU can see a large number of high CPU processes, through Jstack view are in progress FULLGC.

Jmap-clstats 1706
First time: total = 131 8016 13892091 N/a alive=45, dead=86 N/A
Second time: total = 1345 37619 77242171 N/a alive=1170, dead=175 N/A
Alive's ClassLoader are basically created by themselves.
ClassLoader is constantly increasing, and each GC is not recycled ClassLoader
The class in the VM can be reclaimed by GC only if the following three conditions are met, which is the class being unloaded (unload):

• All instances of the class have been GC, that is, no instance of the class exists in the JVM.
• The ClassLoader that loaded the class has already been GC. ClassLoader is recycled all instances of the class that require all classloader are recycled.
• The Java.lang.Class object of this class is not referenced anywhere, such as the method of accessing the class from anywhere by reflection

Jcmd 1706 Gc.class_stats | awk ' {print $13} ' | sort | uniq-c | sort-nrk1 > Topclass.txt

Class.png

Classes that are loaded by custom ClassLoader are repeated several times, and the number has been increased.
See a large number of class repetitions

GC Log Analysis:

first time FULLGC:
[Heap Dump (before full GC):, 0.4032181 secs]2018-01-10t16:37:44.658+0800:21.673: [Full GC (Metadata GC Threshold) [Psyo unggen:14337k->0k (235520K)] [paroldgen:18787k->30930k (761856K)] 33125k->30930k (997376K), [Metaspace: 37827k->37827k (1083392K)], 0.1360661 secs] [times:user=0.65 sys=0.04, real=0.14 secs]
Mainly metaspace here: [metaspace:37827k->37827k (1083392K)] reached the initial value we set for 38m, and the GC did not reclaim memory. 1083392K This value is suspected to have been caused by the use of Compressedclassspacesize = 1073741824 (1024.0MB).
fourth time FULLGC:
[Heap Dump (before full GC):, 5.3642805 secs]2018-01-10t16:53:43.811+0800:980.825: [Full GC (Metadata GC Threshold) [PSY ounggen:21613k->0k (231424K)] [paroldgen:390439k->400478k (761856K)] 412053k->400478k (993280K), [Metaspace : 314108k->313262k (1458176K)], 1.2320834 secs] [times:user=7.86 sys=0.06, real=1.23 secs]
The main is metaspace here: [metaspace:314108k->313262k (1458176K)] reached our set of Minmetaspacefreeratio, and the GC has hardly reclaimed memory. 1458176K this value is Compressedclassspacesize = 1073741824 (1024.0MB) and maxmetaspacesize = 503316480 (480.0MB) and.

The following is the repetition of the frequency fullgc quickly.

3. Problem solving:

With the above foundation, we will know how to solve the problem encountered.
Summary of reasons: ClassLoader constantly create, ClassLoader constantly loaded class, the previous ClassLoader and class in the FULLGC is not recycled.

1. The program avoids creating duplicate classloader, reducing the number of classloader created.
2. Increase the size of Xx:minmetaspacefreeratio (default 40) and you can see that it is now (100-67.19).
3. Set a larger maxmetaspacesize.

jdk8metadataspace Reference:
http://www.sczyh30.com/posts/Java/jvm-metaspace/
http://blog.csdn.net/ouyang111222/article/details/53688986
http://lovestblog.cn/blog/2016/10/29/metaspace/
https://bugs.openjdk.java.net/browse/JDK-8151845
http://blog.csdn.net/ouyang111222/article/details/53688986
Https://blogs.oracle.com/poonam/about-g1-garbage-collector%2c-permanent-generation-and-metaspace
Http://zhuanlan.51cto.com/art/201706/541920.htm
http://blog.yongbin.me/2017/03/20/jaxb_metaspace_oom/

Extended read conditions for triggering GC in JDK:

Invocation of the 1,SYSTEM.GC () method
System.GC (), the invocation of this method is to recommend that the JVM perform full GC, although it is only recommended but not necessarily, but in many cases it triggers the full GC, which increases the frequency of the full GC, which increases the number of intermittent pauses. It is strongly recommended that you do not use this method, let the virtual machine to manage its own memory, by-xx:+ DISABLEEXPLICITGC to prohibit RMI calls SYSTEM.GC.
2, old generation space (old/tenured) Insufficient
The old age space is only insufficient when the new generation objects are transferred and created as large objects and large arrays, and when there is still insufficient space after executing full GC, the following error is thrown: Java.lang.OutOfMemoryError:Java heap Space In order to avoid the above two conditions caused by the full GC, tuning should try to make the object in the minor GC phase is recycled, so that the object in the Cenozoic to survive for a period of time and do not create too large objects and arrays.
3, the living area (perm) space is insufficient (jdk<=7, in the jdk8 inside is Metaspace, later will focus on the description)
The method area in the runtime data region in the JVM specification, which is also used in the Hotspot virtual machine as an immortal or immortalized zone, is stored in permanet generation for some class information, constants, static variables, and so on, when the system loads classes, When a class is reflected and a method is called, permanet generation may be full, and a fully GC will be executed if it is not configured to adopt a CMS GC. If the full GC is still not recycled, then the JVM will throw the following error message: Java.lang.OutOfMemoryError:PermGen space to avoid the perm Gen to be filled with a fully GC phenomenon, the method used to increase the perm Gen space or switch to using CMS GC.
promotion failed and concurrent mode failure appear when 4,CMS GC
For programs that use the CMS for an older GC, pay particular attention to the promotion failed and concurrent mode failure two conditions in the GC log, which may trigger full GC when both conditions occur. Promotion failed is in the minor GC, Survivor space, objects can only be placed in the old age, and at this time the old age also can not be caused; Concurrent mode failure is executing the CMS In the process of GC, there are objects to be put into the old age, and at this time the old space is insufficient (sometimes "insufficient space" is the CMS GC when the current floating garbage too much caused the temporary lack of space to trigger the full GC). The measures are: increase survivor space, older generation spaces, or reduce the rate of triggering concurrent GC (-XX:CMSINITIATINGOCCUPANCYFRACTION=70, 70%), but in JDK 5.0+, 6.0+ Version may cause the CMS to trigger sweeping action long after the remark is completed due to the bug29 of the JDK. For this condition, you can avoid this by setting the-xx:cmsmaxabortableprecleantime=5 (in MS).
5. The average size of the minor GC promoted to the old generation (Eden to S2 and S1 to S2) is greater than the remaining space in the old age.
This is a more complex trigger situation, the hotspot in order to avoid the new generation of objects promoted to the old generation of space shortage of old generation, in the minor GC, made a judgment, if the previous statistics obtained minor GC promoted to the old generation of the average size greater than the old generation of the remaining space, Then the full GC is triggered directly. For example, after the program first triggers the minor GC, there are 6MB objects promoted to the old generation, then the next time the minor GC occurs, first check whether the old generation of the remaining space is greater than 6MB, if less than 6MB, then execute full GC. When the new generation of PS GC, the way slightly different, PS GC is also checked after the minor GC, such as the first minor GC in the above example, the PS GC will check that the old generation of the remaining space is greater than 6MB, if less than, trigger the recovery of the old generation.
In addition to the 4 scenarios above, the full GC is executed one hour by default for sun JDK applications that use RMI for RPC or management. You can set the time interval for full GC execution by using-java-dsun.rmi.dgc.client.gcinterval=3600000 at startup or by-xx:+ DISABLEEXPLICITGC to disallow RMI calls to System.GC.
6. Large objects allocated in the heap
A large object is a Java object that requires a large amount of contiguous memory space, such as a very long array, which goes directly into the old age, while the old age has a lot of space left, but cannot find enough contiguous space to allocate to the current object, which triggers the JVM to perform a full GC.
To solve this problem, the CMS garbage collector provides a configurable parameter, the-xx:+usecmscompactatfullcollection switch parameter, for an extra free defragmentation process after "enjoy" the full GC service, The space debris problem is gone, but the Tim time must not be longer, and the JVM designers also provide another parameter,-xx:cmsfullgcsbeforecompaction, which is used to set the number of uncompressed full GC executions, followed by a compression.

Extended reading reference:
http://engineering.xueqiu.com/blog/2015/06/25/jvm-gc-tuning/
Http://www.cnblogs.com/redcreen/archive/2011/05/04/2037057.html
http://blog.csdn.net/chenleixing/article/details/46706039

The metaspace of the FULLGC of JDK8