Remember once PermGen the process of continuous growth

Objective

The cause of this problem is a permgen continuous growth of the alarm, and the solution of the problem is PermGen, class loading, Spring, JDK and other knowledge fusion.

This problem, from the occurrence to the final solution, has undergone a long time, the process, there are engineering methods, JVM optimization, there is source tracking, thinking collision, finally solve the problem, found that the spring 4.1.1.RELEASE bug, Since the 4.2.4, this bug has been fixed, the problem of the resolution, for our low to hight process, the deep benefit.

The solution to the problem is the result of the co-operation of a number of colleagues, and I will summarize it again in order to deepen my understanding and benefit the readers.

Background

SRE to jvm.memory.perm.used.percent increase alarm, so, began to constantly receive the alarm.

After the alarm, we use JSTAT-GCCAUSE-H10 PID 1000, monitoring process, found that PermGen full, and was FULLGC recycled, next, we need to find out the reason for the continued growth of PermGen.

Falcon Monitoring

Permanent generation

Gc

Load Class

Preliminary investigation of the cause

Traceclassloading

What causes the PermGen to continue to grow, to guess, to use some way to help locate the problem.

Adding-xx:+traceclassloading to the application startup parameters, each loading a class, loads a class, so we see:

[Loaded sun.reflect.GeneratedSerializationConstructorAccessor10037 from __jvm_defineclass__] [Loaded sun.reflect.GeneratedSerializationConstructorAccessor10038 from __jvm_defineclass__]

MAT

This also has a way of looking at the class, first through the jmap-dump:format=b,file=xxx PID dump heap, and then through the Mat tool to view, such as.

Help google,http://rednaxelafx.iteye.com/blog/548536, find the following code and log match.

BTrace

What exactly is called this code, through the Btrace call stack, to locate the problem generated business code. The Btrace code is as follows.

@BTracepublic class Classloadtester {         @OnMethod (clazz= "Java.lang.reflect.Constructor", method= " Acquireconstructoraccessor ") public    static void DefineClass (@ProbeClassName String probeclass, @ProbeMethodName String probemethod) {        print (Strings.strcat ("entered", Probeclass));        println (Strings.strcat (".", Probemethod));        Jstack ();        println ("==========================");}    }

The reason for the initial finding, is a applicationcontext.getbean operation, from the spring container to take out a prototype class, because this class is designed to be stateful, so can not use singleton.

The reason seems to be found, but there are many questions, prototype class multiple instances of the problem, there will be more than one class is generated, if so, it is not all the use of the spring container storage Prototpe class scene, there will be problems, and, Do we not see the classes that continue to load our business?

These questions were shelved, want to business class non-state transformation, found that the workload is very large, so temporarily shelved.

JVM Optimizations

The old alarm is not a thing, two JVM parameters are found:

-xx:+cmspermgensweepingenabled

-xx:cmsinitiatingpermoccupancyfraction=70

When the CMS takes over the perm,70%, it collects garbage and avoids full FULLGC.

Through the diagram, we can see that between PermGen 90%~100%, the full GC was executed, recovering the new generation, the old age, and the permanent generation.

Increased the PermGen parameter, in 70% of the time, garbage collection, recovery of the new generation, permanent generation, will not recover the old age, such as.

PermGen 90% Alarm, I 70% will give you recycling, the leadership will not receive the alarm, this actually did not solve the problem, just astray:) Just shut down the alarm a little stronger.

Cause further troubleshooting

Have doubts about programming-related knowledge, write a code to verify, is a good way.

So, a colleague began to write a code very similar to the business code, but a lot simpler, easy to change, debugging, execution.

Most importantly, with simple code, we can navigate to what line of code is behind the problem, so we can see why there is a problem after this line of code.

Look at the code call chain

1. Applicationcontext.getbean ("Errorservice", Errorservice.class)
   
2. Abstractapplicationcontext.getbean
   
3. Abstractrefreshableapplicationcontext.getbeanfactory
   
4. Abstractbeanfactory.getbean, Dogetbean
   
5. Abstractautowirecapablebeanfactory.createbean, Docreatebean, Createbeaninstance, InstantiateBean, Applybeanpostprocessorsafterinitialization
   
6. Cglibaopproxy.getproxy
   
7. Objenesiscglibaopproxy.createproxyclassandinstance
   
8. Objenesisbaseget.newinstance, Getinstantiatorof
   
9. Stdinstantiatorstrategy.newinstantiatorof
   
10. Nativemethodaccessorimpl.invoke
    
11. Delegatingmethodaccessorimpl.invoke
12. Method.invoke
    
13. Nativemethodaccessorimpl.invoke
    
14. Methodaccessorgenerator.generate
    
15. Classdefiner.defineclass

Say a few key codes

1. After using AOP injection on prototype labeled classes, does spring generate a new class each time it generates an AOP proxy?

As we can see, enhancer calls the Createclass () method every time, but when the class is generated, the cache is used to ensure that the same class loader, with the same enhanced configuration as a class, generates only one class.

2, the code that produces the problem

3. The root cause of the problem

Through the above, we see that this class "sun.reflect.GeneratedSerializationConstructorAccessor10037" is loaded in the process of objenesis to help us build the object.

Query objenesis related data, originally it is spring used to instantiate the class, use it to instantiate the class has many advantages, you can see the official documents.

Do you want to instantiate the class with objenesis and load the class every time? Continue to see the official note, you can see:

The original official recommendation of the use of a single case, and Spring4.1.1 in the wrong, non-singleton; Cglibaopproxy is generated by proxyfactory, so each time it will be new, so objenesis will be new. Try to use a later version of spring and find that spring has fixed the bug and changed it to static from 4.2.4.

Emphasize

This problem is caused by a bug in Objenesiscglibaopproxy, and the bug has been fixed at the beginning of Spring4.2.4.

Under what circumstances will we step on this bug?

When the service is declared as prototype, and there are annotations on the method, because only this scenario is used to objenesiscglibaopproxy.

Code Demo

Https://github.com/pumadong/cl-roadshow/tree/master/roadshow-app/src/main/java/com/cl/roadshow/spring/perm

Another 15-Time code optimization problem

In the debug process, we found another phenomenon, as follows:

That is, whenever a class is removed 15 times from the Sping container, the 16th time it is taken out, it will also load such a class, why?

Look at this post: http://rednaxelafx.iteye.com/blog/548536.

Summarize

This problem is mainly through two stages:

The first stage, using engineering methods, such as Falcon Monitoring, Jstack, Jmap, Mat, traceclassloading, BTrace, locate the problem;

The second stage, in the case of not finding the root cause, to simplify, through a simple demo code, reproduce the problem, and through the detailed debug tracking, find Spring 4.1.1 in "Objenesiscglibaopproxy", using "OBJENESISSTD" A bug when instantiating a class.

The first phase focuses on the engineering approach, the steps to analyze the problem, and the second stage is the emphasis on code analysis, which is progressively positioned to the problem code via debug.

The inference that should be made during the process:

1, "Loaded sun.reflect.GeneratedSerializationConstructorAccessor10037 from __jvm_defineclass__", this hint is very obvious, Although the PermGen is full, but is filled by this class, is not our business class, so we should not suspect that our business class has been enhanced to produce multiple, because no our business class is enhanced and be loaded prompt, so, just find out the reason for this class loading is good;

2, Btrace has helped us to locate the call stack, so if you want to fundamentally solve the problem, write the test code to reproduce the problem as soon as possible, the error does not reproduce, can not be said to find the root cause, then if the production changes, it is likely to produce the wrong solution.

Reference

Http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/70e3553d9d6e/src/share/classes/sun/reflect/MethodAccessorGenerator.java

http://rednaxelafx.iteye.com/blog/548536

Http://objenesis.org/tutorial.html

http://objenesis.org/

Remember once PermGen the process of continuous growth