FULLGC issues caused by groovy scripts

This is caused by an online problem:

Background:
The groovy engine embedded in the app dynamically executes the incoming expression and returns the result of the execution
Online questions:

• The discovery of the FULLGC of the machine began to skyrocket at some point and continued;
• Log on to the machine, with jstat-gcutil command observation, found that the perm area has been 100%,FULLGC can not be recycled;
• Dump the memory of this machine for analysis;
• In Class View, a large number of groovy.lang.groovyclassloader$innerloader are found;
• A lot of groovy's innerloader are also seen in the ClassLoader view;

• Can basically locate the problem at the loading point of the groovy script;

  Preliminary analysis of the problem:

Groovy executes the script every time, generates a script class object, and new one innerloader to load the object, and Innerloader and script objects cannot be reclaimed at FULLGC time, so after a period of time will perm full, Always triggers FULLGC.

So, follow the source code for groovy's compiled script:

The script-compiled entry is the parse method of Groovyshell:

public Script parse(GroovyCodeSource codeSource)    throws CompilationFailedException { return InvokerHelper.createScript(parseClass(codeSource), this.context);}

All scripts are loaded by Groovyclassloader, and each load script will generate a new innerloader to load the script, but Innerloader just inherits Groovyclassloader, loading the script, is also given to Groovyclassloader to load:

To create a new innerloader:

InnerLoader loader = (InnerLoader)AccessController.doPrivileged(new PrivilegedAction() {public GroovyClassLoader.InnerLoader run() {return new GroovyClassLoader.InnerLoader(GroovyClassLoader.this);     }   });

Innerloader Inheritance Groovyclassloader:

 public static class InnerLoader extends GroovyClassLoader {    private final GroovyClassLoader delegate;   private final long timeStamp;    public InnerLoader(GroovyClassLoader delegate) {     super();       this.delegate = delegate;      this.timeStamp = System.currentTimeMillis();   }

The class load for Innerloader is given to Groovyclassloader:

public Class loadClass(String name, boolean lookupScriptFiles, boolean preferClassOverScript, boolean resolve) throws ClassNotFoundException, CompilationFailedException {    Class c = findLoadedClass(name);   if (c != null) return c;      return this.delegate.loadClass(name, lookupScriptFiles, preferClassOverScript, resolve);     }

Class loading for Groovyclassloader:

Private Class Doparseclass (Groovycodesource codesource) {validate (Codesource);    Compilationunit unit = createcompilationunit (This.config, Codesource.getcodesource ());     Sourceunit su = null;    File File = Codesource.getfile ();    if (file = null) {su = unit.addsource (file);      } else {URL url = codesource.geturl ();       if (URL! = null) {su = Unit.addsource (URL);   } else {su = Unit.addsource (Codesource.getname (), Codesource.getscripttext ());}     } classcollector collector = Createcollector (unit, SU);    Unit.setclassgencallback (collector);   int goalphase = 7;    if (this.config! = null) && (this.config.getTargetDirectory () = null)) Goalphase = 8;    Unit.compile (goalphase);     Class answer = Collector.generatedclass;     String MainClass = Su.getast (). Getmainclassname ();       For (Object o:collector.getloadedclasses ()) {Class Clazz = (Class) O;       String clazzname = Clazz.getname ();    Definepackage (Clazzname);  Setclasscacheentry (Clazz);    if (Clazzname.equals (mainClass)) answer = Clazz; } return answer;}

The reason for using Innerloader to load scripts is described in Groovy's classloader loading principle, which is summarized for the following reasons, but in this online issue, although the script is loaded with the newly created Innerloader, the FULLGC Both script objects and Innerloader cannot be recycled:

• Since a classloader can only be loaded once for a class of the same name, if it is loaded by Groovyclassloader, then when the class C is defined in a script, another script defines a class C, The Groovyclassloader cannot be loaded.
• Because when a class's ClassLoader is GC, this class can be GC, if the class is loaded by Groovyclassloader, then only if Groovyclassloader is GC, all these classes can be GC, In the case of Innerloader, since the source code has been compiled, there is no external reference to it, except for the class it loads, so as long as the class it loads is not referenced, it and its loaded classes can be GC.

Innerloader Dependent Path:

[email protected]  [email protected]  [email protected]  [email protected]  [email protected]  

Here's the problem, the JVM satisfies the GC's condition:

The class in the JVM can be reclaimed by GC only if the following three conditions are met, that is, the class is unloaded (unload):

• All instances of the class have been GC, that is, no instance of the class exists in the JVM.
• The ClassLoader that loaded the class has already been GC.
• The Java.lang.Class object of this class is not referenced anywhere, such as the method of accessing the class from anywhere through reflection.

Check the conditions of the GC article by clause:

• Groovy compiles the script into a class named Scriptxx, which is run with reflection to generate an instance and invoke its main function, which is executed only once, and there is no other place in the application that references the class or the instance it generates.

Groovy executes the script code:

Final Groovyobject object = (groovyobject) scriptclass. newinstance ();                If (object instanceof script) {script = (script) object;                    } else {//It could just is a class, so lets wrap it in a Script//wrapper Though the bindings would be ignored script = new script () {Publi                            C Object Run () {Object args = Getbinding (). Getvariables (). Get ("args");                            Object argstopass = Empty_main_args;                            if (args! = null && args instanceof string[]) {argstopass = args;                            } object.invokemethod ("main", Argstopass);                        return null;                    }                    };           SetProperties (object, Context.getvariables ());     } 

• As already mentioned, groovy specifically compiles each script with a new innerloader to solve the GC problem, so the Innerloader should be independent and not be referenced in the application;

Only a third possibility is left:

• The class object for this category has a referenced

Further observe the dump snapshot of the memory, locate the Scriptxx class object in the object view, and then view its referenced path in the perm generation and the root path of the GC.

The class object found Scriptxxx is referenced by a hashmap, as follows:

Classcache Groovy.lang.GroovyClassLoader

Discover that there is a cache of class objects in the Groovyclassloader, further following it, and discovering that the object will be cached in the map every time the script is compiled, namely:

Setclasscacheentry (Clazz);

Confirm the cause of the problem again:

Each time groovy compiles the script, it caches the script's class object, and the next time the script is compiled, it will be read first from the cache, saving compilation time. This cached map is held by Groovyclassloader, where key is the class name of the script, and the class name of the script differs from the naming conventions of the scripts in different compilation scenarios (from the file read script/read script from the stream/read script from a string), when passing text, the class object's naming convention is:

"script" + System.currentTimeMillis() + Math.abs(text.hashCode()) + ".groovy"

Therefore, each compilation of the object name is different, will add a class object in the cache, resulting in the class object is not released, as the number of times, the compiled class object will be Perm area full.

To further prove that groovy script loading is the result of a local simulation, the state of memory and GC is tested, respectively, when the groovy script is loaded repeatedly and the normal object is loaded:

Code to load the groovy script:

public void testMemory() throws Throwable {        while (true) {            for (int i = 0; i < 10000; i++) {                testExecuteExpr();            }            Thread.sleep(1000);            System.gc();        }    }

Load the code for the normal object:

public void testCommonMemory() throws InterruptedException {    while (true) {        for (int i = 0; i < 10000; i++) {            com.alipay.baoxian.trade.util.groovy.test.Test test = new com.alipay.baoxian.trade.util.groovy.test.Test() {                public void test() {                }            };            test.test();        }        Thread.sleep(1000);    }}

After running for a while, the Java process that loaded the groovy script was crash out of Oom, while the Java process that loaded the normal object could run all the time.

Add the JVM parameters, load the unloaded information of the class and the information of the GC:
-xx:+traceclassloading
-xx:+traceclassunloading
-xx:+cmsclassunloadingenabled
-XLOGGC:*/gc.log
-xx:+printgcdetails
-xx:+printgcdatestamps

Observing the GC log, it was found that groovy runtime fullgc was almost impossible to reclaim the perm zone, while the other could be recycled normally.

Groovy's GC logs:

[Full GC 2015-03-11t20:48:23.090+0800:50.168: [cms:44997k->44997k (458752K), 0.2805613 secs] 44997k->44997k ( 517760K), [CMS perm:83966k->83966k (83968K)], 0.2806654 secs] [times:user=0.28 sys=0.00, real=0.28 secs]

Modify the code to empty the cache before each execution of the script:

shell.getClassLoader().clearCache();

Groovyclassloader provides a way to empty the cache, directly call on it, and execute again, this time the FULLGC can recover the memory properly:

[Full GC 2015-03-11t19:42:22.908+0800:143.055: [cms:218134k->33551k (458752K), 0.4226301 secs] 218134k->33551k ( 517760K), [CMS perm:83967k->25740k (83968K)], 0.4227156 secs] [times:user=0.42 sys=0.00, real=0.43 secs]

Ways to resolve this problem:

Prior to a simple performance test of groovy, explaining that groovy is time-consuming to execute is three times times the time it takes to compile. Most of the time, groovy is compiled and executed, actually in this scenario, although the script is passed in as parameters, but in fact most of the script's content is the same, so I think we should modify groovy to name the script class, so that the same script each get the same name, In groovy, there is no way to add a class object each time, and then periodically cache cleanup, remove the long-running script, and the total number of scripts under a certain limit, should be able to solve the problem of groovy's perm is full.

Reference links
Java Security Model
Example demonstration
Groovy's ClassLoader Loading principle
Explore the Java class loader in depth
Analysis of Java class loading principle
A brief analysis of Java class loader
Brief analysis of ClassLoader principle

FULLGC issues caused by groovy scripts