The perm area of the JVM continues to grow resulting in an oom problem record

Find a problem

First use the Jstack to see if the line stacks is normal, after confirming normal with Jmap view (because the online openjdk, need to install Debuginfo package) in the heap snapshot situation. Jmap Some commands may cause the Java process to hang, especially if jmap-permstat causes STW and the program cannot respond. It is recommended that the Jmap command be isolated from the online environment.

A class object that uses Jmap-permstat to discover a large number of dead states, where class is Groovy/lang/groovyclassloader$innerloader.

Class_loader Classes bytes Parent_loader alive? Type <bootstrap> 2801 17853536 null live <internal> 0x0000000781d20040 279464 0x0000000 77b328568 dead groovy/lang/groovyclassloader$innerloader@0x00000007e3f853a8 0x0000000793e8ad28 279464 0x00000007 7b328568 dead groovy/lang/groovyclassloader$innerloader@0x00000007e3f853a8 0x000000078e3106f8 279464 0x000000077 b328568 dead groovy/lang/groovyclassloader$innerloader@0x00000007e3f853a8 0x000000077bf13df0 1 3072 0x0000000778 325b10 dead Sun/reflect/delegatingclassloader@0x00000007e005c4c8 0x000000079ed982d8 279464 0x000000077b328568 D EAD groovy/lang/groovyclassloader$innerloader@0x00000007e3f853a8 0x000000079d4954c0 279464 0x000000077b328568 de Ad groovy/lang/groovyclassloader$innerloader@0x00000007e3f853a8 0x000000077a3df5c8 1 3080 0x0000000778325b10 DEA  D Sun/reflect/delegatingclassloader@0x00000007e005c4c8 0x00000007ae218838 20 2794640x000000077b328568 dead groovy/lang/groovyclassloader$innerloader@0x00000007e3f853a8 0x000000077a441f58 1 3072 0 X0000000778325b10 dead Sun/reflect/delegatingclassloader@0x00000007e005c4c8 0x000000078c6ea450 279464 0x00000007 7b328568 dead groovy/lang/groovyclassloader$innerloader@0x00000007e3f853a8 0x000000077a3f9718 1 1896 0x000000077  8325B10 dead sun/reflect/delegatingclassloader@0x00000007e005c4c8 0x000000077a3f5a58 1 3072 0X0000000778325B10 Dead sun/reflect/delegatingclassloader@0x00000007e005c4c8 ... total = 10414 180017 2395296248 N/a alive=1    
 , dead=10413 N/A

There was a preliminary suspicion that the use of groovy scripts was problematic. Using the full-text search program Groovy information in Ideaj, we found that groovy checksums were used in 2 classes. One of the classes is recently added, which is suspected to be a problem with this class of validation.

    @NotNull (when = "Groovy:_this.seatcode = = null")
    @NotBlank
    private String customerId;

    @NotNull (when = "Groovy:_this.customerid = = null")
    @NotBlank
    private String seatcode;

Positioning Problems

The problem is in the validator method in Validatoraspect. Each check interface parameter instantiates a Net.sf.oval.Validator object. This is not necessary. The reason is: 1. First Net.sf.oval.Validator is thread-safe, regardless of thread safety, 2.net.sf.oval.validator objects are heavier, and each instantiation wastes a lot of memory resources; 3. Net.sf.oval.Validator The groovy script is executed at the time of the test, Threadscriptcache caches the, and if the instance is regenerated each time it causes the cache to fail.

class<? Extends validatoradapter> VDA = P.adapter ();
            If an adapter is not specified, the default is to use the Oval validation object
            if (Vda.getname (). Equals (ValidatorAdapter.class.getName ())) {
                if (o! = null) {// When the validation object is not NULL, use the Oval validation framework to verify
                    net.sf.oval.Validator Validator = new Net.sf.oval.Validator ();
                    list<constraintviolation> ret = validator.validate (o);

...

Groovy script generates class entry code. Because the Net.sf.oval.Validator instance is newly generated each time the check is made, the cache Scriptcache is regenerated every time, and the cache is invalidated, and the groovy script is re-parsed each time. Static variables Groovy_shell each time the groovy script is parsed, the new class loads into the perm area, causing the problem of oom to occur.

public class Expressionlanguagegroovyimpl implements Expressionlanguage {private static final log log = Log.getlog (Ex

    Pressionlanguagegroovyimpl.class);

    private static final Groovyshell Groovy_shell = new Groovyshell (); Private final threadlocalobjectcache<string, script> threadscriptcache = new threadlocalobjectcache<string,

    Script> ();
    Public Object Evaluate (final String expression, Final map<string,? > Values) throws Expressionevaluationexception
            {try {final objectcache<string, script> scriptcache = Threadscriptcache.get ();
            Script script = scriptcache.get (expression);
                if (script = = null) {script = Groovy_shell.parse (expression);
            Scriptcache.put (expression, script);
            Final binding binding = new binding (); For (final entry<string,? > Entry:values.entrySet ()) {Binding. SetVariable (Entry.getkey (), Entry.getvalue ());
            } log.debug ("Evaluating Groovy expression: {1}", expression);
            Script.setbinding (binding);
        return Script.run ();  } catch (Final Exception ex) {throw new Expressionevaluationexception ("Evaluating script with
        Groovy failed. ", ex);
 }
    }

...

Why no Groory script-generated class was not reclaimed by GC.
Because Groovy_shell is static, this is definitely not GC-recoverable. Groovy_shell class information is cached every time the parse is executed

    private static final Groovyshell Groovy_shell = new Groovyshell ();

Groovyclassloader is cached in Sourcecache when Parseclass, and the cached key is the name of the groovy script, which is generated differently each time. So the class is regenerated every time, and this is done to dynamically execute groovy's class. The potential problem is that the class will be loaded indefinitely into the perm area of the virtual machine.

public class Groovyclassloader extends URLClassLoader {public class Parseclass (Groovycodesource Codesource, Boolean sho Uldcachesource) throws Compilationfailedexception {synchronized (sourcecache) {Class answer = (class
            ) Sourcecache.get (Codesource.getname ());

            if (answer! = null) return answer;
            was neither already loaded nor compiling, so compile and add to//cache.
            Compilationunit unit = createcompilationunit (config, Codesource.getcodesource ());
            Sourceunit su = null; if (codesource.getfile () = = null) {su = Unit.addsource (Codesource.getname (), Codesource.getinputstream ())
            ;
            } else {su = Unit.addsource (Codesource.getfile ());
            } classcollector collector = Createcollector (unit, SU);
            Unit.setclassgencallback (collector);
            int goalphase = phases.class_generation; if (config! = null &&amP
            Config.gettargetdirectory () = null) Goalphase = Phases.output;

            Unit.compile (goalphase);
            Answer = Collector.generatedclass; for (Iterator iter = collector.getloadedclasses (). Iterator (); Iter.hasnext ();)
                {Class Clazz = (Class) Iter.next ();
            Setclasscacheentry (Clazz);
            } if (Shouldcachesource) Sourcecache.put (Codesource.getname (), answer);
        return answer; }
    }

The Codesource.getname () above gets the name of the script. The script name is generated in Groovy_shell, and each time the name is generated is different.

    Protected synchronized String Generatescriptname () {
        return "Script" + (++counter) + ". Groovy";
    }

solve the problem

Static instantiation of Net.sf.oval.Validator

    private static final Net.sf.oval.Validator Validator = new Net.sf.oval.Validator ();

Thinking Questions

Now the architecture is mostly SOA or microservices architecture, services through RPC calls are mostly stateless, generally the situation of oom is relatively small. Most oom reasons unreasonable use of the introduced third-party middleware or third-party jar packages. As subsequent business volumes increase, more attention and research is needed to introduce third-party middleware or third-party jar packages.