Write a JVM metaspace overflow troubleshooting

Multi-image Warning!

• Environment: System test (Windows SERVER/JRE8/TOMCAT7)
• Symptom: A few days after the application runs, the access timeout occurs and the server CPU utilization is high
• Issue log: Outofmemoryerror:metaspace
• Problem Analysis:
  • Cause Analysis: Metaspace is the memory space in which the JVM stores class information, and the possible causes of overflow occur:
    • Metaspace settings are too small to be applied
    • Application Metaspace continues to grow, exceeding metaspace limits
  • Positioning: The problem is first reported from the Devicestatusmonitortask, and this new version of the scheduled task changes the state of the synchronization device, mainly to communicate with the device status information.
• Guess:
  • Dynamic generation of proxy classes in Device status monitoring tasks, causing Metaspace to continue to consume
• Reproduce:
  • Run the app locally, add multiple available devices, and shorten the task execution interval
  • Turn on Java VISUALVM monitoring
  • Limit metaspace Max:-xx:maxmetaspacesize=100m

　　

From the monitoring view of JVISUALVM, we can visually see that every minute there is a spike in the number of threads and a ladder-like increase in class loading.

As the number of class loads increases, the metaspace space gradually grows from 60M to 100M, resulting in a memory overflow that causes the JVM to trigger full GC frequently and consumes a lot of CPU resources.

• Analyze-Find the problem code

Task class Run method code:

The Red box section is the new code, specifically implemented as follows:

The primary logic is to communicate the query run state with the underlying component and then update the status based on the results. Direct exclusion of the DAO operation suspects, extraction and communication parts, organized into separate test code:

Step: 1. Set JVM parameters: -verbose:class Print class load information

2. Clean up the console log and debug your code.

During the debugging process, I found that each loop will have a new class loaded:

These classes are loaded after the following line of code is run.

With the implementation of class loading information and SendRequest method, the basic confirmation problem is caused by jaxbutil processing XML and JavaBean.

Continue debugging the analysis and discover that the class is dynamically loaded when the Jaxbcontext object is initialized, and that jaxbutil each call will recreate a jaxbcontext.

• Solution Solutions

The root cause of the problem has been found, the resolution of the natural clear clear ideas.

Taking into account that the JAXB tool classes already in the JDK provide XML and JavaBean, the source discovery JAXB uses the weak reference cache object to buffer the Jaxbcontext.

/** * Cache.     We don ' t want to prevent the {@link Cache#type} from Gc-ed, * Hence {@link weakreference}.    */private static volatile weakreference<cache> Cache;     /** * Obtains the {@link Jaxbcontext} from the given type, * by using the cache if possible. * * <p> * We don ' t use locks to control access to {@link #cache}, but this code * should is Thread-safe     Thanks to the immutable {@link Cache} and {@code volatile}. */private static <T> Jaxbcontext getcontext (class<t> type) throws Jaxbexception {weakreference<        cache> c = cache;            if (c!=null) {Cache d = c.get ();        if (d!=null && d.type==type) return d.context;        }//overwrite the cache cache d = new cache (type);        cache = new Weakreference<cache> (d);    return d.context; }

Combined with the actual scenario of the application, the above implementation avoids the frequent creation of Jaxbcontext in a short time. However, the weak reference cache is quickly reclaimed by GC without reference, so each scheduled task regenerates the context, and the cache object can store only one context, which may cause a context switchover during the run of the scheduled task due to other interface traffic. In summary, the implementation of JAXB does not meet the needs of current applications.

There is no ready-made solution, but to write one yourself.

caused by the creation of Jaxbcontext, it prolongs the life cycle of the object and reduces the new object. For the same class, you can use the same context object to convert to and from XML. Because of the limited number of connectors, the XML message format is not many, so maintaining a static Map<class<?>, jaxbcontext> to store the context object consumes less memory. In view of the concurrent execution of communication with the Concurrenthashmap, concurrency security is ensured with the implementation of the.

The final code is as follows:

• Result validation

Simulate the previous test code with a slight modification of the scheduled task, executed once every 10s and repeated 50 times.

Turning on the JVISUALVM monitoring view, you can clearly see that the class mount number is close to the maximum at the first cycle, and only a handful of classes are loaded in the follow-up process to prove that the scheme is feasible.

Using the modified code to run the entire project, the monitoring image after 16 hours shows that the class load count remains stable and the metaspace size is almost unchanged.

• Summarize
  • The idea of troubleshooting a problem applies to a typical JVM's permanent or meta-space overflow.
  • This paper mainly uses the idea of Singleton mode to solve the problem of resource consumption caused by creating a large number of complex objects.

In addition, the -verbose:class parameter was used to troubleshoot the oom problem caused by the repeated initialization of the WebService client generated by Apache CXF during the previous period. The client initialization process is also dynamically generated from the WSDL file and loaded, so you should use the singleton mode whenever possible using CXF client code.

Write a JVM metaspace overflow troubleshooting