JVM Intermittent Crash Analysis

1 Problem Description

A service has two machines, every few days will alarm load high, a start to see the detection of GC time jitter is very large, thought to have occurred FULLGC caused the lag without attention, and then log on to the online machine view logs found to be a JVM crash caused the service restart to cause GC time jitter. In one day, for example, the service has a JVM crash at 7 o'clock in the morning and 10 o'clock in the morning, and if a crash at the same time will result in a line outage, the consequences can be disastrous.

2 Problem Analysis

The crash log shows that a JVM crash occurs when the flag clears the root path.

Search for this bug, found to be a known bug in the JVM,https://bugs.openjdk.java.net/browse/JDK-8020236, This bug was found in both 1.6 and 1.7, but was not fixed because of the difficulty of reproducing.

Someone has the same problem as US (http://hllvm.group.iteye.com/group/topic/43404), he passed the pressure test found when "parallelcmsthreads > Parallelgcthreads"Causes this crash, and when" Parallelcmsthreads <= parallelgcthreads "The problem is no longer reproduced. The "parallelcmsthreads > Parallelgcthreads" issue is also in the JVM bug list (https://bugs.openjdk.java.net/browse/ JDK-6668573), this bug has been given the solution of the idea is to set Parallelcmsthreads to <=parallelgcthreads.

3 workaround

Looking at the JVM parameter configuration of our Junglepoi-service service, we found that Parallelcmsthreads was set to 4 and Parallelgcthreads was not set. By default parallelgcthreads = (Ncpus <= 8)? Ncpus:3 + ((Ncpus * 5)/8), where Ncpus is the machine's number of cores due to the machine where the Junglepoi-service service resides configured for 2-core 4G, so parallelgcthreads=2 by default, Parallelcmsthreads > Parallelgcthreads at this time.

WORKAROUND: 1) Set Parallelcmsthreads to 2 or 1;2) or do not set parallelcmsthreads, parallelcmsthreads = (parallelgcthreads + 3) by default 4, if the default parallelcmsthreads= (2+3)/4=1 is not set.

We set the Parallelcmsthreads to 2 and did not reproduce the JVM crash exception for two days on line, followed by continuous observation.

4 revelation

It is not possible to copy the JVM parameter configuration of other items easily, and it needs to combine the characteristics of the project, the machine environment and other information to synthesize the configuration.

JVM Intermittent Crash Analysis