Remember the Java Core dump analysis process

#背景提要

It's been a long time since you deployed the code yourself, and the command line's intimacy is getting lower. Released the keyboard, accustomed to the mouse operation of the Windows environment. Sudden practical deployment is also good.

Often at the time of deployment, OPS students are blind to [hs_err_pid] files. As everyone knows, this is the Java Virtual machine crash log.

#这次是如何分析问题的? First, view Log Header file

## A fatal error has been detected by the Java Runtime Environment:##  SIGSEGV (0xb) at pc=0x00007f90e71183a8, pid=6167, tid=0x00007f8fab3e6700## JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01)# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)# Problematic frame:# C  [libc.so.6+0x1503a8]  __memmove_ssse3_back+0x138## Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again## If you would like to submit a bug report, please visit:#   http://bugreport.java.com/bugreport/crash.jsp# The crash happened outside the Java Virtual Machine in native code.# See problematic frame for where to report the bug.#

The log header file contains summary information that outlines the cause of the crash.

The crash happened outside the Java Virtual Machine in native code.

It is obvious that this crash is due to JNI.

Second, the next view thread information during crash

---------------T H R E A D---------------Current thread (0x00007f90cc03f800): Javathread "Restartedmain" [_thread_in_ Native, id=23577, Stack (0x00007f8fab2e6000,0x00007f8fab3e7000)]siginfo:si_signo:11 (SIGSEGV), Si_code:1 (segv_ Maperr), Si_addr:0x00007f90e7be5000registers:rax=0x00007f8fcc04e4f2, Rbx=0x00007f90e7be34e2, RCX= 0X00007F8FCC04E4F0, Rdx=0x00000000000013d5rsp=0x00007f8fab3e2c68, rbp=0x00007f8fab3e2cc0, RSI=0X00007F90E7BE4FF0, Rdi=0x00007f8fcc050000r8 =0x00007f8fcc04e4f2, R9 =0x0000000000002f84, r10=0x0000000000000070, R11= 0x00007f90e714c6d0r12=0x00007f90e7bdd085, r13=0x00007f8fcc014390, r14=0x0000000000002f63, R15= 0x00007f90e021f5e0rip=0x00007f90e71183a8, eflags=0x0000000000010202, csgsfs=0x0000000000000033, ERR=   0x0000000000000004 Trapno=0x000000000000000etop of Stacks: (sp=0x00007f8fab3e2c68) 0x00007f8fab3e2c68:   00007f90e5065c09 00007f90d10469600x00007f8fab3e2c78:6863639000000000 000000000000642f0x00007f8fab3e2c88: 00000000014b502f 00007f8fcc0143900x00007f8fab3e2c98:00007f90e021f7d0 00007f900c01da900x00007f8fab3e2ca8:00007f90e021f5e0 000000004d79b2100x00007f8fab3e2cb8:000   000000000000d 00007f8fab3e2d200x00007f8fab3e2cc8:00007f90e506618f 00007F90CC03F8000X00007F8FAB3E2CD8:   01007F902466F0C0 00007f8fab3e2d400x00007f8fab3e2ce8:00007f8fab3e2d4e 00007f8fab3e2d4c0x00007f8fab3e2cf8:   00007F8FAB3E2D40 000000000000000d0x00007f8fab3e2d08:00007f90cc03f9f8 00007f8fab3e31980x00007f8fab3e2d18:   0000000000000001 00007f8fab3e31800x00007f8fab3e2d28:00007f90e5057ec0 00007f8fab3e2d700x00007f8fab3e2d38:   00007f90e021f5e0 632f63642f6d6f630x00007f8fab3e2d48:0000002f6e696168 00007f90bb0423d80x00007f8fab3e2d58:   00007f90cc03f800 00007f8fab3e2db00x00007f8fab3e2d68:00007f8fab3e2e30 00007f90bb0423d80x00007f8fab3e2d78:   00007f90cc03f800 00007f8fab3e2db00x00007f8fab3e2d88:00007f90cc02e850 000000076f6d39100x00007f8fab3e2d98: 00007f90bb0423d8 00007f90cc03d8000x00007f8fab3e2da8:00007f90cc03f800 00007f8fab3e2f400x00007f8fab3e2db8:00007f90e66a739f 00007f90d2ad3d500x00007f8fab3e2dc8:0000000000000000 00007F8FAB3E2E200X00007F8FAB3E2DD8:   00007F8FAB3E2EA0 00007f8fab3e2f700x00007f8fab3e2de8:00007f8fab3e2ff0 00007f90cc03f9f80x00007f8fab3e2df8:   00007f90e66c64d0 00007F9004028A700X00007F8FAB3E2E08:00007F9004028A80 00007f9004028e580x00007f8fab3e2e18:   00000000000003d8 00007f8fab3e35a00x00007f8fab3e2e28:000000076f6d3910 00000006c7c1da600x00007f8fab3e2e38:   00007f90d2ad4078 00007f90d2ad3d500x00007f8fab3e2e48:0000000000000000 00007f8fab3e37400x00007f8fab3e2e58: 00007f8fab3e3680 00007f90cc03f800

The name from the thread is missing JavaThread "restartedMain" . Called to restart the method thread, triggering a crash.

Third, again to see crash when thread stack information

Stack: [0x00007f8fab2e6000,0x00007f8fab3e7000], sp=0x00007f8fab3e2c68, free space=1011knative frames: (J=compiled Java code, j=interpreted, VV=VM Code, c=native code) C [libc.so.6+0x1503a8] __memmove_ssse3_back+0x138c [libzip.so+0x121  8f] ZIP_GETENTRY2+0XFFC [LIBZIP.SO+0X3EC0] java_java_util_zip_zipfile_getentry+0xf0j 19 Java.util.zip.ZipFile.getEntry (J[BZ) J (0 bytes) @ 0x00007f90d110140e [0x00007f90d1101340+0xce]j 6501 C2 Java.util.zip.ZipFile.getEntry (ljava/lang/string;) ljava/util/zip/zipentry; (101 bytes) @ 0x00007f90d1d10708 [0x00007f90d1d10380+0x388]j 11361 C2 java.util.jar.JarFile.getEntry (ljava/lang/ String;) Ljava/util/zip/zipentry; (bytes) @ 0x00007f90d13b7018 [0x00007f90d13b6fe0+0x38]j 11334 C2 Sun.misc.urlclasspath$jarloader.getresource (Ljava /lang/string; Z) Lsun/misc/resource; (bytes) @ 0x00007f90d11ba604 [0x00007f90d11ba580+0x84]j 5624 C2 Sun.misc.urlclasspath$1.next () Z (+ bytes) @ 0x00007f9 0d2299ae0 [0x00007f90d22998c0+0x220]j 29485 C2 java.net.urlclassloader$3$1.run () Ljava/lang/object; (5 bytes) @ 0x00007f90d34e1090 [0x00007f90d34e1040+0x50]v ~stubroutines::call_stubv [libjvm.so+0x693e96] JavaCalls:: Call_helper (javavalue*, methodhandle*, javacallarguments*, thread*) +0x1056v [libjvm.so+0x72d16c] JVM_DoPrivileged+ 0X27CJ 1053 java.security.AccessController.doPrivileged (ljava/security/privilegedaction; Ljava/security/accesscontrolcontext;) Ljava/lang/object; (0 bytes) @ 0x00007f90d1460095 [0x00007f90d145ffc0+0xd5]j 25745 C2 java.net.urlclassloader$3.hasmoreelements () Z (5 bytes) @ 0x00007f90d54a81a4 [0x00007f90d54a8100+0xa4]j 29484 C2 sun.misc.CompoundEnumeration.hasMoreElements () Z (5  bytes) @ 0x00007f90d4b42620 [0x00007f90d4b425a0+0x80]j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.doFindAllClassPathResources (Ljava/lang  /string;) ljava/util/set;+35j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources (ljava/lang/ String;) [lorg/springframework/core/io/resource;+19j Org.spRingframework.core.io.support.PathMatchingResourcePatternResolver.getResources (ljava/lang/string;) [lorg/  Springframework/core/io/resource;+52j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources (ljava/lang/  String;) [lorg/springframework/core/io/resource;+17j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources (ljava/lang/string;) [lorg/  Springframework/core/io/resource;+38j Org.springframework.boot.devtools.restart.ClassLoaderFilesResourcePatternResolver.getResources (ljava/lang/  String;) [lorg/springframework/core/io/resource;+13j Org.springframework.context.support.GenericApplicationContext.getResources (ljava/lang/string;) [lorg/  Springframework/core/io/resource;+18j Org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents (  ljava/lang/string;) ljava/util/set;+49j Org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan ([ljava/lang/string;) Ljava/util/set;+39j Org.springframework.context.annotation.ComponentScanAnnotationParser.parse (lorg/springframework /core/annotation/annotationattributes; ljava/lang/string;) ljava/util/set;+517j 25607 C1 Org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass (lorg/ Springframework/context/annotation/configurationclass; Lorg/springframework/context/annotation/configurationclassparser$sourceclass;) Lorg/springframework/context/ Annotation/configurationclassparser$sourceclass; (515 bytes) @ 0x00007f90d17aa93c [0x00007f90d17aa220+0x71c]j 3183 C1 Org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass (lorg/springframework  /context/annotation/configurationclass;) V (143 bytes) @ 0x00007f90d1c875ac [0x00007f90d1c870e0+0x4cc]j Org.springframework.context.annotation.ConfigurationClassParser.parse (lorg/springframework/core/type/ Annotationmetadata; ljava/lang/string;) v+10j ORG.SPRINGFRAMEWORK.CONTEXT.ANNOTATION.CONFIGURATIONCLASSPARSER.PArse (ljava/util/set;) v+66j Org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions (lorg/  Springframework/beans/factory/support/beandefinitionregistry;) v+291j Org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry (lorg/  Springframework/beans/factory/support/beandefinitionregistry;) v+107j Org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors (Ljava/util/collection;  Lorg/springframework/beans/factory/support/beandefinitionregistry;) v+28j Org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors (lorg/ Springframework/beans/factory/config/configurablelistablebeanfactory;  ljava/util/list;) v+229j Org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors (lorg/ springframework/beans/factory/config/configurablelistablebeanfactory;) v+5j Org.springframework.context.supPort.  Abstractapplicationcontext.refresh () v+28j  Org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh () v+1j  Org.springframework.boot.SpringApplication.refresh (lorg/springframework/context/applicationcontext;) v+11j Org.springframework.boot.SpringApplication.refreshContext (lorg/springframework/context/ Configurableapplicationcontext;) v+2j Org.springframework.boot.SpringApplication.run ([ljava/lang/string;) lorg/ Springframework/context/configurableapplicationcontext;+91j Org.springframework.boot.SpringApplication.run ([ Ljava/lang/object;  [ljava/lang/string;) lorg/springframework/context/configurableapplicationcontext;+9j Org.springframework.boot.SpringApplication.run (ljava/lang/object;[  ljava/lang/string;) lorg/springframework/context/configurableapplicationcontext;+9j Com.dc.chain.Application.main ([ljava/lang/string;) v+31v ~STUBROUTINES::CALL_STUBV [libjvm.so+0x693e96] Javacalls: : Call_helper (javavalue*, methodhandle*, javacallarguments*, thread*) +0x1056v [LIBJVM.SO+0X9A0447] Reflection::invoke (Instanceklasshandle, Methodhandle, Handle, bool, Objarrayhandle, BasicType, Objarrayhandle, BOOL, thread*) +0x5d7v [libjvm.so+0x9a3b57] Reflection::invoke_method (oopdesc*, Handle,  Objarrayhandle, thread*) +0x147v [libjvm.so+0x72754b] JVM_INVOKEMETHOD+0X26BJ 1920 Sun.reflect.NativeMethodAccessorImpl.invoke0 (Ljava/lang/reflect/method; Ljava/lang/object; [Ljava/lang/object;) Ljava/lang/object; (0 bytes) @ 0x00007f90d17c586d [0x00007f90d17c5780+0xed]j 12554 C2 Sun.reflect.NativeMethodAccessorImpl.invoke (ljava/ Lang/object; [Ljava/lang/object;) Ljava/lang/object; (104 bytes) @ 0x00007f90d27308f4 [0x00007f90d2730880+0x74]j 31113 C2 Org.springsource.loaded.ri.OriginalClassInvoker.invoke (ljava/lang/object;[ Ljava/lang/object;) Ljava/lang/object; (bytes) @ 0x00007f90d3671c98 [0x00007f90d3671be0+0xb8]j 25653 C2 Org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke (Ljava/lang/reflect/method; Ljava/lang/object; [Ljava/lang/object;) Ljava/lang/object; (2214 bytes) @ 0x00007f90d23f1a5c [0x00007f90d23f1380+0x6dc]j Org.springframework.boot.devtools.restart.RestartLauncher.run () v+41v ~STUBROUTINES::CALL_STUBV [libjvm.so+ 0X693E96] Javacalls::call_helper (javavalue*, methodhandle*, javacallarguments*, thread*) +0x1056V [libjvm.so+ 0X6943A1] Javacalls::call_virtual (javavalue*, Klasshandle, symbol*, symbol*, javacallarguments*, Thread*) +0x321V [ LIBJVM.SO+0X694847] Javacalls::call_virtual (javavalue*, Handle, Klasshandle, symbol*, symbol*, Thread*) +0x47V [ LIBJVM.SO+0X72FBD0] Thread_entry (javathread*, thread*) +0xa0v [libjvm.so+0xa79753] Javathread::thread_main_inner () + 0x103v [libjvm.so+0xa7989c] Javathread::run () +0X11CV [libjvm.so+0x92a338] Java_start (thread*) +0x108C [libpthread.so .0+0x7e25] Start_thread+0xc5java frames: (j=compiled Java Code, j=interpreted, VV=VM code) J Java.util.zip.ZipFile.get Entry (J[BZ) J (0 bytes) @ 0x00007f90d1101398 [0x00007f90d1101340+0x58]j 6501 C2 java.util.zip.ZipFile.getEntry (ljava/ lang/string;) ljava/uTil/zip/zipentry; (101 bytes) @ 0x00007f90d1d10708 [0x00007f90d1d10380+0x388]j 11361 C2 java.util.jar.JarFile.getEntry (ljava/lang/ String;) Ljava/util/zip/zipentry; (bytes) @ 0x00007f90d13b7018 [0x00007f90d13b6fe0+0x38]j 11334 C2 Sun.misc.urlclasspath$jarloader.getresource (Ljava /lang/string; Z) Lsun/misc/resource; (bytes) @ 0x00007f90d11ba604 [0x00007f90d11ba580+0x84]j 5624 C2 Sun.misc.urlclasspath$1.next () Z (+ bytes) @ 0x00007f9 0d2299ae0 [0x00007f90d22998c0+0x220]j 29485 C2 Java.net.urlclassloader$3$1.run () Ljava/lang/object;  (5 bytes) @ 0x00007f90d34e1090 [0x00007f90d34e1040+0x50]v ~stubroutines::call_stubj 1053 Java.security.AccessController.doPrivileged (ljava/security/privilegedaction; Ljava/security/accesscontrolcontext;) Ljava/lang/object; (0 bytes) @ 0x00007f90d1460023 [0x00007f90d145ffc0+0x63]j 25745 C2 java.net.urlclassloader$3.hasmoreelements () Z (5 bytes) @ 0x00007f90d54a81a4 [0x00007f90d54a8100+0xa4]j 29484 C2 sun.misc.CompoundEnumeration.hasMoreElements () Z (5 bytes) @ 0x00007f90d4b42620 [0x00007f90d4b425a0+0x80]j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.doFindAllClassPathResources (Ljava/lang  /string;) ljava/util/set;+35j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources (ljava/lang/  String;) [lorg/springframework/core/io/resource;+19j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources (ljava/lang/string;) [lorg/  Springframework/core/io/resource;+52j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources (ljava/lang/  String;) [lorg/springframework/core/io/resource;+17j Org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources (ljava/lang/string;) [lorg/  Springframework/core/io/resource;+38j Org.springframework.boot.devtools.restart.ClassLoaderFilesResourcePatternResolver.getResources (ljava/lang/ String;) [lorg/springframework/core/io/resource;+13j Org.springframework.context.support.  Genericapplicationcontext.getresources (ljava/lang/string;) [lorg/springframework/core/io/resource;+18j Org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents (  ljava/lang/string;) ljava/util/set;+49j Org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan ([ljava/lang/string;) ljava/util/ set;+39j Org.springframework.context.annotation.ComponentScanAnnotationParser.parse (lorg/springframework/core/ Annotation/annotationattributes; ljava/lang/string;) ljava/util/set;+517j 25607 C1 Org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass (lorg/ Springframework/context/annotation/configurationclass; Lorg/springframework/context/annotation/configurationclassparser$sourceclass;) Lorg/springframework/context/ Annotation/configurationclassparser$sourceclass; (515 bytes) @ 0x00007f90d17aa93c [0x00007f90d17aa220+0x71c]j 3183 C1 Org.springframework.context.annotation.ConfigurationClassParser.processCoNfigurationclass (Lorg/springframework/context/annotation/configurationclass;) V (143 bytes) @ 0x00007f90d1c875ac [ 0x00007f90d1c870e0+0x4cc]j Org.springframework.context.annotation.ConfigurationClassParser.parse (lorg/ Springframework/core/type/annotationmetadata; ljava/lang/string;) v+10j Org.springframework.context.annotation.ConfigurationClassParser.parse (ljava/util/set;) V+66j Org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions (lorg/  Springframework/beans/factory/support/beandefinitionregistry;) v+291j Org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry (lorg/  Springframework/beans/factory/support/beandefinitionregistry;) v+107j Org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors (Ljava/util/collection;  Lorg/springframework/beans/factory/support/beandefinitionregistry;) v+28j Org.springframework.context.support.PostProcessorRegistrAtiondelegate.invokebeanfactorypostprocessors (lorg/springframework/beans/factory/config/ Configurablelistablebeanfactory;  ljava/util/list;) v+229j Org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors (lorg/  springframework/beans/factory/config/configurablelistablebeanfactory;) v+5j  Org.springframework.context.support.AbstractApplicationContext.refresh () v+28j  Org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh () v+1j  Org.springframework.boot.SpringApplication.refresh (lorg/springframework/context/applicationcontext;) v+11j Org.springframework.boot.SpringApplication.refreshContext (lorg/springframework/context/ Configurableapplicationcontext;) v+2j Org.springframework.boot.SpringApplication.run ([ljava/lang/string;) lorg/ Springframework/context/configurableapplicationcontext;+91j Org.springframework.boot.SpringApplication.run ([ Ljava/lang/object; [ljava/lang/string;) lorg/springframework/context/configurableapplicationcontext;+9J Org.springframework.boot.SpringApplication.run (ljava/lang/object;[  ljava/lang/string;) lorg/springframework/context/configurableapplicationcontext;+9j  Com.dc.chain.Application.main ([ljava/lang/string;) v+31v ~stubroutines::call_stubj 1920 Sun.reflect.NativeMethodAccessorImpl.invoke0 (Ljava/lang/reflect/method; Ljava/lang/object; [Ljava/lang/object;) Ljava/lang/object; (0 bytes) @ 0x00007f90d17c57f7 [0x00007f90d17c5780+0x77]j 12554 C2 Sun.reflect.NativeMethodAccessorImpl.invoke (ljava/ Lang/object; [Ljava/lang/object;) Ljava/lang/object; (104 bytes) @ 0x00007f90d27308f4 [0x00007f90d2730880+0x74]j 31113 C2 Org.springsource.loaded.ri.OriginalClassInvoker.invoke (ljava/lang/object;[ Ljava/lang/object;) Ljava/lang/object; (bytes) @ 0x00007f90d3671c98 [0x00007f90d3671be0+0xb8]j 25653 C2 Org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke (Ljava/lang/reflect/method; Ljava/lang/object; [Ljava/lang/object;) Ljava/lang/object; (2214 bytes) @ 0x00007f90d23f1a5c [0x00007f90d23f1380+0x6dc]j orG.springframework.boot.devtools.restart.restartlauncher.run () v+41v ~stubroutines::call_stub 

Clearly see the top of the stack call as:Java_java_util_zip_ZipFile_getEntry+0xf0

Four, according to the top of the stack information query

Of course, a song search engine, find the following address:
(Crash in [libzip.so+0x1218f] zip_getentry2+0xff) [https://bugs.openjdk.java.net/browse/JDK-8175970]

By querying the results, you know that this bug has been fixed in JDK 9.
This is a fixed in JDK 9 with JDK-8145260.
Most of the times, the crashes in zip_getentry occur when the jar file being accessed have been modified/overwritten while The JVM instance was running. The following property can is used to disable the memory mapping of the central directory structure of the Jar files:
-dsun.zip.disablememorymapping=true
Please note that enabling this property would has some performance impact on the application as the JVM needs to read the Central directory structure from the jar files on the disk again and again whenever it reads a Jar file entry. So, it's best to ensure that the jar files be not modified or overwritten while the JVM had an image of them loaded.

For details refer the following:
Https://blogs.oracle.com/poonam/entry/crashes_in_zip_getentry

Remember the Java Core dump analysis process