記一次Java Core Dump分析過程

標籤：ebean   handle   inter   x11   att   d3d   out   pthread   MIXED   

#背景提要

很久沒有親自動手部署代碼了，命令列的親切感越來越低。放飛了鍵盤，習慣了滑鼠操作的windows環境。冷不丁實操部署也是不錯的。

常常在部署時，營運同學對於[hs_err_pid]檔案視而不見。殊不知這是JAVA 虛擬機器崩潰日誌。

#這次是如何分析問題的？一、首先查看 日誌標頭檔

## A fatal error has been detected by the Java Runtime Environment:##  SIGSEGV (0xb) at pc=0x00007f90e71183a8, pid=6167, tid=0x00007f8fab3e6700## JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build 1.8.0_144-b01)# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64 compressed oops)# Problematic frame:# C  [libc.so.6+0x1503a8]  __memmove_ssse3_back+0x138## Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again## If you would like to submit a bug report, please visit:#   http://bugreport.java.com/bugreport/crash.jsp# The crash happened outside the Java Virtual Machine in native code.# See problematic frame for where to report the bug.#

日誌標頭檔包含概要資訊，簡述了導致crash的原因。

The crash happened outside the Java Virtual Machine in native code.

能明顯看到本次崩潰是因為JNI。

二、接下來查看 崩潰時線程資訊

---------------  T H R E A D  ---------------Current thread (0x00007f90cc03f800):  JavaThread "restartedMain" [_thread_in_native, id=23577, stack(0x00007f8fab2e6000,0x00007f8fab3e7000)]siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f90e7be5000Registers:RAX=0x00007f8fcc04e4f2, RBX=0x00007f90e7be34e2, RCX=0x00007f8fcc04e4f0, RDX=0x00000000000013d5RSP=0x00007f8fab3e2c68, RBP=0x00007f8fab3e2cc0, RSI=0x00007f90e7be4ff0, RDI=0x00007f8fcc050000R8 =0x00007f8fcc04e4f2, R9 =0x0000000000002f84, R10=0x0000000000000070, R11=0x00007f90e714c6d0R12=0x00007f90e7bdd085, R13=0x00007f8fcc014390, R14=0x0000000000002f63, R15=0x00007f90e021f5e0RIP=0x00007f90e71183a8, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004  TRAPNO=0x000000000000000eTop of Stack: (sp=0x00007f8fab3e2c68)0x00007f8fab3e2c68:   00007f90e5065c09 00007f90d10469600x00007f8fab3e2c78:   6863639000000000 000000000000642f0x00007f8fab3e2c88:   00000000014b502f 00007f8fcc0143900x00007f8fab3e2c98:   00007f90e021f7d0 00007f900c01da900x00007f8fab3e2ca8:   00007f90e021f5e0 000000004d79b2100x00007f8fab3e2cb8:   000000000000000d 00007f8fab3e2d200x00007f8fab3e2cc8:   00007f90e506618f 00007f90cc03f8000x00007f8fab3e2cd8:   01007f902466f0c0 00007f8fab3e2d400x00007f8fab3e2ce8:   00007f8fab3e2d4e 00007f8fab3e2d4c0x00007f8fab3e2cf8:   00007f8fab3e2d40 000000000000000d0x00007f8fab3e2d08:   00007f90cc03f9f8 00007f8fab3e31980x00007f8fab3e2d18:   0000000000000001 00007f8fab3e31800x00007f8fab3e2d28:   00007f90e5057ec0 00007f8fab3e2d700x00007f8fab3e2d38:   00007f90e021f5e0 632f63642f6d6f630x00007f8fab3e2d48:   0000002f6e696168 00007f90bb0423d80x00007f8fab3e2d58:   00007f90cc03f800 00007f8fab3e2db00x00007f8fab3e2d68:   00007f8fab3e2e30 00007f90bb0423d80x00007f8fab3e2d78:   00007f90cc03f800 00007f8fab3e2db00x00007f8fab3e2d88:   00007f90cc02e850 000000076f6d39100x00007f8fab3e2d98:   00007f90bb0423d8 00007f90cc03d8000x00007f8fab3e2da8:   00007f90cc03f800 00007f8fab3e2f400x00007f8fab3e2db8:   00007f90e66a739f 00007f90d2ad3d500x00007f8fab3e2dc8:   0000000000000000 00007f8fab3e2e200x00007f8fab3e2dd8:   00007f8fab3e2ea0 00007f8fab3e2f700x00007f8fab3e2de8:   00007f8fab3e2ff0 00007f90cc03f9f80x00007f8fab3e2df8:   00007f90e66c64d0 00007f9004028a700x00007f8fab3e2e08:   00007f9004028a80 00007f9004028e580x00007f8fab3e2e18:   00000000000003d8 00007f8fab3e35a00x00007f8fab3e2e28:   000000076f6d3910 00000006c7c1da600x00007f8fab3e2e38:   00007f90d2ad4078 00007f90d2ad3d500x00007f8fab3e2e48:   0000000000000000 00007f8fab3e37400x00007f8fab3e2e58:   00007f8fab3e3680 00007f90cc03f800

從線程名已經缺定了JavaThread "restartedMain"。調用重啟方法線程，觸發了崩潰。

三、再來查看 崩潰時線程棧資訊

Stack: [0x00007f8fab2e6000,0x00007f8fab3e7000],  sp=0x00007f8fab3e2c68,  free space=1011kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)C  [libc.so.6+0x1503a8]  __memmove_ssse3_back+0x138C  [libzip.so+0x1218f]  ZIP_GetEntry2+0xffC  [libzip.so+0x3ec0]  Java_java_util_zip_ZipFile_getEntry+0xf0J 19  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f90d110140e [0x00007f90d1101340+0xce]J 6501 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (101 bytes) @ 0x00007f90d1d10708 [0x00007f90d1d10380+0x388]J 11361 C2 java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (22 bytes) @ 0x00007f90d13b7018 [0x00007f90d13b6fe0+0x38]J 11334 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f90d11ba604 [0x00007f90d11ba580+0x84]J 5624 C2 sun.misc.URLClassPath$1.next()Z (63 bytes) @ 0x00007f90d2299ae0 [0x00007f90d22998c0+0x220]J 29485 C2 java.net.URLClassLoader$3$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f90d34e1090 [0x00007f90d34e1040+0x50]v  ~StubRoutines::call_stubV  [libjvm.so+0x693e96]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056V  [libjvm.so+0x72d16c]  JVM_DoPrivileged+0x27cJ 1053  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d1460095 [0x00007f90d145ffc0+0xd5]J 25745 C2 java.net.URLClassLoader$3.hasMoreElements()Z (5 bytes) @ 0x00007f90d54a81a4 [0x00007f90d54a8100+0xa4]J 29484 C2 sun.misc.CompoundEnumeration.hasMoreElements()Z (5 bytes) @ 0x00007f90d4b42620 [0x00007f90d4b425a0+0x80]j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.doFindAllClassPathResources(Ljava/lang/String;)Ljava/util/Set;+35j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+19j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+52j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+17j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+38j  org.springframework.boot.devtools.restart.ClassLoaderFilesResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+13j  org.springframework.context.support.GenericApplicationContext.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+18j  org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(Ljava/lang/String;)Ljava/util/Set;+49j  org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan([Ljava/lang/String;)Ljava/util/Set;+39j  org.springframework.context.annotation.ComponentScanAnnotationParser.parse(Lorg/springframework/core/annotation/AnnotationAttributes;Ljava/lang/String;)Ljava/util/Set;+517J 25607 C1 org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass;)Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass; (515 bytes) @ 0x00007f90d17aa93c [0x00007f90d17aa220+0x71c]J 3183 C1 org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;)V (143 bytes) @ 0x00007f90d1c875ac [0x00007f90d1c870e0+0x4cc]j  org.springframework.context.annotation.ConfigurationClassParser.parse(Lorg/springframework/core/type/AnnotationMetadata;Ljava/lang/String;)V+10j  org.springframework.context.annotation.ConfigurationClassParser.parse(Ljava/util/Set;)V+66j  org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+291j  org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+107j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(Ljava/util/Collection;Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+28j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;Ljava/util/List;)V+229j  org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;)V+5j  org.springframework.context.support.AbstractApplicationContext.refresh()V+28j  org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh()V+1j  org.springframework.boot.SpringApplication.refresh(Lorg/springframework/context/ApplicationContext;)V+11j  org.springframework.boot.SpringApplication.refreshContext(Lorg/springframework/context/ConfigurableApplicationContext;)V+2j  org.springframework.boot.SpringApplication.run([Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+91j  org.springframework.boot.SpringApplication.run([Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9j  org.springframework.boot.SpringApplication.run(Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9j  com.dc.chain.Application.main([Ljava/lang/String;)V+31v  ~StubRoutines::call_stubV  [libjvm.so+0x693e96]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056V  [libjvm.so+0x9a0447]  Reflection::invoke(instanceKlassHandle, methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*)+0x5d7V  [libjvm.so+0x9a3b57]  Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0x147V  [libjvm.so+0x72754b]  JVM_InvokeMethod+0x26bJ 1920  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d17c586d [0x00007f90d17c5780+0xed]J 12554 C2 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007f90d27308f4 [0x00007f90d2730880+0x74]J 31113 C2 org.springsource.loaded.ri.OriginalClassInvoker.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (17 bytes) @ 0x00007f90d3671c98 [0x00007f90d3671be0+0xb8]J 25653 C2 org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (2214 bytes) @ 0x00007f90d23f1a5c [0x00007f90d23f1380+0x6dc]j  org.springframework.boot.devtools.restart.RestartLauncher.run()V+41v  ~StubRoutines::call_stubV  [libjvm.so+0x693e96]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056V  [libjvm.so+0x6943a1]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321V  [libjvm.so+0x694847]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47V  [libjvm.so+0x72fbd0]  thread_entry(JavaThread*, Thread*)+0xa0V  [libjvm.so+0xa79753]  JavaThread::thread_main_inner()+0x103V  [libjvm.so+0xa7989c]  JavaThread::run()+0x11cV  [libjvm.so+0x92a338]  java_start(Thread*)+0x108C  [libpthread.so.0+0x7e25]  start_thread+0xc5Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)J 19  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f90d1101398 [0x00007f90d1101340+0x58]J 6501 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (101 bytes) @ 0x00007f90d1d10708 [0x00007f90d1d10380+0x388]J 11361 C2 java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (22 bytes) @ 0x00007f90d13b7018 [0x00007f90d13b6fe0+0x38]J 11334 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f90d11ba604 [0x00007f90d11ba580+0x84]J 5624 C2 sun.misc.URLClassPath$1.next()Z (63 bytes) @ 0x00007f90d2299ae0 [0x00007f90d22998c0+0x220]J 29485 C2 java.net.URLClassLoader$3$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f90d34e1090 [0x00007f90d34e1040+0x50]v  ~StubRoutines::call_stubJ 1053  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d1460023 [0x00007f90d145ffc0+0x63]J 25745 C2 java.net.URLClassLoader$3.hasMoreElements()Z (5 bytes) @ 0x00007f90d54a81a4 [0x00007f90d54a8100+0xa4]J 29484 C2 sun.misc.CompoundEnumeration.hasMoreElements()Z (5 bytes) @ 0x00007f90d4b42620 [0x00007f90d4b425a0+0x80]j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.doFindAllClassPathResources(Ljava/lang/String;)Ljava/util/Set;+35j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findAllClassPathResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+19j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+52j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.findPathMatchingResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+17j  org.springframework.core.io.support.PathMatchingResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+38j  org.springframework.boot.devtools.restart.ClassLoaderFilesResourcePatternResolver.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+13j  org.springframework.context.support.GenericApplicationContext.getResources(Ljava/lang/String;)[Lorg/springframework/core/io/Resource;+18j  org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(Ljava/lang/String;)Ljava/util/Set;+49j  org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan([Ljava/lang/String;)Ljava/util/Set;+39j  org.springframework.context.annotation.ComponentScanAnnotationParser.parse(Lorg/springframework/core/annotation/AnnotationAttributes;Ljava/lang/String;)Ljava/util/Set;+517J 25607 C1 org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass;)Lorg/springframework/context/annotation/ConfigurationClassParser$SourceClass; (515 bytes) @ 0x00007f90d17aa93c [0x00007f90d17aa220+0x71c]J 3183 C1 org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(Lorg/springframework/context/annotation/ConfigurationClass;)V (143 bytes) @ 0x00007f90d1c875ac [0x00007f90d1c870e0+0x4cc]j  org.springframework.context.annotation.ConfigurationClassParser.parse(Lorg/springframework/core/type/AnnotationMetadata;Ljava/lang/String;)V+10j  org.springframework.context.annotation.ConfigurationClassParser.parse(Ljava/util/Set;)V+66j  org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+291j  org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+107j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(Ljava/util/Collection;Lorg/springframework/beans/factory/support/BeanDefinitionRegistry;)V+28j  org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;Ljava/util/List;)V+229j  org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(Lorg/springframework/beans/factory/config/ConfigurableListableBeanFactory;)V+5j  org.springframework.context.support.AbstractApplicationContext.refresh()V+28j  org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh()V+1j  org.springframework.boot.SpringApplication.refresh(Lorg/springframework/context/ApplicationContext;)V+11j  org.springframework.boot.SpringApplication.refreshContext(Lorg/springframework/context/ConfigurableApplicationContext;)V+2j  org.springframework.boot.SpringApplication.run([Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+91j  org.springframework.boot.SpringApplication.run([Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9j  org.springframework.boot.SpringApplication.run(Ljava/lang/Object;[Ljava/lang/String;)Lorg/springframework/context/ConfigurableApplicationContext;+9j  com.dc.chain.Application.main([Ljava/lang/String;)V+31v  ~StubRoutines::call_stubJ 1920  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007f90d17c57f7 [0x00007f90d17c5780+0x77]J 12554 C2 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007f90d27308f4 [0x00007f90d2730880+0x74]J 31113 C2 org.springsource.loaded.ri.OriginalClassInvoker.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (17 bytes) @ 0x00007f90d3671c98 [0x00007f90d3671be0+0xb8]J 25653 C2 org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (2214 bytes) @ 0x00007f90d23f1a5c [0x00007f90d23f1380+0x6dc]j  org.springframework.boot.devtools.restart.RestartLauncher.run()V+41v  ~StubRoutines::call_stub

明顯看到棧頂調用為： Java_java_util_zip_ZipFile_getEntry+0xf0

四、根據棧頂資訊查詢

當然是用的某歌搜尋引擎，查到如下地址：
(Crash in [libzip.so+0x1218f] ZIP_GetEntry2+0xff)[https://bugs.openjdk.java.net/browse/JDK-8175970]

查詢到結果，可以知道在JDK 9中修正了這個Bug。
This is fixed in JDK 9 with JDK-8145260.
Most of the times, the crashes in ZIP_GetEntry occur when the jar file being accessed has been modified/overwritten while the JVM instance was running. The following property can be used to disable the memory mapping of the central directory structure of the Jar files:
-Dsun.zip.disableMemoryMapping=true
Please note that enabling this property would have some performance impact on the application as the JVM needs to read the central directory structure from the Jar files on the disk again and again whenever it reads a Jar file entry. So, it is best to ensure that the jar files are not modified or overwritten while the JVM has an image of them loaded.

For details refer the following :
https://blogs.oracle.com/poonam/entry/crashes_in_zip_getentry

記一次Java Core Dump分析過程