Performance tuning case sharing: Reasons for JVM crash 2

3.core Dump Analysis

With the core dump file, the next thing to do is to parse the file through the command to locate the specific problem, the following three commands are mainly:

(1) First execute the GDB $JAVA _home$/bin/java core-java-16427-1325846515, and then execute BT, the output results are as follows:

Loaded symbols for/opt/.../oracle/lib/libnnz10.so
Reading symbols from/opt/.../install/oracle/10.2.0.3/lib/libociei.so ... (No debugging symbols found) ... done.
Loaded symbols for/opt/.../oracle/lib/libociei.so
Reading symbols from/usr/lib/gconv/gb18030.so ... (No debugging symbols found) ... done.
Loaded symbols for/usr/lib/gconv/gb18030.so
#0 0x4662051e in TTCRD2R ()
from/opt/.../oracle/lib/libclntsh.so.10.1
(GDB) bt
#0 0x4662051e in TTCRD2R () from/opt/.../oracle/lib/libclntsh.so.10.1
#1 0x46617212 in Ttcrbur () from/opt/.../oracle/lib/libclntsh.so.10.1
#2 0x466176e6 in Ttcbur () from/opt/.../oracle/lib/libclntsh.so.10.1
#3 0x465ba8c7 in Ttcdrv () from/opt/.../oracle/lib/libclntsh.so.10.1
#4 0x464a1bc8 in Nioqwa () from/opt/.../oracle/lib/libclntsh.so.10.1
#5 0x4630d618 in Upirtrc () from/opt/.../oracle/lib/libclntsh.so.10.1
#6 0x462825f6 in KPURCSC () from/opt/.../oracle/lib/libclntsh.so.10.1
#7 0x46236a0d in KPUEXECV8 () from/opt/.../oracle/lib/libclntsh.so.10.1
#8 0x46238ec4 in Kpuexec () from/opt/.../oracle/lib/libclntsh.so.10.1
#9 0x463121b6 in Ocistmtexecute () from/opt/.../oracle/lib/libclntsh.so.10.1
#10 0X46E725AA in Java_oracle_jdbc_driver_t2cconnection_t2cclearallapplicationcontext ()
From/opt/.../install/oracle/10.2.0.3/lib/libocijdbc10.so
#11 0x46e7580f in Java_oracle_jdbc_driver_t2cstatement_t2cdefineexecutefetch ()
From/opt/.../install/oracle/10.2.0.3/lib/libocijdbc10.so
#12 0xb125a8b8 in?? ()
#13 0x47e01bc8 in?? ()
#14 0x47d46bb0 in?? ()
#15 0x47d46c40 in?? ()
#16 0x4babf774 in?? ()
#17 0x00000000 in?? ()

With this debug command you can see what is being done when the crash is applied, here is the JDBC connection, but this does not mean that the problem is here, it is possible that this is only a result, and the cause needs further troubleshooting.

(2) The above information is not enough to locate the problem, you can use the command jstack $JAVA _home$/bin/java core-java-16427-1325846515 get the application crash when the thread stack, the output is as follows:

Thread 11586: (state = BLOCKED)
-Java.lang.Object.wait (Long) @bci =0 (Compiled frame; information may be imprecise)
-Java.lang.Object.wait () @bci =2, line=474 (Compiled frame)
-Edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingDeque.takeFirst () @bci =20, line=389 (Compiled frame)
-Edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingDeque.take () @bci =1, line=577 (Compiled frame)
-Edu.emory.mathcs.backport.java.util.concurrent.threadpoolexecutor$worker.run () @bci =18, line=674 (Compiled frame)
Error occurred during stack walking:

Thread 10924: (state = in_native)
-Oracle.jdbc.driver.T2CStatement.t2cDefineExecuteFetch (oracle.jdbc.driver.OracleStatement, long, int, int, int, int , Boolean, Boolean, byte[], int, byte, int, int, short[], int, byte[], char[], int, int, short[], byte[], int, int, Boolea N, Boolean, oracle.jdbc.driver.accessor[], byte[][][], long[], byte[], int, char[], int, short[], int) @bci =0 (interpreted Frame
-Oracle.jdbc.driver.T2CPreparedStatement.doDefineExecuteFetch () @bci =190, line=878 (interpreted frame)
-Oracle.jdbc.driver.T2CPreparedStatement.executeForRows (Boolean) @bci =34, line=760 (interpreted frame)
-Oracle.jdbc.driver.OracleStatement.executeMaybeDescribe () @bci =154, line=1062 (interpreted frame)
-Oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout () @bci =102, line=1126 (interpreted frame)
-Oracle.jdbc.driver.OraclePreparedStatement.executeInternal () @bci =94, line=3339 (interpreted frame)
-Oracle.jdbc.driver.OraclePreparedStatement.execute () @bci =17, line=3445 (interpreted frame)
-Org.jboss.resource.adapter.jdbc.CachedPreparedStatement.execute () @bci =4, line=216 (interpreted frame)
-Org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.execute () @bci =12, line=209 (Compiled frame)
-Com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery (Com.ibatis.sqlmap.engine.scope.RequestScope, Java.sql.Connection, Java.lang.String, java.lang.object[], int, int, Com.ibatis.sqlmap.engine.mapping.statement.RowHandlerCallback) @bci =135, line=180 (Compiled frame)
-Com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery ( Com.ibatis.sqlmap.engine.scope.RequestScope, Java.sql.Connection, Java.lang.String, java.lang.object[], int, int, Com.ibatis.sqlmap.engine.mapping.statement.RowHandlerCallback) @bci =15, line=205 (interpreted frame)
-Com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback ( Com.ibatis.sqlmap.engine.scope.RequestScope, Java.sql.Connection, Java.lang.Object, Java.lang.Object, Com.ibatis.sqlmap.client.event.RowHandler, int, int) @bci =169, line=173 (Compiled frame)
-Com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForObject ( Com.ibatis.sqlmap.engine.scope.RequestScope, Com.ibatis.sqlmap.engine.transaction.Transaction, Java.lang.Object, Java.lang.Object) @bci =28, line=104 (Compiled frame)
-Com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject ( Com.ibatis.sqlmap.engine.scope.SessionScope, Java.lang.String, Java.lang.Object, Java.lang.Object) @bci =58, line= 561 (Compiled frame)
-Com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject ( Com.ibatis.sqlmap.engine.scope.SessionScope, java.lang.String, Java.lang.Object) @bci =5, line=536 (interpreted frame )
-Com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject (java.lang.String, Java.lang.Object) @bci =10, line =93 (interpreted frame)
-Org.springframework.orm.ibatis.sqlmapclienttemplate$1.doinsqlmapclient ( Com.ibatis.sqlmap.client.SqlMapExecutor) @bci =9, line=273 (interpreted frame)
-Org.springframework.orm.ibatis.SqlMapClientTemplate.execute ( Org.springframework.orm.ibatis.SqlMapClientCallback) @bci =242, line=209 (Compiled frame)
-Org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject (java.lang.String, java.lang.Object) @bci = 11, line=271 (Compiled frame)
......
-Org.springframework.web.servlet.DispatcherServlet.doService (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =219, line=809 (interpreted frame)
-Org.springframework.web.servlet.FrameworkServlet.processRequest (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =10, line=571 (Compiled frame)
-Org.springframework.web.servlet.FrameworkServlet.doPost (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =3, line=511 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =139, line=717 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci = 30, line=810 (Compiled frame)
-Org.apache.catalina.core.ApplicationFilterChain.doFilter (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =101, line=173 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.invoke (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =396, line=672 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.processRequest (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =64, line=463 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.doForward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =469, line=398 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.forward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =55, line=301 (interpreted frame)

-Org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel (Java.util.Map, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =244, line=240 (interpreted Frame
-Org.springframework.web.servlet.view.AbstractView.render (Java.util.Map, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =157, line=252 (interpreted frame)
-Org.springframework.web.servlet.DispatcherServlet.render (Org.springframework.web.servlet.ModelAndView, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =225, line=1173 (Compiled frame)
-Org.springframework.web.servlet.DispatcherServlet.doService (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =219, line=809 (interpreted frame)
-Org.springframework.web.servlet.FrameworkServlet.processRequest (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =10, line=571 (Compiled frame)
-Org.springframework.web.servlet.FrameworkServlet.doPost (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =3, line=511 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =139, line=717 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci = 30, line=810 (Compiled frame)
-Org.apache.catalina.core.ApplicationFilterChain.doFilter (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =101, line=173 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.invoke (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =396, line=672 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.processRequest (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =64, line=463 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.doForward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =469, line=398 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.forward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =55, line=301 (interpreted frame)

-Org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel (Java.util.Map, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =244, line=240 (interpreted Frame
-Org.springframework.web.servlet.view.AbstractView.render (Java.util.Map, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =157, line=252 (interpreted frame)
-Org.springframework.web.servlet.DispatcherServlet.render (Org.springframework.web.servlet.ModelAndView, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =225, line=1173 (Compiled frame)
-Org.springframework.web.servlet.DispatcherServlet.doService (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =219, line=809 (interpreted frame)
-Org.springframework.web.servlet.FrameworkServlet.processRequest (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =10, line=571 (Compiled frame)
-Org.springframework.web.servlet.FrameworkServlet.doPost (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =3, line=511 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =139, line=717 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci = 30, line=810 (Compiled frame)
-Org.apache.catalina.core.ApplicationFilterChain.doFilter (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =101, line=173 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.invoke (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =396, line=672 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.processRequest (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =64, line=463 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.doForward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =469, line=398 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.forward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =55, line=301 (interpreted frame)

-Org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel (Java.util.Map, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =244, line=240 (interpreted Frame
-Org.springframework.web.servlet.view.AbstractView.render (Java.util.Map, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =157, line=252 (interpreted frame)
-Org.springframework.web.servlet.DispatcherServlet.render (Org.springframework.web.servlet.ModelAndView, Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =225, line=1173 (Compiled frame)
-Org.springframework.web.servlet.DispatcherServlet.doService (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =219, line=809 (interpreted frame)
-Org.springframework.web.servlet.FrameworkServlet.processRequest (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =10, line=571 (Compiled frame)
-Org.springframework.web.servlet.FrameworkServlet.doPost (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =3, line=511 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.http.HttpServletRequest, Javax.servlet.http.HttpServletResponse) @bci =139, line=717 (interpreted frame)
-Javax.servlet.http.HttpServlet.service (Javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci = 30, line=810 (Compiled frame)
-Org.apache.catalina.core.ApplicationFilterChain.doFilter (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =101, line=173 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.invoke (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =396, line=672 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.processRequest (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =64, line=463 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.doForward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =469, line=398 (interpreted frame)
-Org.apache.catalina.core.ApplicationDispatcher.forward (Javax.servlet.ServletRequest, Javax.servlet.ServletResponse) @bci =55, line=301 (interpreted frame)

......

Through this thread stack you can see that the root cause of the problem is always forward, the code is in a dead loop, the total number of forward is more than 100 times, until the thread stack overflows. This also illustrates that getting a JDBC connection as seen in the previous GDB command is just the last time a loop is being performed, not a real cause. This thread's dead loop forward also affects other threads, resulting in an error occurred during stack walking error, the thread is always in blocked state, in summary the thread stack is very helpful in analyzing the cause of the problem.

(3). You can see the memory usage of the app by command Jmap $JAVA _home$/bin/java core-java-16427-1325846515, the results are as follows:

Using parallel threads in the new generation.
Using Thread-local object allocation.
Concurrent Mark-sweep GC

Heap Configuration:
Minheapfreeratio = 40
Maxheapfreeratio = 70
Maxheapsize = 1342177280 (1280.0MB)
NewSize = 134217728 (128.0MB)
Maxnewsize = 134217728 (128.0MB)
Oldsize = 402653184 (384.0MB)
Newratio = 15
Survivorratio = 1024
PermSize = 100663296 (96.0MB)
MaxPermSize = 100663296 (96.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
Capacity = 134152192 (127.9375MB)
used = 22525464 (21.481956481933594MB)
Free = 111626728 (106.4555435180664MB)
16.790977220856742% used
Eden Space:
Capacity = 134086656 (127.875MB)
used = 22525464 (21.481956481933594MB)
Free = 111561192 (106.3930435180664MB)
16.799183954591275% used
From Space:
Capacity = 65536 (0.0625MB)
Used = 0 (0.0MB)
Free = 65536 (0.0625MB)
0.0% used
To Space:
Capacity = 65536 (0.0625MB)
Used = 0 (0.0MB)
Free = 65536 (0.0625MB)
0.0% used
Concurrent Mark-sweep Generation:
Capacity = 1207959552 (1152.0MB)
used = 764300224 (728.8934936523438MB)
Free = 443659328 (423.10650634765625MB)

Exception in thread "main" java.lang.OutOfMemoryError:requested 4096 bytes for jbyte in/build_area/jdk1.5.0_08/hotspot/ Src/share/vm/prims/jni.cpp. Out of swap space?

The last line appeared oom, this is because the dump out of the file is too large, 32-bit machine supports the maximum memory is 2G, so in the resolution of the oom problem, can be put on 64-bit server to parse.

4. Common Core dump causes

There are many reasons for coredump, such as memory access, illegal pointers, stack overflow and so on, can also be interrupted by sending operating system signals, such as KILL-SIGSEGV <pid> This is an invalid storage signal. If you want to generate a core dump file manually, you can generate it through gdb-p <pid>.

Performance tuning case sharing: Reasons for JVM crash 2