Thread Dump and Java Application Diagnostics (RPM)

Thread Dump and Java application diagnostics
Thread dump is a useful tool for diagnosing Java application problems, and each Java virtual machine has the ability to generate a thread-dump that shows all threads in a certain state in a timely fashion. Although the Java Virtual machine thread dump print output format slightly different, but the thread dumps out of the information contained threads, thread of the running state, identity and call of the stack; The call stack contains the full class name, the method executed, and, if possible, the number of lines in the source code.

Thread Dump Features:

? can be used under various operating systems
? can be used under a variety of Java application servers
? can be used in a production environment without affecting the performance of the system
? You can position the problem directly on the code line of the application
The problems that Thread dump can diagnose include:

To find memory leaks, it's common for the program to load a lot of data into the cache
? Discover the deadlock thread
Sun's JVM can generate thread dump stack information in the following ways:

1,solaris OS
<ctrl>-'/' (Control-backslash)
Kill-quit <pid>

2, Hp-ux/unix/linux
Kill-3 PID
PID is obtained by the following method
PS-EFHL | grep ' java ' * *. **

3,windows
Directly to the Msdos window of the program press Ctrl-break

Some Java application server is running on the console, such as WebLogic, in order to facilitate the acquisition of Threaddump information, at the time of WebLogic startup, it is best to redirect its standard output to a file, with "Nohup sh startweblogic.sh > Start.log & "command, execute" kill-3 <pid> ", Stack trace will be output to start.log. The thread dump of Tomcat is output to the command line console or to the logs catalina.out file. In order to reflect the dynamic change of thread state, we need to do thread dump multiple times, each interval 10-20s.

The thread Dump is generated under the IBM JVM:

With IBM's JVM on AIX, memory overflows generate Javacore files (about the CPU) and Heapdump files (about memory) by default. If you do not refer to the following methods:
1 Choose one cluster member, set the following before this server start:
Set the following environment variables (which can be added to the startup script) before was is started
Export Ibm_heapdump=true
Export Ibm_heap_dump=true
Export Ibm_heapdump_outofmemory=true
Export Ibm_heapdumpdir=<directory path>

2 Please use the SET command to make sure don't have disable_javadump parameter
Then start this cluster member.
Check the parameter settings with the SET command, ensure that Disable_javadump is not set, and then start the server

3 When do I find free memory < 50% If no heavy access, please run kill-3 <pid>
Execute kill-3 <pid> command can generate Javacore file and Heapdump file (PID is the ID number of the was Java process, can be found in Ps-ef|grep Java), can be executed several times, follow the following action

Ps-ef > Psef1.txt
PS aux > psaux1.txt
Vmstat 5 > Vmstat.txt
kill-3 <app Server id>
Wait for 2 mins
kill-3 <app Server id>
Wait for 2 mins
kill-3 <app Server id>
Netstat-an> Netstat2.txt
Ps-ef > Psef2.txt
PS aux > psaux2.txt
Copy the TXT and/usr/websphere/appserver/javacore* files and heapdump files generated above to local, and then delete these files, because they take up large file system space.
Copy the log generated on the day of the/usr/websphere/appserver/logs/wlmserver1 (or 2) directory

Apply the common thread state of the server Web container in the Javacore or Threaddump file generated by the IBM JVM:

Idle thread: A thread that is ready to accept the request, but does not establish a connection to the plug-in or client
Keep-alive thread: is a thread that is ready to accept the request and has established a connection with the plug-in or client
The thread that is accepting the request: is a thread reading the contents of request or the header

The following shows the forms of the various threads in Javacore or threaddump:

Idle thread:
"servlet.engine.transports:20" (tid:0x427f190, Sys_thread_t:0x15d175e8, state:r, native Id:0xbb8) prio =5
at java.lang.Object.wait (Native Method)
@ java.lang.Object.wait (object.java:429)
at Com.ibm.ws.util.BoundedBuffer.take (boundedbuffer.java:161)
at Com.ibm.ws.util.ThreadPool.getTask ( Threadpool.java (Compiled Code)) at Com.ibm.ws.util.threadpool$worker.run (Threadpool.java (Compiled code))

Keep-alive thread (non-SSL mode):
"Servlet.engine.transports:20" (tid:0x427f190, Sys_thread_t:0x15d175e8, state:r, native Id:0xbb8) prio=5
At Java.net.SocketInputStream.socketRead (Native Method)
At Java.net.SocketInputStream.read (socketinputstream.java:86)
At Com.ibm.ws.io.Stream.read (Stream.java)
At Com.ibm.ws.io.ReadStream.readBuffer (Readstream.java)
At Com.ibm.ws.io.ReadStream.read (Readstream.java)
At Com.ibm.ws.http.HttpRequest.readRequestLine (Httprequest.java)
At Com.ibm.ws.http.HttpRequest.readRequest (Httprequest.java)
At Com.ibm.ws.http.HttpConnection.readAndHandleRequest (Httpconnection.java)
At Com.ibm.ws.http.HttpConnection.run (Httpconnection.java)
At Com.ibm.ws.util.CachedThread.run (Threadpool.java)

Keep-alive thread (SSL mode):
"Servlet.engine.transports:12" (Tid:0x458dba18, sys_thread_t:0x60b297c0, state:r, native id:0x427e) prio=5
At Java.net.SocketInputStream.socketRead (Native Method)
At Java.net.SocketInputStream.read (Socketinputstream.java (Compiled Code))
At COM.IBM.SSLITE.S.A (Unknown Source) (Compiled Code)
At com.ibm.sslite.s.b (Unknown Source) (Compiled Code)
At COM.IBM.SSLITE.S.A (Unknown Source) (Compiled Code)
At Com.ibm.sslite.a.read (Unknown Source) (Compiled Code)
At Com.ibm.jsse.a.read (Unknown Source) (Compiled Code)
At Com.ibm.ws.io.Stream.read (Stream.java (Compiled Code))
At Com.ibm.ws.io.ReadStream.readBuffer (Readstream.java (inlined Compiled Code))
At Com.ibm.ws.io.ReadStream.read (Readstream.java (inlined Compiled Code))
At Com.ibm.ws.http.HttpRequest.readRequestLine (Httprequest.java (Compiled Code))
At Com.ibm.ws.http.HttpRequest.readRequest (Httprequest.java (Compiled Code))
At Com.ibm.ws.http.HttpConnection.readAndHandleRequest (httpconnection)
At Com.ibm.ws.http.HttpConnection.run (Httpconnection.java (Compiled Code))
At Com.ibm.ws.util.threadpool$worker.run (threadpool.java:672)

Thread that is accepting the request:
At Java.net.SocketInputStream.socketRead (Native Method)
At Java.net.SocketInputStream.read (socketinputstream.java:85)
At Com.ibm.ws.io.Stream.read (stream.java:17)
At Com.ibm.ws.io.ReadStream.readBuffer (readstream.java:411)
At Com.ibm.ws.io.ReadStream.read (readstream.java:110)
At Com.ibm.ws.http.HttpConnection.run (httpconnection.java:448)
At Com.ibm.ws.util.threadpool$worker.run (threadpool.java:672)

Common thread states for Sun JVMs:

For the thread dump information, the main focus is on the state of the thread and its execution stack
The state of a thread is generally three classes
Runnable (R): Threads that are currently running
Waiting on Monitor (CW): Thread active wait
Waiting for monitor entry (MW): Thread lock
The general focus is on the first and third States of the thread
CPU is busy, focus on runnable threads
CPU idle focus on waiting for monitor entry thread
A typical deadlock is a resource that is requested by the same WebLogic instance server in a server-side application (such as a servlet)
The solution is to put the servlet in the other execution queue to execute

A typical deadlock thread is given below (note the Stuck keyword):

"[STUCK] Executethread: ' 2 ' for queue: ' Weblogic.kernel.Default (self-tuning) '" Daemon prio=10 tid=02fe9a18 nid=35 lwp_id =7518924 runnable [440dd000. 440DB878]
At java.net.SocketInputStream.socketRead0 (Native Method)
At Java.net.SocketInputStream.read (socketinputstream.java:134)
At Weblogic.jdbc.oracle.net8.OracleDataProvider.getArrayOfBytesFromSocket (Unknown Source)
At Weblogic.jdbc.oracle.net8.OracleDataProvider.readFirstPacketInBuffer (Unknown Source)
At Weblogic.jdbc.oracle.net8.OracleDataProvider.readPacket (Unknown Source)
At Weblogic.jdbc.oracle.net8.OracleDataProvider.receive (Unknown Source)
At Weblogic.jdbc.oracle.net8.OracleNet8NSPTDAPacket.sendRequest (Unknown Source)
At Weblogic.jdbc.oracle.OracleImplStatement.fetchNext (Unknown Source)
At Weblogic.jdbc.oracle.OracleImplStatement.fetchNext2 (Unknown Source)
At Weblogic.jdbc.oracle.OracleImplResultset.fetchAtPosition (Unknown Source)
At Weblogic.jdbc.base.BaseImplResultSet.next (Unknown Source)
At Weblogic.jdbc.base.BaseResultSet.next (Unknown Source)
-Locked <55f25550> (a weblogic.jdbc.oracle.OracleConnection)
At Weblogic.jdbc.wrapper.ResultSet_weblogic_jdbc_base_BaseResultSet.next (Unknown Source)
At Org.hibernate.loader.Loader.doQuery (loader.java:685)

Unix/linux use the top, Vmstat, or PRSTAT commands to observe system resource status

Mandy Chung's blog has an article on thread Dump and Concurrency locks blog, picked as follows:
Thread dumps is very useful for diagnosing synchronization related problems such as deadlocks on object monitors. ctrl-/on Solaris/linux or ctrl-break on Windows have been a common a-to get a thread dump of a running application. On Solaris or Linux, you can send a QUIT signal to the target application. The target application in both cases prints a thread dump to the standard output and also detects if there are any deadlock involving object monitors.
Jstack, a new troubleshooting utility introduced in Tiger (J2SE 5.0), provides another the ' to ' obtain a thread dump of an A Pplication. Alan Bateman have a nice blog about Jstack and their several improvements in Mustang (Java SE 6). Mustang Jstack works like a remote ctrl-/or Ctrl-break if is on Windows.
Jconsole is jmx-complaint GUI tool which allows you to get a thread dump on the fly. The ' Using JConsole to Monitor applications ' article gives you a overview of the Tiger monitoring and management function Ality.
Mustang extends the thread dump, Jstack, and jconsole to support java.util.concurrent.locks to improve its diagnosability. For example, the Threads tab in the Mustang jconsole now shows which synchronizer a thread was waiting to acquire when the Thread was blocked to lock a reentrantlock and also which thread was owning that lock.

In addition, it had a new "Detect deadlock" button (in the bottom). When you click on the "Detect deadlock" button, it'll send a request to the target application to perform the deadlock D Etection operation. If the target application is running in Mustang, it finds deadlocks involving both object monitors as well as the Java.uti L.concurrent.locks. If the target application is running in Tiger, it finds deadlocks involving object monitors only. Each deadlock cycle is displayed in a separate Deadlock tab.

Click here to see a wider form of this screenshot.
JDK 6 have a nice demo fullthreaddump under $JDK _home/demo/management/fullthreaddump where Jdk_home are the location of your JDK 6. This demo have been included in JDK 5.0 and are updated to use the new Mustang API. It demonstrates the use of the java.lang.management API to get the thread dump and detect deadlock programmatically.

Thread Dump and Java Application Diagnostics (RPM)