Thread dump and Java application Diagnosis

Thread dump and Java application Diagnosis

Thread dump is a very useful tool for diagnosing Java application problems. Every Java Virtual Machine can generate thread-dump that shows all threads in a certain state in a timely manner. Although the output formats of thread dump on each Java virtual machine are slightly different, the information provided by thread dumps includes the thread, the running status, identity, and call stack of the thread; the called Stack contains the complete class name, the method executed, and the number of lines of source code if possible.

Thread dump features:

• Available in various operating systems
• Can be used in a variety of Java application servers
• It can be used in the production environment without affecting the system performance.
• You can direct the problem to the application code line.

Problems that can be diagnosed by thread dump include:

• Search for memory leaks. It is common to load a large amount of data into the cache in a program.
• Deadlock thread found

Sun's JVM can generate thread dump stack information using the following methods:

1, Solaris OS
<Ctrl>-'/' (Control-backslash)
Kill-Quit <pid>

HP-UX/Unix/Linux
Kill -3 PID 
PID obtained through the following method
ps -efHl | grep 'java' **. **

3, Windows
Directly press Ctrl-break for the msdos window Program

Some Java application servers run on the console, such as weblogic. To facilitate threaddump information, it is best to redirect standard output to a file when Weblogic is started, use "nohup sh startweblogic. sh> Start. log & "command, Run" Kill-3 <pid> ", and stack trace will output to start. log. The thread dump of Tomcat is output to the command line console or the Catalina. Out file of logs. To reflect the dynamic changes in the thread status, thread dump needs to be performed multiple times at an interval of 10-20 s.

Thread dump generated in ibm jvm:

When ibm jvm is used on AIX, by default, javacore files (about CPU) and heapdump files (about memory) are generated when memory overflow occurs ). If you do not use the following methods:
1 choose one cluster member, set the following before this server start:
Set the following environment variables before was starts (can be added to the startup script)
Export ibm_heapdump = true
Export ibm_heap_dump = true
Export ibm_heapdump_outofmemory = true
Export ibm_heapdumpdir = <directory path>

2 Please use set command to make sure you do not have disable_javadump Parameter
Then start this cluster member.
Run the SET command to check the parameter settings, make sure that disable_javadump is not set, and then start the server

3 when you find free memory <50% when no heavy access, please run kill-3 <pid>
Run the kill-3 <pid> command to generate the javacore file and heapdump file (PID is the ID number of the was Java Process, which can be checked by PS-Ef | grep Java ), you can perform the following operations several times:

PS-EF> limit f1.txt
PS aux> psaux1.txt
Vmstat 5 10> vmstat.txt
Kill-3 <app server ID>
Wait for 2 mins
Kill-3 <app server ID>
Wait for 2 mins
Kill-3 <app server ID>
Netstat-An> netstat2.txt
PS-EF> psef2.txt
PS aux> psaux2.txt
Copy the TXT and/usr/WebSphere/appserver/javacore * files and heapdump files generated above to the local device, and then delete these files, because these files occupy a large amount of file system space.
Copy the logs generated on the current day in the/usr/WebSphere/appserver/logs/wlmserver1 (or 2) directory.

Common thread statuses of Web containers on the application server in javacore or threaddump files generated by ibm jvm:

Idle thread: a thread that is ready to accept requests, but has not established a connection with the plug-in or client.
Keep-alive thread: a thread that is ready to accept requests and has established a connection with the plug-in or client.
Request receiving thread: the content or header of the request being read by a thread.

The following describes the representation of various threads in javacore or threaddump:

Idle thread:
"Servlet. Engine. Transports: 20" (TID: 0x427f190, sys_thread_t: 0x15d175e8, state: R, native ID: 0xbb8) PRIO = 5
At java. Lang. Object. Wait (native method)
At java. Lang. Object. Wait (object. Java: 429)
At com. IBM. ws. util. boundedbuffer. Take (boundedbuffer. Java: 161)
At COM. IBM. WS. util. threadpool. gettask (threadpool. java (compiled code) at com. IBM. WS. util. threadpool $ worker. run (threadpool. java (compiled code ))

Keep-alive thread (non-SSL mode ):
"Servlet. Engine. Transports: 20" (TID: 0x427f190, sys_thread_t: 0x15d175e8, state: R, native ID: 0xbb8) PRIO = 5
At java.net. socketinputstream. socketread (native method)
At java.net. socketinputstream. Read (socketinputstream. Java: 86)
At com. IBM. ws. Io. Stream. Read (stream. Java)
At com. IBM. ws. Io. readstream. readbuffer (readstream. Java)
At com. IBM. ws. Io. readstream. Read (readstream. Java)
At com. IBM. ws. http. httprequest. readrequestline (httprequest. Java)
At com. IBM. ws. http. httprequest. readrequest (httprequest. Java)
At com. IBM. ws. http. httpconnection. readandhandlerequest (httpconnection. Java)
At com. IBM. ws. http. httpconnection. Run (httpconnection. Java)
At com. IBM. ws. util. cachedthread. Run (threadpool. Java)

Keep-alive thread (SSL mode ):
"Servlet. Engine. Transports: 12" (TID: 0x458dba18, sys_thread_t: 0x60b297c0, state: R, native ID: 0x427e) PRIO = 5
At java.net. socketinputstream. socketread (native method)
At java.net. socketinputstream. Read (socketinputstream. Java (compiled code ))
At com. IBM. sslite. s. A (unknown source) (compiled code)
At com. IBM. sslite. S. B (unknown source) (compiled code)
At com. IBM. sslite. s. A (unknown source) (compiled code)
At com. IBM. sslite. A. Read (unknown source) (compiled code)
At com. IBM. JSSE. A. Read (unknown source) (compiled code)
At com. IBM. ws. Io. Stream. Read (stream. Java (compiled code ))
At com. IBM. ws. Io. readstream. readbuffer (readstream. Java (inlined compiled code ))
At com. IBM. ws. Io. readstream. Read (readstream. Java (inlined compiled code ))
At com. IBM. ws. http. httprequest. readrequestline (httprequest. Java (compiled code ))
At com. IBM. ws. http. httprequest. readrequest (httprequest. Java (compiled code ))
At com. IBM. ws. http. httpconnection. readandhandlerequest (httpconnection)
At com. IBM. ws. http. httpconnection. Run (httpconnection. Java (compiled code ))
At com. IBM. ws. util. threadpool $ worker. Run (threadpool. Java: 672)

Request receiving thread:
At java.net. socketinputstream. socketread (native method)
At java.net. socketinputstream. Read (socketinputstream. Java: 85)
At com. IBM. ws. Io. Stream. Read (stream. Java: 17)
At com. IBM. ws. Io. readstream. readbuffer (readstream. Java: 411)
At com. IBM. ws. Io. readstream. Read (readstream. Java: 110)
At com. IBM. ws. http. httpconnection. Run (httpconnection. Java: 448)
At com. IBM. ws. util. threadpool $ worker. Run (threadpool. Java: 672)

Common thread statuses of Sun JVM:

The thread dump information mainly focuses on the thread status and its execution stack.
The thread status is generally three types
Runnable (R): the thread that can be run currently.
Waiting on Monitor (CW): active thread wait
Waiting for monitor entry (MW): thread lock
Generally, threads in the first and third States are concerned.
When the CPU is busy, pay attention to the runnable thread.
When the CPU is idle, pay attention to the waiting for monitor entry thread.
A typical deadlock occurs when a server application (such as a servlet) Requests resources from the server of the same WebLogic instance.
The solution is to put the servlet in another execution queue for execution.

The following shows a typical deadlock thread (note:Stuck keyword):

"[Stuck] Executethread: '2' for queue: 'weblogic. kernel. Default (Self-tuning) '"daemon PRIO = 10 tid = 02fe9a18 nid = 35 lwp_id = 7518924Runnable[Export dd000 .. mongodb878]
At java.net. socketinputstream. socketread0 (native method)
At java.net. socketinputstream. Read (fig. Java: 134)
At weblogic. JDBC. Oracle. net8.oracledataprovider. getarrayofbytesfromsocket (unknown source)
At weblogic. JDBC. Oracle. net8.oracledataprovider. readfirstpacketinbuffer (unknown source)
At weblogic. JDBC. Oracle. net8.oracledataprovider. readpacket (unknown source)
At weblogic. JDBC. Oracle. net8.oracledataprovider. Receive (unknown source)
At weblogic. JDBC. Oracle. net8.oraclenet8nsptdapacket. sendrequest (unknown source)
At weblogic. JDBC. Oracle. oracleimplstatement. fetchnext (unknown source)
At weblogic. JDBC. Oracle. oracleimplstatement. fetchnext2 (unknown source)
At weblogic. JDBC. Oracle. oracleimplresultset. fetchatposition (unknown source)
At weblogic. JDBC. Base. baseimplresultset. Next (unknown source)
At weblogic. JDBC. Base. baseresultset. Next (unknown source)
-Locked<55f25550> (A weblogic. JDBC. Oracle. oracleconnection)
At weblogic. JDBC. wrapper. resultset_weblogic_jdbc_base_baseresultset.next (unknown source)
At org. hibernate. loader. loader. docquery (loader. Java: 685)

Use top, vmstat, or prstat commands in Unix/Linux to observe system resources

Mandy Chung's blog has a blog about thread dump and concurrency locks, which is extracted as follows:

Thread dumps are very useful for diagnosing synchronization related problems such as deadlocks on Object monitors. CTRL-/on Solaris/Linux or Ctrl-break on Windows has been a common way to get a thread dump of a running application. on Solaris or Linux, you can send a quit signal to the target application. the target application in both cases prints a thread dump to the standard output and also detects if there is any deadlock involving object monitors.

Jstack, a new troubleshooting utility introduced in Tiger (j2se 5.0), provides another way to obtain a thread dump of an application. alan Bateman has a nice blog about jstack and its several improvements in Mustang (Java SE 6 ). mustang jstack works like a remote Ctrl-/or Ctrl-break if you are on Windows.

Jconsole is JMX-complaint GUI tool which allows you to get a thread dump on the fly. The "using jconsole to monitor Applications" Article gives you an overview of the tiger monitoring and management functionality.

Mustang extends the thread dump, jstack, and jconsole to support Java. util. concurrent. locks to improve its diagnosability. for example, the threads tab in the Mustang jconsole now shows which synchronizer a thread is waiting to acquire when the thread is blocked to lock a reentrantlock and also which thread is owning that lock.

In addition, it has a new "detect deadlock" button (in the bottom ). when you click on the "detect deadlock" button, it will send a request to the target application to perform the Deadlock Detection operation. if the target application is running on Mustang, it finds deadlocks involving both object monitors as well as the java. util. concurrent. locks. if the target application is running on tiger, it finds deadlocks involving object monitors only. each deadlock cycle will be displayed in a separate deadlock tab.

Click here to see a wider form of This screenshot.

JDK 6 has a nice demoFullthreaddumpUnder $ jdk_home/demo/management/fullthreaddump where jdk_home is the location of your JDK 6. this demo has been encoded in JDK 5.0 and is updated to use the new Mustang API. it demonstrates the use of the Java. lang. management API to get the thread dump and detect deadlock programmatically.