How to diagnose Java application failures using Jstack

In recent times, our production system upgrade frequently failure, the specific failure phenomenon is 10 minutes after the start of the transaction slow, processing thread exhaustion and so on, and the frequency of the failure occurs quite high. After detailed diagnosis and troubleshooting, we finally found the problem, which is that groovy runs in OSGi with ClassLoader deadlock, and finally we solve the problem.
If it is difficult to find this problem by looking at the code alone, in this troubleshooting, I also learned how to better use the JVM monitoring tools to diagnose, mainly using the Jstack and JMAP commands, Jmap said the last time, no longer speak, Here's an example of how to use Jstack to diagnose a Java program.
First, let's take a look at the role of the Jstack command, Jstack is a utility that can return a full dump of various threads running on the application that you can use to pinpoint the problem. Jstack [-l] <pid>,jpid can view the Jpid value of the current Java program by using the JPS Command, and-L is an optional parameter that shows the thread blocking/deadlock condition.

/*** Dead Lock Example*/ Public classDeadLock { Public Static voidMain (string[] args) {FinalObject obj_1 =NewObject (), obj_2 =NewObject (); Thread T1=NewThread ("T1") {@Override Public voidrun () {synchronized(obj_1) {Try{Thread.Sleep (3000); } Catch(Interruptedexception e) {}synchronized(obj_2) {System.out.println ("Thread T1 done.");                }                }            }        }; Thread T2=NewThread ("T2") {@Override Public voidrun () {synchronized(obj_2) {Try{Thread.Sleep (3000); } Catch(Interruptedexception e) {}synchronized(obj_1) {System.out.println ("Thread T2 done.");                }                }            }        };        T1.start ();    T2.start (); }    }

The above deadlock class is a deadlock example, if we do not know the case, after running deadlock, found that wait n long did not print thread completion information on the screen. At this point we can use JPS to view the program's Jpid value and use Jstack to produce the stack result problem.

$ java-CP Deadlock.jar Deadlock &

$ JPS   3076 Jps   448  448 > Deadlock.jstack

The result file Deadlock.jstack content is as follows:

2011-03-20 23:05:20full thread dump Java HotSpot (TM) Client VM (19.1-b02 mixed mode, sharing): "DESTROYJAVAVM" Prio=6 t id=0x00316800 NID=0X9FC waiting on condition [0x00000000] Java.lang.Thread.State:RUNNABLE Locked ownable Synchronizer S:-None "T2" prio=6 tid=0x02bcf000 nid=0xc70 waiting for monitor entry [0x02f6f000] Java.lang.Thread.State:BLOCKED (on object monitor) in Com.demo.deadlock$2.run (deadlock.java:40)-Waiting to lock<0x22a297a8>(a java.lang.Object)-locked<0x22a297b0>(a java.lang.Object) Locked ownable synchronizers:-None "T1" prio=6 tid=0x02bce400 nid=0xba0 waiting for Monito     R entry [0x02f1f000] Java.lang.Thread.State:BLOCKED (on object monitor) at Com.demo.deadlock$1.run (deadlock.java:25) -Waiting to lock<0x22a297b0>(a java.lang.Object)-locked<0x22a297a8> (a java.lang.Object) Locked ownable synchronizers:-None "low Memory Detector" daemon    prio=6 tid=0x02bb9400 nid=0xa6c runnable [0x00000000] Java.lang.Thread.State:RUNNABLE Locked ownable synchronizers: -None "CompilerThread0" daemon prio=10 tid=0x02bb2800 nid=0xcb8 waiting on condition [0x00000000] Java.lang.Thread.Sta  Te:runnable Locked ownable synchronizers:-None "Attach Listener" daemon prio=10 tid=0x02bb1000 nid=0x7f4 waiting on condition [0x00000000] Java.lang.Thread.State:RUNNABLE Locked ownable synchronizers:-None "Signal Dispatcher" da Emon prio=10 tid=0x02bd2800 nid=0xd80 runnable [0x00000000] Java.lang.Thread.State:RUNNABLE Locked ownable Synchroniz ERS:-None "Finalizer" daemon prio=8 tid=0x02bab000 nid=0xe1c in object.wait () [0x02d3f000] Java.lang.thread.state:w Aiting (on object monitor) at java.lang.Object.wait (Native Method)-Waiting on <0x229e1148>(a java.lang.ref.referencequeue$lock) at Java.lang.ref.ReferenceQueue.remove (referencequeue.java:118)-locked <0x229e1148>(a java.lang.ref.referencequeue$lock) at Java.lang.ref.ReferenceQueue.remove (referencequeue.java:134) at JAVA.L Ang.ref.finalizer$finalizerthread.run (finalizer.java:159) Locked ownable synchronizers:-None "Reference Handler" Dae    Mon prio=10 tid=0x02ba6800 nid=0xbe0 in object.wait () [0x02cef000] Java.lang.Thread.State:WAITING (on Object Monitor) At java.lang.Object.wait (Native Method) – Waiting on<0x229e1048>(a java.lang.ref.reference$lock) at java.lang.Object.wait (object.java:485) at java.lang.ref.reference$reference Handler.run (reference.java:116)-Locked<0x229e1048>(a java.lang.ref.reference$lock) Locked ownable synchronizers:-None "VM Thread" prio=10 tid=0x02b6a400 nid=0x56  8 runnable "VM periodic Task Thread" prio=10 tid=0x02bc8400 nid=0x75c waiting on condition JNI global References:878found One java-level deadlock:============================= "T2": Waiting to lock monitor 0X02BAAEEC (Object 0x22a297a8, a Java . lang. Object), which is held by "T1" "T1": Waiting to lock monitor 0X02BAA2BC (Object 0x22a297b0, a java.lang.Object), which I s held by ' T2 ' Java stack information for the threads listed above:=================================================== "T2 ": At Com.demo.deadlock$2.run (deadlock.java:40) – Waiting to lock<0x22a297a8>(a java.lang.Object)-locked<0x22a297b0>(a java.lang.Object) "T1": at Com.demo.deadlock$1.run (deadlock.java:25) – Waiting to lock<0x22a297b0>(a java.lang.Object)-locked<0x22a297a8>(a java.lang.Object) Found 1 deadlock.

From this result file we see a deadlock, in particular, thread T2 waiting for thread T1, and thread T1 in waiting for the thread T2, but also record the thread stack and number of lines of code, through this stack and the number of rows we can check the corresponding code block, to identify problems and solve problems.

http://crane-ding.iteye.com/blog/968862

How to diagnose Java application failures using Jstack (GO)