Three examples demo Java Thread Dump log Analysis

Jstack the thread state in the Dump log fileIn the dump file, the interesting thread states are:

1. Deadlock,Deadlock (focus on)
2. In execution,Runnable
3. Waiting for resources,waiting on condition (focus)
4. Waiting to get the monitor,waiting on Monitor entry (focus)
5. Pause,Suspended
6. Object Waits,object.wait () or timed_waiting
7. Blocking,Blocked (Focus)
8. Stop,parked .

Let's start with the first example and then list the meanings and considerations of the different thread states, and then add two more instances at a later stage.

Comprehensive demonstration One: waiting to lock and BlockedExamples are as follows: "RMI TCP Connection (267865) -172.16.5.25" Daemon prio=10 tid=0x00007fd508371000 nid=0x55ae Waiting for monitor entry[0x00007fd4f8684000] Java.lang.Thread.State: BLOCKED (on object monitor)At Org.apache.log4j.Category.callAppenders (category.java:201)- waiting to lock <0x00000000acf4d0c0>(a Org.apache.log4j.Logger) at Org.apache.log4j.Category.forcedLog (category.java:388) at Org.apache.log4j.Category.log (category.java:853) at Org.apache.commons.logging.impl.Log4JLogger.warn ( log4jlogger.java:234) at Com.tuan.core.common.lang.cache.remote.SpyMemcachedClient.get (Spymemcachedclient.java : 110).... 1) The thread status is Blocked, blocking state. Description thread waits for resource timeout!  2) "Waiting to lock <0x00000000acf4d0c0>" means that the thread is waiting for the 0x00000000acf4d0c0 address to be locked (English can be described as: trying to obtain 0X00000000ACF4D0C0 lock). 3) Find the string 0x00000000acf4d0c0 in the dump log and find that a lot of threads are waiting for the address to be locked. If you can find out who gets this lock (such as locked < 0X00000000ACF4D0C0 >) in the log, you can track it down. 4) " Waiting for monitor entry"describes this thread through synchronized (obj) {...} The application entered the critical section and entered the "Entry Set" queue in 1, but the corresponding monitor for the obj was owned by another thread, so this thread waits in the Entry Set queue. 5) in the first line, "RMI TCP Connection (267865) -172.16.5.25" is the Thread Name. Tid refers to the Java Thread ID. Nid refers to the ID of the native thread. Prio is a thread priority. [0x00007fd4f8684000] is the thread stack start address. thread state meaning and considerations in dump file

The meanings are as follows:

• Deadlock: A deadlock thread, generally referred to as multiple thread invocations, entering into the mutual resource occupation, leading to a situation where the wait cannot be released.
• Runnable: Generally refers to the thread in the executing state, the thread is consuming resources, is processing a request, it is possible to pass SQL to database execution, it is possible to operate on a file, it is possible to convert data types.
• Waiting on condition: waits for a resource, or waits for a condition to occur. The specific reason needs to combine stacktrace to analyze.
  • If the stack information is explicitly applied code, it proves that the thread is waiting for the resource. In general, when a resource is read in bulk, and the resource is in a resource lock, the thread waits for the resource to be read.
  • Or, waiting for other threads to execute, and so on.
  • If you find that a large number of threads are in wait on condition, looking from the thread stack, waiting for the network to read and write, this can be a symptom of a network bottleneck. The thread could not execute because of network blocking.
    • One situation is that the network is very busy, almost consumes all the bandwidth, still has a large amount of data waiting for the network to read and write;
    • Another situation may be network idle, but due to problems such as routing, the package does not arrive properly.
  • Another common scenario in which wait on condition occurs is when the thread is in sleep, waiting for the time of sleep to be awakened.
• Blocked: Thread blocking, refers to the current thread execution process, the required resources for a long wait but has not been acquired, by the container's thread manager is identified as blocking state, can be understood as waiting for the resource time-out of the thread.
• Waiting for monitor entry and in Object.wait (): monitor is the primary means used in Java to achieve mutual exclusion and collaboration between threads, which can be seen as an object or class lock. Every object has, and only one monitor. As can be seen from 1, each monitor can only be owned by one thread at a time, and the thread is "Active thread", while the other threads are "waiting thread", and wait in two queue "Entry set" and "Wait set" respectively. The state of the thread waiting in "Entry set" is "Waiting for monitor Entry", while the thread state waiting in "wait set" is "in Object.wait ()".

Figure 1 A Java Monitor

Comprehensive Model II: W aiting on condition and Timed_waitingExamples are as follows: "RMI TCP Connection (idle)" Daemon prio=10 tid=0x00007fd50834e800 nid=0x56b2 Waiting on condition[0x00007fd4f1a59000] Java.lang.Thread.State: timed_waiting (parking)At Sun.misc.Unsafe.park (Native Method)- parking to-wait for <0x00000000acd84de8>(a java.util.concurrent.synchronousqueue$transferstack) at Java.util.concurrent.locks.LockSupport.parkNanos ( locksupport.java:198) at Java.util.concurrent.synchronousqueue$transferstack.awaitfulfill (SynchronousQueue.java : 424) at Java.util.concurrent.synchronousqueue$transferstack.transfer (synchronousqueue.java:323) at Java.util.concurrent.SynchronousQueue.poll (synchronousqueue.java:874) at Java.util.concurrent.ThreadPoolExecutor.getTask (threadpoolexecutor.java:945) at Java.util.concurrent.threadpoolexecutor$worker.run (threadpoolexecutor.java:907) at Java.lang.Thread.run ( thread.java:662) 1) " timed_waiting (parking)Timed_waiting refers to the wait state, but the time is specified here, and the waiting state is automatically exited after the specified time; parking refers to the thread being suspended.
2) " Waiting on condition"Need to be in the stack." parking to-wait for <0x00000000acd84de8>(a java.util.concurrent.synchronousqueue$transferstack) "in combination. First of all, this thread must be waiting for a certain condition to happen, to wake itself up. Second, Synchronousqueue is not a queue, just a mechanism for transferring information between threads, and when we put an element into synchronousqueue, there must be another thread waiting to accept the handover, so this is the condition that this thread is waiting for. 3) Nothing else can be seen.

Integrated model Three: in Obejct.wait () and Timed_waiting

Examples are as follows: "RMI renewclean-[172.16.5.19:28475]" daemon prio=10 tid=0x0000000041428800 nid=0xb09 In object.wait ()[0x00007f34f4bd0000] Java.lang.Thread.State: timed_waiting (on object monitor)At java.lang.Object.wait (Native Method)- Waiting on <0x00000000aa672478> (a java.lang.ref.referencequeue$lock)At Java.lang.ref.ReferenceQueue.remove (referencequeue.java:118)- locked <0x00000000aa672478> (a java.lang.ref.referencequeue$lock)At Sun.rmi.transport.dgcclient$endpointentry$renewcleanthread.run (dgcclient.java:516) at Java.lang.Thread.run ( thread.java:662) 1) " timed_waiting (on object monitor)", for this example, it is because this thread called java.lang.Object.wait (long Timeout) and entered the wait state.

2) The state of the thread waiting in "Wait Set" is " In object.wait ()”。 When the thread obtains the monitor and enters the critical section, if the condition that the thread continues to run is not satisfied, it calls the Wait () method of the object (typically the object being synchronized), discards the monitor, and enters the "Wait Set" queue. Only if another thread calls notify () or Notifyall () on the object, the "Wait Set" queue is given the opportunity to compete, but only one thread obtains the object's Monitor and reverts to the running state.

3) RMI Renewclean is part of the dgcclient. DGC refers to the distributed GC, which is distributed garbage collection.

4) Please note that the first locked <0x00000000aa672478>After Waiting on <0x00000000aa672478>, the first lock is the same object, see the following code implementation: Static Private class Lock {};p rivate lock lock = new Lock ();p ublic reference< Extends t> remove (long timeout) {synchronized (lock) {reference<?extends t> r = Reallypoll (); if (r! = null) return R; for (;;) {lock.wait (timeout); r = Reallypoll ();...... }} That is, in the execution of a thread, the Monitor of this object is first obtained with synchronized (corresponding to locked <0x00000000aa672478>); When executing to lock.wait (timeout), the thread discards the ownership of Monitor and enters the "Wait Set" queue (corresponding to the Waiting on <0x00000000aa672478>）。 5) from the stack information, is the remote references to remote objects is being cleaned up, the reference lease to the, distributed garbage collection in one-off.

Reference Resources:1) cubrid,2012,how to Analyze Java Thread dumps;2) iteye,2012, virtual machine stack full analysis, 3) iteye,2008, how to analyze Java virtual machine deadlock; 4) Csdn,2012,java A description and explanation of some common threads generated during JVM operation in stack dump; 5) 2009,java thread dump analysis; 6) jiacheo,2012,tomcat thread dump analysis; Spymemcached related articles: 1) spymemcached usenagle problem and TCP/IP delay send data 2) spymemcached: One MC node operation continuously timeout more than 998 times auto-reconnect feature 3) about Multiget hole:spymemcached implementation of this method 1) 55 Best Practice Series: MongoDB Best Practices (2012-12-15 15:48)
2) 55 Best Practice Series: Logging Best Practices (2012-12-15 16:43) 3) Best Practices Series: Front-end code standards and best Practices (2012-12-19 23:33) A gift map:

Three examples demo Java Thread Dump log Analysis