JVM: How to parse the thread stack

English Original: Jvm:how to analyze Thread Dump

In this article I will teach you how to analyze the thread stack of the JVM and how to find the root cause of the problem from the stack information. In my opinion, threading stack analysis technology is a technology that Java EE Product support engineers must master. The information stored in the thread stack is often far beyond your imagination, and we can take advantage of this information in our work.

My goal is to share the knowledge and experience that I have accumulated over the past more than 10 years in online analytics. This knowledge and experience was gained in the various versions of the JVM and in-depth analysis of the JVM vendors of each vendor, and in this process I also summed up a number of common problem templates.

So, are you ready to bookmark this article now, and in the next few weeks I'll bring you a series of special articles. For what, please share this thread analysis training program with your colleagues and friends quickly.

Sounds good, I really should upgrade my threading stack analysis skills ... But where do I start?

My advice is to follow me through this threading analysis training program. Here are the training content that we will cover. At the same time, I will share the actual cases I have handled to everyone, so as to learn and understand with you.

1) thread stack Overview and basics
2) How thread stacks are generated and related tools
3) Differences in format of different JVM thread stacks (Sun HotSpot, IBM JRE, oracal JRockit)
4) Thread stack log introduction and parsing methods
5) thread stack analysis and related technologies
6) Common problem templates (thread state, deadlock, Io call dead, garbage collection/outofmemoryerror problem, dead loop, etc.)
7) Thread stack problem instance analysis

I hope this series of trainings will give you a real help, so keep an eye on the weekly article updates.

But what if I have questions in the course of my studies or don't understand the content of the article?

Don't worry, just think of me as your mentor. Any questions about the thread stack can be consulted (provided the problem is not too low). Please feel free to choose the following ways to get in touch with me:

1) Comment directly below this article (sorry to be anonymous)
2) Submit your thread stack data to root cause analysis forum
3) Email me the address is @[email protected]

Can you help me analyze the problems of our products?

Sure, you can send me your stack site data via email or forum Root cause analysis Forum if you want. To deal with practical problems is to learn to improve skills of the kingly.

I really hope that you will enjoy this training. So I will do my best to provide you with high-quality materials and answer all the questions.

Before we introduce the thread stack analysis technique and problem mode, we need to tell you about the basic content. So in this post, I'm going to cover the basics so that you can better understand the interactions between the JVM, the middleware, and the Java EE container.

Java VM Overview

The Java Virtual machine is the foundation of the Jave EE platform. It is where middleware and applications are deployed and run.

The JVM provides the following things to middleware software and your Java/java EE program:

– (binary form) Java/java EE program operating environment
– some program features and tools (IO infrastructure, data structures, thread management, security, monitoring, etc.)
– Dynamic memory allocation and management with garbage collection

Your JVM can reside in many operating systems (Solaris, AIX, Windows, and so on.) , and depending on your physical server configuration, you can install 1 to multiple JVM processes on each physical/virtual server.

Interaction between the JVM and the middleware

The following diagram shows a high-level interaction model between the JVM, middleware, and applications.

The diagram shows some simple and typical interactions between the JVM, middleware, and application components. As you can see, the thread allocation of a standard Java EE application is done between the middleware kernel and the JVM. (with the exception of course, the application can call the API directly to create threads, which is not common and is especially careful in the process of use)

Also, note that some threads are managed internally by the JVM, a typical example of which is the garbage collection thread, which is used internally by the JVM for parallel garbage collection processing.

Because most of the thread allocations are done by the Java EE container, it is important for you to be able to understand and recognize thread stack traces and to recognize them from the thread stack data. This allows you to quickly know what type of request the Java EE container is going to perform.

From the analysis perspective of a thread dump stack, you will be able to understand the differences between the thread pools discovered from the JVM and identify the types of requests.

The last section gives you an overview of what a JVM thread stack is for a hotsop VM, as well as the various different threads you will encounter. The details on the IBM VM thread stack will be provided to you in section fourth.

Note that you can get a sample thread stack for this article from the root Cause analysis forum.

JVM thread Stack-what is it?

The JVM thread stack is a snapshot of a given time and can provide you with a complete list of all the Java threads that were created.

Each Java thread that is discovered will give you the following information:

– The name of the thread, often used by the middleware vendor to identify the thread, typically with the assigned thread pool name and state (run, block, and so on).

– Thread type & priority, for example: Daemon prio=3 * * Middleware programs typically create their threads in the form of a daemon, which means that these threads are running in the background, and they provide services to their users, such as: To your Java EE application * *

–java thread ID, for example: tid=0x000000011e52a800 * * This is the Java thread ID obtained through JAVA.LANG.THREAD.GETID (), which is often implemented with self-growing long-shaping 1..n**

– The native thread ID, for example: nid=0x251c*, is critical because the native thread ID allows you to obtain information such as the most CPU time that the thread has used in your JVM from the perspective of the operating system. * *

–java thread status and details, for example: Waiting for monitor entry [0xfffffffea5afb000] Java.lang.Thread.State:BLOCKED (on object monitor)
* * Can quickly understand the possible causes of extremely current blocking of thread state * *

–java thread stack tracking; This is the most important data you can find from the thread stack so far. This is where you spend most of your analysis time, because the Java stack trace provides 90% of the information you will need to learn about the root cause of many types of problems in a later practice session.

–java heap memory decomposition; Starting with the Hotspot VM version 1.6, the memory usage of the hotspot can be seen at the end of the thread stack, such as Java heap memory (Younggen, Oldgen) & PermGen space. This information is useful for analyzing problems caused by frequent GC. You can use known thread data or patterns to do a quick positioning.

?

123456789

HeapPSYoungGen      total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)to   space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)PSOldGen        total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)PSPermGen       total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

Large disassembly of thread stack information

In order to get a better understanding, we provide the following diagram, in this diagram, the thread stack information on the hotspot VM and the threads pool are disassembled in detail, as shown in:

You can see that the thread stack is made up of several different parts. This information is important for problem analysis, but the analysis of different problem patterns uses different parts (problem patterns are modeled and demonstrated in later articles.) ）

Now, through this sample analysis, we give you a detailed explanation of the various components of the Hotespot on-line stack information:

# Full Thread Dump identifier

"Full thread Dump" is a globally unique keyword that you can find in the output log of the thread stack information of the middleware and stand-alone version of Java (for example, use under Unix: Kill-3 <PID>). This is the starting part of the thread stack snapshot.

?

1	Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

# Java EE Middleware, third parties and threads in custom application software

This part is the core part of the entire thread stack, and it is also the part that usually takes the most time to parse. The number of threads in the stack depends on the middleware you use, the third-party libraries (there may be separate threads), and your application (which is usually not a good practice if you create custom threads).

In our sample thread stack, WebLogic is the middleware we use. Starting with WebLogic 9.2, a self-managed thread pool that is uniquely identified with "' Weblogic.kernel.Default (self-tuning)" is used

?

12345678

"[STANDBY] ExecuteThread: ‘414‘ for queue: ‘weblogic.kernel.Default (self-tuning)‘"daemon prio=3tid=0x000000010916a800nid=0x2613in Object.wait() [0xfffffffe9edff000]   java.lang.Thread.State: WAITING (on object monitor)        at java.lang.Object.wait(Native Method)        - waiting on <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)        at java.lang.Object.wait(Object.java:485)        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:160)        - locked <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

# HotSpot VM Thread
This is an internal thread that has a hotspot VM management to perform internal native operations. You don't have to be too paranoid about this, unless you find high CPU usage (through the associated thread stacks and prstat or native thread IDs).

?

1	"VM Periodic Task Thread"prio=3 tid=0x0000000101238800 nid=0x19 waiting on condition

# HotSpot GC Thread
When using a hotspot for parallel GC (which is now common in environments with multiple physical cores), a hotspot VM created by default or each JVM manages a GC thread with a specific identity. These GC threads allow the VM to perform its periodic GC cleanup in parallel, which results in a total decrease in GC time, and at the expense of increased CPU usage time.

?

123	"GC task thread#0 (ParallelGC)"prio=3 tid=0x0000000100120000 nid=0x3 runnable"GC task thread#1 (ParallelGC)"prio=3 tid=0x0000000100131000 nid=0x4 runnable………………………………………………………………………………………………………………………………………………………………

This is very critical data, because when you encounter GC-related problems, such as excessive GC, memory leaks, and so on, you will be able to exploit these threads ' native ID values associated with the operating system or Java threads to discover any high consumption of CPI time. Future articles you will learn how to identify and diagnose such problems.

# JNI Global Reference count
The global reference to JNI (the Java Local interface) is the basic object reference from the local code to Java objects managed by the Java garbage collector. Its role is to block the garbage collection of objects that are still being used by local code, but are technically not "active" in Java code.

It is also important to be aware of JNI references in order to detect JNI-related leaks. If your program uses JNI directly or a third-party tool such as a listener, it can easily cause a local memory leak.

?

1	JNI global references: 1925

# Java Stack Usage view

This data is added back to JDK 1.6, which gives you a short, quick view of the hotspot stack. I find it useful when I'm dealing with GC-related issues with high CPU usage, and you can see both the thread stack and the Java heap in a single snapshot so that you can parse (or exclude) any key points in a particular Java heap memory space. As you can see in our sample thread stack, Java's heap Oldgen exceeds the maximum value!

?

123456789

Heap PSYoungGen      total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)  eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)  from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)  to   space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000) PSOldGen        total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)  object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000) PSPermGen       total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)  object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

I hope this article will help you understand the basic information of the Hotspot VM thread stack. The next article will provide you with a thread stack overview and analysis of IBM VMS.

Please free Humben to express your views or questions.

Reference: How to parse the thread stack--the first part, how to parse the thread stack--Part Two & how to parse the thread stack--The third section comes from our Java EE support Patterns & Java Tuto Rial blog on the JCG collaborators Hugues Charbonneau.

JVM: How to parse the thread stack