Tracking deadlocks in a WebSphere application Server V6.1 Application

Learn how to use the thread Dump tool in Ibm®websphere®application Server V6.1 to understand your system environment, check for deadlocks, and extract information to help you avoid or resolve your application's deadlock situations.

Introduction

A deadlock occurs when two or more threads form a cyclic dependency between each other. For example, if thread A is waiting for thread B to wait while thread B is waiting for thread A, a deadlock occurs. Once this happens, neither thread a nor thread B can make any progress, since the two threads are now suspended indefinitely. Why would someone create such a system? Of course, you don't intend to do this, but because of the large number of threads and complex transactions, this can easily happen.

This article describes how to use the IBM WebSphere application Server V6.1 thread Dump tool to check the system to determine if a deadlock has occurred. To help understand this, a real example is referenced and described in the article. In this example, the enterprise application runs in the Web container, while the OSGi package provides access to the protocol. When traffic suddenly rises to a sufficient level, the application running on the server hangs and is extremely serious, and the only task that can be performed is to issue a kill-9 command to terminate the process. This article describes how to discover and resolve this typical deadlock scenario, which involves methods that can be used to avoid or debug similar conditions in your application.

Problem

In many protocols, a state machine is used to manage the status of each protocol connection. These state machines are sometimes referred to as connecting finite state machines (Connection finite states MACHINES,CFSM). The state machine requires atomic manipulation. This means that different parts of the state machine cannot be updated at different times, and when an update occurs, the entire state machine must be updated in the operation, so the next operation is applied to the new state, and so on. In order to achieve this, the state is designed for any time only one thread can perform operations on the state machine. The access methods for the state machine are all synchronized. The thread that wants to access the state machine is blocked before the thread of the current access state machine completes the method call.

In such a sample protocol, the State is changed at any time when a packet is received, transmitted, or the operation times out. Other conditions, such as starting, stopping, and disconnecting, can also cause the state machine to change. These inputs of the state machine make it a high-risk area for searching, especially if any number of threads can cause these events to occur.

The first time this problem occurred in this application, the application server lockup was very serious, no logging occurred, and even the console could not be opened. In fact, the WebSphere application Server process does not have TCP traffic in and out. This is a typical symptom of a deadlock. CPU utilization is zero, no events occur, and no requests are received. The only tool available for debugging is the thread dump of the WebSphere application Server process.

A thread dump (or core file) is generated by a name in the following format:

Javacore.date.time.id.txt

For example: Javacore.20070919.204717.27050.txt

In some cases, a thread dump is automatically created for the Java™virtual Machine (JVM), such as a situation in which WebSphere application Server stops in a way other than ordinary stopserver requests. Thread dumps can also be triggered by signaling to the WebSphere application Server process. For example, to generate a thread dump in a UNIX® environment, you can run this command:

Kill-3 process_id

Where process_id is the process ID of the WebSphere application Server JVM. Thread dumps can also be created using Wsadmin. To force a thread dump using the Wsadmin command prompt, issue the following command:

Wsadmin

Wsadmin>set JVM [$AdminControl completeobjectname TYPE=JVM, process=server1,*]

Wsadmin> $AdminControl Invoke $JVM dumpthreads

This will create a file similar to Javacore.20071012.080508.4252.txt in Was_profile_root.

When you open a thread dump, you see the following: The stack trace for each single thread in the WebSphere application Server. But more importantly, it also lists all the locks in the system. Very good!

Thread Dump Content

A thread dump is a simple text file that you can open using any text editor. You can find a lot of meaningful and useful information in it, see the description later.