Debugging of multithreaded deadlock problems under Linux

Recently write service, often single process, multi-threaded, add a variety of locks, very worried about the deadlock problem, specifically learned about the diagnosis of the deadlock problem.

Deadlock (Deallocks): Refers to two or more than two processes (threads) in the course of execution, because of the contention for resources caused by a mutual waiting phenomenon, if there is no external force, they will not be able to proceed. At this point, the system is in a deadlock state or the system generates a deadlock, and these processes (threads) that are always waiting on each other are called deadlock processes (threads). Because resource consumption is mutually exclusive, when a process requests resources, so that the process (threads) with no external assistance, will never be allocated the necessary resources and can not continue to run, which creates a special phenomenon of deadlock.

A cross-lock deadlock scenario in which two or more threads in the execution program are permanently blocked (waiting), and each thread waits for resources that are consumed by other threads and blocked. For example, if thread 1 locks record A and waits for record B, and thread 2 locks record B and waits for record A, a deadlock occurs on two threads. In computer systems, if the system's resource allocation policy is inappropriate, it may be more common that programmers write programs with errors and so on, which can lead to a process due to improper competition resources to create a deadlock phenomenon.

Four necessary conditions for generating a deadlock

(1) Mutex condition: A resource can only be used by one process (thread) at a time.
(2) Request and hold condition: When a process (thread) is blocked by a request for resources, the acquired resources are persisted.
(3) No deprivation condition: the resources that this process (thread) has obtained cannot be forcibly stripped until the end of use.
(4) Cyclic wait condition: a cyclic waiting resource relationship between multiple processes (threads) is formed.

You can use the Pstack and GDB tools to analyze the deadlock program.

A brief introduction to Pstack on the Linux platform

Pstack is a useful tool for Linux (such as Red Hat Linux systems, Ubuntu Linux systems, etc.), and its function is to print out the stack information for this process. You can output the call stack for all threads.

A brief introduction to GDB on the Linux platform

GDB is a powerful UNIX program debugging tool released by the GNU Open source organization. The Linux system contains the GNU debug program GDB, a debugger for debugging C and C + + programs. You can enable program developers to observe the internal structure and memory usage of the program while the program is running.

Some of the main features that GDB provides are as follows:

1 run the program, set the parameters and environment that can affect the operation of the program;

2 The control program stops running under the specified conditions;

3 when the program stops, you can check the status of the program;

4 when the program crash, you can check the core file;

5 can modify the program error, and re-run the program;

6 You can dynamically monitor the value of variables in the program;

7 You can step through the code to observe the running state of the program.

Example:

A service process in my project has 45 threads running in parallel. A program exception is found and can be

The pstack process number looks at the stack information for each thread.

The Pstack Process number command needs to be executed several times in succession. View the stack of function call relationships for each thread for analysis. When the process hangs, multiple uses, the deadlock of the thread will remain in the state of the lock, to determine that some threads have not changed, has been in the state of the lock. Then these threads are likely to be deadlocked. If you suspect which threads are deadlocked, you can use GDB to further attach threads and analyze them.

As above two graphs, we found that thread 18 and thread are all in the same lock state, waiting for the same lock, continuous observation is the phenomenon.

So: Execute the command GDB attach 9368 (process number) into the GDB debug terminal.

Run: (GDB) Info thread

System calls can cause deadlocks,

Syslog

System

Memory release

Exit

Debugging of multithreaded deadlock problems under Linux