GDB debugging multi-Threaded Threads Summary

GDB debugging multi-Threaded Threads Summary

SOURCE https://www.cnblogs.com/jingzhishen/p/4324071.html

one, multi-threaded debugging
1. Multi-threaded debugging, the most important of several commands:
Info Threads View the threads of the current process.
GDB assigns an ID to each thread, which is used when the thread is being manipulated.
Previous * is the thread that is currently being debugged.
Thread <ID> Switches Debug threads to the specified ID of the thread.
Break file.c:100 thread all sets a breakpoint on line 100th of the file.c file for all threads passing through here.
Set Scheduler-locking Off|on|step
When debugging a currently debugged thread using the step or Continue command, the other threads are executed concurrently,
How do you just let the debugger execute it?
This requirement can be achieved with this command.
Off does not lock any threads, that is, all threads are executed, which is the default value.
On only the currently debugged program will execute.
Step in the case of a single step, except for the next function
(people familiar with the situation may know that this is actually a setting breakpoint and then continue the behavior),
Only when the front-line will execute.
Thread apply ID1 ID2 command to have one or more threads execute GDB commands
The thread apply All command lets all the debugged threads execute GDB command commands.

2. Examples of use:
Thread Generation Notification: GDB gives a hint when a new thread is generated
(GDB) R
Starting program:/root/thread
[New Thread 1073951360 (LWP 12900)]
[New Thread 1082342592 (LWP 12907)]---The following three new threads are created
[New Thread 1090731072 (LWP 12908)]
[New Thread 1099119552 (LWP 12909)]

View Threads: Use info threads to view the threads that are running.
(GDB) Info threads
4 Thread 1099119552 (LWP 12940) 0xffffe002 in?? ()
3 Thread 1090731072 (LWP 12939) 0xffffe002 in?? ()
2 Thread 1082342592 (LWP 12938) 0xffffe002 in?? ()
* 1 Thread 1073951360 (LWP 12931) Main (argc=1, argv=0xbfffda04) at thread.c:21
(GDB)
Note that the line header is the thread ID number assigned by GDB, and the ID number is used when the thread is switched.
Additionally, an asterisk at the beginning of the line identifies the currently active thread
To switch Threads:
Use thread threadnumber to switch threadnumber to the thread ID number mentioned above.
The following example shows switching the active thread from 1 to 4.
(GDB) Info threads
4 Thread 1099119552 (LWP 12940) 0xffffe002 in?? ()
3 Thread 1090731072 (LWP 12939) 0xffffe002 in?? ()
2 Thread 1082342592 (LWP 12938) 0xffffe002 in?? ()
* 1 Thread 1073951360 (LWP 12931) Main (argc=1, argv=0xbfffda04) at thread.c:21
(GDB) Thread 4
[Switching to thread 4 (thread 1099119552 (LWP 12940)]) #0 0xffffe002 in?? ()
(GDB) Info threads
* 4 Thread 1099119552 (LWP 12940) 0xffffe002 in?? ()
3 Thread 1090731072 (LWP 12939) 0xffffe002 in?? ()
2 Thread 1082342592 (LWP 12938) 0xffffe002 in?? ()
1 Thread 1073951360 (LWP 12931) Main (argc=1, argv=0xbfffda04) at thread.c:21
(GDB)
These are some of the basic commands for debugging multithreading with GDB.
In addition, GDB provides breakpoint settings on threads and commands for issuing commands to specified or all threads

Second, debug macros
Under GDB, we cannot print the macro definition because the macro is precompiled.
But we still have a way to debug the macro, which requires GCC mates.
When you compile the program in GCC, add
-ggdb3parameter so that you can debug the macro.

Alternatively, you can use the following GDB macro debugging commands to view the related macros.
Info macro to see which files the macro is referenced in and what the macro definition is.
Macro to see how the macros unfold.

Third, the source file
GDB, you are prompted not to find the source file.
The following checks are required:
Compiles whether the programmer added the-G parameter to contain the debug information.
The path is set correctly.
Use the GDB Directory command to set the directory for the source files.

Below is an example of a debug/bin/ls (under Ubuntu)
$ apt-get Source Coreutils
$ sudo apt-get install Coreutils-dbgsym
$ gdb/bin/ls
GNU gdb (GDB) 7.1-ubuntu
(GDB) List main
1192 Ls.c:no such file or directory.
In LS.C
(gdb) directory ~/src/coreutils-7.4/src/
Source directories searched:/home/hchen/src/coreutils-7.4: $cdir: $CWD
(GDB) List main
1192}
1193}
1194
1195 int
1196 Main (int argc, char **argv)
1197 {
1198 int i;
1199 struct pending *thispend;
N_files int;
1201

Iv. Conditional Breakpoints
The conditional breakpoint is the syntax:
Break [where] if [condition]
This breakpoint really works.
Especially in a loop or recursion, or to monitor a variable.
Note that this setting is in gdb, except that GDB will help you check if the condition is satisfied every time you pass that breakpoint.

Five, command-line parameters
Sometimes we need to debug a program that requires command-line arguments, and there are three ways to do it:
GDB command-line-args parameter
The set args command in the GDB environment.
The run parameter in the GDB environment

Vi. Variables for GDB
Sometimes, when you debug a program, we don't just look at the variables at run time,
We can also directly set the variables in the program, to simulate some difficult to appear in the test, compare some errors,
or switch's branch statement. Use the SET command to modify variables in a program.
Also, do you know that there can be variables in GDB?
Just like the shell, the variables in GDB start with $, like you want to print an array of elements, you can do this:
(GDB) Set $i = 0
(GDB) P a[$i + +]
... #然后就一路回车下去了
Of course, here is just an example, which indicates that the variables of the program and the GDB variables can be interacted with.

Seven, x command
Perhaps, you like to use the P command.
Therefore, when you do not know the variable name, you may be unprepared, because the P command always requires a variable name.
The x command is used to view the memory, and in GdB, "Help X" You can view its assistance.
x/x with hexadecimal output
X/D Output in decimal
X/C output with single character
X/i Disassembly – Typically, we use x/10i $ip-20来 to view the current assembly ($IP is the instruction register)
X/S output as a string

Viii. command Commands
How to automate debugging.
Here we introduce command command, simple understanding, it is a set of GDB command package, a bit like word processing software "macro."
Here is an example:
(GDB) Break func
Breakpoint 1 at 0x3475678:file test.c, line 12.
(GDB) Command 1
Type commands for when Breakpoint 1 are hit, one per line.
End with a line saying just "End".
>print arg1
>print arg2
>print Arg3
>end
(GDB)
When our breakpoint arrives, the three commands in the command are automatically executed, and the three parameter values of Func are called out.

Multi-threaded viewing tool under Linux (Pstree, PS, Pstack)

Http://www.cnblogs.com/aixingfou/archive/2011/07/28/2119875.html

http://blog.csdn.net/nancygreen/article/details/14226925

First, we introduce the basic command of GDB multi-threaded debugging. Info Threads shows all threads currently available for debugging, each with a GDB-assigned ID, which is used when the thread is being manipulated. Previous * is the thread that is currently being debugged. The thread ID Toggles the currently debugged thread for the specified ID. Break thread_test.c:123 Thread all sets a breakpoint on the corresponding line in all threads thread apply ID1 ID2 command Have one or more threads execute GDB command commands. The thread apply all command lets all the debugged threads execute GDB command commands. The set scheduler-locking off|on|step estimate is that anyone who has actually used multithreaded debugging can find that other threads are executing at the same time using the step or Continue command to debug the currently debugged thread. How do you just let the debugger execute it? This requirement can be achieved with this command. Off does not lock any threads, that is, all threads are executed, which is the default value. On only the currently debugged program will execute. Step in a single step, in addition to the case of next over a function (the person familiar with the situation may know that this is actually a set breakpoint and then continue behavior), only when the front-thread will execute.

GDB has the following support for debugging multi-threaded threads:

• Thread Generation Notification: GDB gives a hint when a new thread is generated

(GDB) R
Starting program:/root/thread
[New Thread 1073951360 (LWP 12900)]
[New Thread 1082342592 (LWP 12907)]---The following three new threads are created
[New Thread 1090731072 (LWP 12908)]
[New Thread 1099119552 (LWP 12909)]

• View threads: Use to view the threads that are running. Info threads

(GDB) Info threads
4 Thread 1099119552 (LWP 12940) 0xffffe002 in?? ()
3 Thread 1090731072 (LWP 12939) 0xffffe002 in?? ()
2 Thread 1082342592 (LWP 12938) 0xffffe002 in?? ()
* 1 Thread 1073951360 (LWP 12931) Main (argc=1, argv=0xbfffda04) at thread.c:21
(GDB)

Note that the blue text at the beginning of the line is the thread number assigned by GDB, which is used when switching threads, instead of the green number indicated above.

In addition, the red asterisk at the beginning of the line identifies the currently active thread

• Toggle Thread: Use thread threadnumber to switch threadnumber to the thread number mentioned above. The following example shows switching the active thread from 1 to 4.

(GDB) Info threads
4 Thread 1099119552 (LWP 12940) 0xffffe002 in?? ()
3 Thread 1090731072 (LWP 12939) 0xffffe002 in?? ()
2 Thread 1082342592 (LWP 12938) 0xffffe002 in?? ()
* 1 Thread 1073951360 (LWP 12931) Main (argc=1, argv=0xbfffda04) at thread.c:21
(GDB) Thread 4
[Switching to thread 4 (thread 1099119552 (LWP 12940)]) #0 0xffffe002 in?? ()
(GDB) Info threads
* 4 Thread 1099119552 (LWP 12940) 0xffffe002 in?? ()
3 Thread 1090731072 (LWP 12939) 0xffffe002 in?? ()
2 Thread 1082342592 (LWP 12938) 0xffffe002 in?? ()
1 Thread 1073951360 (LWP 12931) Main (argc=1, argv=0xbfffda04) at thread.c:21
(GDB)

These are some of the basic commands for debugging multithreading with GDB. In addition, GDB provides breakpoint settings on threads and commands for issuing commands to specified or all threads.

The first contact with GDB under the multi-threaded debugging, often ignores the concept of the active thread in GDB. In general, when debugging with GDB, only one thread is the active thread, and if you want to get the output from another thread, you must use the thread command to switch to the specified thread to debug or observe the output.

Finally, I introduce a multi-threaded debugging and resolution I recently met.

The basic problem is that in a Linux environment, debugging multithreaded programming is not normal, and info threads does not see multi-threaded information.
I first use the command maintenance print Target-stack looked at the load of target, found that the target "multi-thread" is not loaded, with GDB debugging GDB, found in the function check_for_ Thread_db returned td_nolibthread when calling function td_ta_new in libthread_db, so the target "multi-thread" was not loaded.
At the time I suspect is not libpthread have a problem, so check a bit found the problem, the environment of Libpthread is strip, I think it may be that the impact of td_ta_new on the Libpthread symbolic information acquisition. When I changed a libpthread without strip, the problem was solved.
Finally, my solution is to copy a. Debug version of Libpthread to the Lib directory, the problem is resolved.
Multi-threaded if dump, more than a segment error, generally involved in memory illegal read and write. This can be done by using the following command to turn on the system switch so that it can generate a core file when it dies.
Ulimit-c Unlimited
In this case, you can see the Core.pid (PID is the process number) file in the current directory when you die. Then use GDB:
GdB./bin./core.pid
After entering, use the frame command when using BT to view the case of the dead stack.

There is a thread in it to stop, and not die, this is generally a deadlock or a message to accept the timeout problem (heard that, not met). In this case, you can use:
Gcore PID (PID number of the debug process)
Manually generate the core file, using Pstack (which looks bad on Linux) to view the stack . If you can not see it, look at the code carefully to see if it is in the If,return,break,continue this statement operation is to forget to unlock, as well as nested lock problem, need to be analyzed clearly.

Original Address http://www.linuxforum.net/forum/gshowflat.php?Cat=&Board=program&Number=692404&page=0&view=collapsed

A simple way to analyze deadlocks on Linux pstack simple introduction to Linux platforms

Pstack is a useful tool for Linux (such as Red Hat Linux systems, Ubuntu Linux systems, etc.), and its function is to print out the stack information for this process. You can output the call stack for all threads.

123456789101112131415161718192021222324252627282930313233343536373839

[email protected]:~/tmp/sisuo$ sudo gdb -q ./lock 13011Reading symbols from ./lock...done.Attaching to program: /home/yingc/tmp/sisuo/lock, process 13011Reading symbols from /lib/i386-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug//lib/i386-linux-gnu/libpthread-2.19.so...done.done.[New LWP 13015][New LWP 13014][New LWP 13013][New LWP 13012][Thread debugging usinglibthread_db enabled]Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".Loaded symbols for/lib/i386-linux-gnu/libpthread.so.0Reading symbols from /lib/i386-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/i386-linux-gnu/libc-2.19.so...done.done.Loaded symbols for/lib/i386-linux-gnu/libc.so.6Reading symbols from /lib/ld-linux.so.2...Reading symbols from /usr/lib/debug//lib/i386-linux-gnu/ld-2.19.so...done.done.Loaded symbols for/lib/ld-linux.so.20xb778bc7c in __kernel_vsyscall ()(gdb) info threads  Id   Target Id         Frame  5    Thread 0xb759db40 (LWP 13012) "lock"0xb778bc7c in __kernel_vsyscall ()  4    Thread 0xb6d9cb40 (LWP 13013) "lock"0xb778bc7c in __kernel_vsyscall ()  3    Thread 0xb659bb40 (LWP 13014) "lock"0xb778bc7c in __kernel_vsyscall ()  2    Thread 0xb5d9ab40 (LWP 13015) "lock"0xb778bc7c in __kernel_vsyscall ()* 1    Thread 0xb759e700 (LWP 13011) "lock" 0xb778bc7c in __kernel_vsyscall ()(gdb) t 5[Switching to thread5 (Thread 0xb759db40 (LWP 13012))]#0  0xb778bc7c in __kernel_vsyscall ()(gdb) bt#0  0xb778bc7c in __kernel_vsyscall ()#1  0xb775b792 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144#2  0xb77571e3 in _L_lock_851 () from /lib/i386-linux-gnu/libpthread.so.0#3  0xb7757020 in __GI___pthread_mutex_lock (mutex=0x804a060 <mutex2>) at ../nptl/pthread_mutex_lock.c:79#4  0x08048738 in func1 () at lock.cpp:18#5  0x080487ee in thread1 (arg=0x0) at lock.cpp:43#6  0xb7754f16 in start_thread (arg=0xb759db40) at pthread_create.c:309#7  0xb768b9fe in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129(gdb)

Pstack on my machine, it looks bad.

123456789101112131415161718

[email protected]:~/tmp/sisuo$ pstack 13011Could not attach to target 13011: Operation not permitted.detach: No such process[email protected]:~/tmp/sisuo$ sudo pstack 13011[sudo] password foryingc:13011: ./lock(No symbols found in )(No symbols found in /lib/i386-linux-gnu/libc.so.6)(No symbols found in /lib/ld-linux.so.2)0xb778bc7c: _fini + 0x2caec (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe00x09279010: _fini + 0x123054c (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe00x09279010: _fini + 0x123054c (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe00x09279010: _fini + 0x123054c (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe00x09279010: _fini + 0x123054c (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe00x09279010: _fini + 0x123054c (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe00x09279010: _fini + 0x123054c (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe00x09279010: _fini + 0x123054c (b759db40, 1, b778bc6c, 15a6a100, e608f43e, 0) + ffffffe0

GDB debugging multi-Threaded Threads Summary