Join me in a face exam-linux threading Programming (7)

It has been difficult to debug multi-threading, on-line search learned a method of multi-threaded debugging

I have always felt that the multi-threaded debugging under Linux is very troublesome, because generally larger program threads will be many, through the GDB info thread command to see all the system calls, see no detailed method, at least I see this ... If you follow each thread with the thread ID to go to BT, it is quite a pain, especially if you info thread saw nearly a few threads of the time t_t. And most of the time, waiting to reproduce the problem or restart the program is very expensive, It is important to look at the thread stack in case the program is running.

Actually have pstack this command question is very simple, pstack concrete usage man pstack.

Gdb+pstack:

Get the program PID via PS.

GDB processname PID, debug in Run program.

The info thread output is similar to the thread information:

(GDB) Info thread
0x2547b90 Thread (LWP 22267)? 0x00795410 in __kernel_vsyscall ()?
22268 Thread 0x5c6eb90 (LWP)? 0x00795410 in __kernel_vsyscall ()?
0x2f48b90 Thread (LWP 22270)? 0x00795410 in __kernel_vsyscall ()?
Thread 0x94e3b90 (LWP 22272)? 0x00795410 in __kernel_vsyscall ()?
22287 Thread 0x3949b90 (LWP)? 0x00795410 in __kernel_vsyscall ()?
22393 Thread 0x3d4ab90 (LWP)? 0x00795410 in __kernel_vsyscall ()?
0x414bb90 Thread (LWP 22394)? 0x00795410 in __kernel_vsyscall ()

From the above, you can only see the system call, it is difficult to determine what each thread is doing, if you have another way to visually see in gdb what each thread is running, to tell:)

My approach is to use Pstack to check, of course, if your Linux is installed on this tool, but most of Linux should be there.

Pstack PID, you will get a lot of information:

Pstack 22266
Thread 0x2547b90 (LWP 22267):
#0? 0x00795410 in __kernel_vsyscall ()
#1? 0x00b5eaa6 in Nanosleep () from/lib/libc.so.6
#2? 0X00B5E8CF in Sleep () from/lib/libc.so.6
#3? 0X08134A3A in Removelog (void*) ()
#4? 0x0085e832 in Start_thread () from/lib/libpthread.so.0
#5? 0X00B9EE0E in Clone () from/lib/libc.so.6
Thread (thread 0x5c6eb90 (LWP 22268)):
#0? 0x00795410 in __kernel_vsyscall ()
#1? 0x00b5eaa6 in Nanosleep () from/lib/libc.so.6
#2? 0X00B5E8CF in Sleep () from/lib/libc.so.6
#3? 0X08134A3A in Removelog (void*) ()
#4? 0x0085e832 in Start_thread () from/lib/libpthread.so.0
#5? 0X00B9EE0E in Clone () from/lib/libc.so.6

For example, I see this thread, the other thread's cprocess::readdevices are locked, because thread safety cprocess::readdevices is lock, many other threads are stuck in Cprocess::wrlock () () or Cprocess::rdlock () (), and the following thread is obviously the culprit, don't ask me why, because I can't explain why =2 ...

Thread (thread 0x631ffb90 (LWP 1161)):
#0? 0x00795410 in __kernel_vsyscall ()
#1? 0x00b979d1 in select () from/lib/libc.so.6
#2? 0X0812F1A2 in Sleep (int) ()
#3? 0x080598e6 in Cprocess::readdevices (fluxcontrol::D evicearray const&) ()
#4? 0X081281A1 in Fluxcontrolimpl::readdevices (fluxcontrol::D evicearray const&) ()
#5? 0x0813b2f2 in _0rl_lcfn_28d700e3085a57ca_30000000 (omnicalldescriptor*, omniservant*) ()
#6? 0x0093b14d in Omnicallhandle::upcall (omniservant*, omnicalldescriptor&) () from/export/home/tools/omniorb-4.0.7 /lib/libomniorb4.so.0
#7? 0x0813c48e in Fluxcontrol::_impl_fluxdeviceconf::_dispatch (omnicallhandle&) ()
#8? 0X009269BF in Omni::omniorbpoa::d Ispatch (omnicallhandle&, omnilocalidentity*) () from/export/home/tools/ omniorb-4.0.7/lib/libomniorb4.so.0
#9? 0x0090a963 in omnilocalidentity::d ispatch (omnicallhandle&) () from/export/home/tools/omniorb-4.0.7/lib/ libomniorb4.so.0
#10 0x0095aa92 in Omni::giop_s::handlerequest () () from/export/home/tools/omniorb-4.0.7/lib/libomniorb4.so.0

Find here in the Pstack can not be associated to the specific code, this time back to see Gdb,gdb is visible:).

The problem threads above Pstack are thread (thread 0x631ffb90 (LWP 1161)), but when you use GDB to view the PID of the program, the thread number is not the same as that of thread 23, assuming that other threads that are not locked will compete, and the serial number will change. , but the thread 0x631ffb90 is immutable, so gdb processname the PID, then info thread looks at the thread and then searches the output thread for 0x631ffb90, and you'll find the corresponding thread, Then execute the thread ThreadID in GDB and then you can go to this thread to view it.

Original address: http://dy601601.blog.163.com/blog/static/12978026320125353956724/

Join me in a face exam-linux threading Programming (7)