Linux Multi-threaded debugging (memory footprint, dead loop, high CPU usage ...) ）

Article Source: http://www.cnblogs.com/cy568searchx/archive/2013/10/28/3391790.html

Your software stops service at some point, CPU usage reaches 100%+, one possible cause is a dead loop, assuming there is a potential dead loop in the program somewhere, and under certain conditions, this article uses an example to locate the location where the dead loop occurs.
When there is a dead loop somewhere in the program, the usual way to locate the problem and narrow it is to add log to the suspect code, or comment out the suspect code, which is good for the program that is prone to reproduce the problem, but it is difficult to debug for "occasional" problems because it is difficult to reproduce the program failure. The debugging process described in this article is precisely in this case, assuming that the problem has arisen, we require environmental protection site, that is, the problem of the program is still in operation.

1. We first need to know which thread is out of the question:
First check the PID of the problem process, for example

[email protected]: ~/mass4/src/icdn/src$ Ps-ef |        grep ICDN ovtsvn 11065 1 50 11:57? 00:00:07./icdn ovtsvn 11076 10971 0 11:57 pts/2 00:00:00 grep[email protected]:~/mass4/src/icdn/src$[email protecte d]:~/mass4/src/icdn/src$

Then the top command looks at the thread information:
Top-h-P 11065

pid user      pr  ni  virt  res   shr s %cpu %mem    time+  command                                                                     11073 ovtsvn    25   0   325m 3980 2236 r  100  0.4   1:40.84 icdn                                                                        11065 ovtsvn    18   0   325m 3980 2236 s    0  0.4   0:00.01  icdn                                                                       11066  ovtsvn    18   0  325m 3980 2236 s     0  0.4   0:00.00 icdn                                                                        11067 ovtsvn    15   0  325m 3980 2236  s    0  0.4   0:00.00 icdn                                                                        11068 ovtsvn    15    0  325m 3980 2236 s    0  0.4   0:00.00 icdn                                                                        11069 ovtsvn 180 325m  39802236 S 00.40:00.00 icdn 11070 ovtsvn 180 325m 39802236  s 00.40:00.00 icdn 11071 ovtsvn 220 325m 39802236 s 00.40:00.00  icdn 11072 ovtsvn 150 325m 39802236 R 00.40:00.00 icdn 

As can be seen from the above, the problem thread pid is 11073

2. Next, we use GDB to attach the target process
: gdb icdn 11065
in GdB, the outgoing status:

(GDB)  info threads   9 Thread 47056948181264  (lwp 11066)    0x00002acc4a3dec91 in nanosleep  ()  from /lib/libc.so.6   8  thread 47056956573968  (lwp 11067)   0x00002acc4a406fc2 in select  ()  from /lib/libc.so.6   7 Thread 47056964966672  (lwp 11068)    0x00002acc4a3dec91 in nanosleep  ()  from /lib/libc.so.6   6  thread 47056973359376  (lwp 11069)   0x00002acc4a3dec91 in nanosleep  ()  from /lib/libc.so.6   5 Thread 47056981752080  (lwp 11070)    0x00002acc4a3dec91 in nanosleep  ()  from /lib/libc.so.6   4  thread 47056990144784  (lwp 11071)   0x00002acc4a40e63c in recvfrom  ()  from /lib/libc.so.6   3 thread 47057194060048  (lwp 11072)    0x00002acc4a406fc2 in select  ()  from /lib/libc.so.6   2 thread  47057226893584  (lwp 11073)   CSendFile::SendFile  (this=0x2acc5d4aff40, [ Email protected])      at .. /src/csendfile.cpp:101   1 thread 47056939784832  (LWP 11065)    0x00002acc4a3dec91 in nanosleep  ()  from /lib/libc.so.6  (GDB)  

GDB already lists the functions that each thread is executing, and we need more information, remembering the 11073 line header, which GDB assigns to the thread ID, here is 2, then performs the switchover:

(GDB) thread 2 [Switching to thread 2 (thread 47057226893584 (LWP 11073)]] #0 csendfile::sendfile (this=0x2acc5d4aff40, [E Mail protected]) at.. /SRC/CSENDFILE.CPP:101 101 while (1) (GDB)

BT a bit:

(GDB) bt #0 csendfile::sendfile (THIS=0X2ACC5D4AFF40, [email protected]) at: /src/csendfile.cpp:101 #1 0x000000000040592e in Cicdn::taskthread (PPARAM=0X7FFF617EAFE0) at.. /src/cicdn.cpp:128 #2 0x00002acc4a90b73a in Start_thread () from/lib/libpthread.so.0 #3 0X00002ACC4A40D6DD in Clone () From/lib/libc.so.6 #4 0x0000000000000000 in?? ()

Take a look at the code for 101 lines:

(GDB) L 98 int csendfile::sendfile (const string& pathname) 99{+ int n;101 while (1)102 { 103 n++;104}//read file and send

Now we've got the code location for the problem, and the loop here is for illustrative purposes only.
Finally, don't forget detach ()

After you have debugged the specified process, you can run the detach command to let GDB free the process and the process to continue running. The detach does not repeat when the carriage returns. When the detach is finished, the process and GDB are no longer relevant, and GDB can attach other processes.

Linux Multi-threaded debugging (memory footprint, dead loop, high CPU usage ...) ）