如何定位死迴圈或高CPU使用率(linux)

如何定位死迴圈或高CPU使用率(linux)

 確定是CPU過高

使用top觀察是否存在CPU使用率過高現象

找出線程

對CPU使用率過高的進程的所有線程進行排序

ps H -e -o pid,tid,pcpu,cmd --sort=pcpu |grep xxx

得到如下結果,其中線程2909使用了7.8%的CPU. 

2907 2913 0.0 ./xxx 

2907 2909 7.8 ./xxx

也可以通過查看/proc中的資訊來確定高CPU線程. 列印了4列,線程ID,線程名,使用者時間和核心程式的時間(排名未分先後) 

awk '{print $1,$2,$14,$15}' /proc/2907/task/*/stat

找出調用棧

使用gdb attach nmsagent所在的進程,在gdb中使用 info threads顯示所有線程

gdbgdb>attach 2907gdb>info threads

得到如下結果,可以發現2909線程的編號是12

  13 Thread 0xad5f2b70 (LWP 2908)  0x004ef0d7 in mq_timedreceive () from /lib/tls/i686/cmov/librt.so.1  12 Thread 0xad58eb70 (LWP 2909)  0x006e0422 in __kernel_vsyscall ()  11 Thread 0xad52ab70 (LWP 2910)  0x006e0422 in __kernel_vsyscall ()  10 Thread 0xad4f8b70 (LWP 2911)  0x006e0422 in __kernel_vsyscall ()  9 Thread 0xad4c6b70 (LWP 2912)  0x006e0422 in __kernel_vsyscall ()  8 Thread 0xad3feb70 (LWP 2913)  0x004ef0d7 in mq_timedreceive () from /lib/tls/i686/cmov/librt.so.1  7 Thread 0xace08b70 (LWP 2914)  0x004ef0d7 in mq_timedreceive () from /lib/tls/i686/cmov/librt.so.1  6 Thread 0xac607b70 (LWP 2915)  0x006e0422 in __kernel_vsyscall ()  5 Thread 0xac5e6b70 (LWP 2916)  0x006e0422 in __kernel_vsyscall ()  4 Thread 0xac361b70 (LWP 2917)  0x006e0422 in __kernel_vsyscall ()  3 Thread 0xac2fdb70 (LWP 2918)  0x006e0422 in __kernel_vsyscall ()  2 Thread 0xac1fcb70 (LWP 2919)  0x004ef0d7 in mq_timedreceive () from /lib/tls/i686/cmov/librt.so.1* 1 Thread 0xb78496d0 (LWP 2907)  0x006e0422 in __kernel_vsyscall ()

使用thread 切換線程,使用bt顯示線程棧

gdb>thread 12gdb>bt

得到如下線程棧

#0  0x006e0422 in __kernel_vsyscall ()#1  0x001cca26 in nanosleep () from /lib/tls/i686/cmov/libc.so.6#2  0x001fc2dc in usleep () from /lib/tls/i686/cmov/libc.so.6#3  0x0806b510 in OspTaskDelay ()#4  0x0805c710 in CDispatchTask::NodeMsgSendToSock() ()#5  0x0805cc74 in DispatchTaskEntry ()#6  0x0806a8e9 in OspTaskTemplateFunc(void*) ()#7  0x00d4780e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8  0x002027ee in clone () from /lib/tls/i686/cmov/libc.so.6

 ps + strace

得到進程ID 21465

ps -e |grep cmu 4996 ?        00:00:25 cmu_fjga_sp321465 pts/5    00:08:10 cmu

得到線程時間, 其中最占CPU的是 EpollRecvTask 21581

ps -eL |grep 21465 

21465 21579 pts/5 00:00:00 CamApp 

21465 21580 pts/5 00:00:00 TimerMan Task 

21465 21581 pts/5 00:09:02 EpollRecvTask 

21465 21582 pts/5 00:00:00 

使用 strace -p 21581 得到線程棧