High Linux cpu usage debugging

High Linux cpu usage debugging

After the top command is used to debug the program today, the cpu usage of the program is found to be very high and it has been running at 99. This is terrible, so we can debug the program.

Use the top command to obtain the following results:

Pid user pr ni virt res shr s % CPU % mem time + COMMAND
1997 root 20 0 358 m 71 m 3208 S 99.1 81: 7.2 test
1 root 20 0 24332 2044 S 1176 0.0. 36 init
2 root 20 0 0 0 S 0.0 0.0. 00 kthreadd
3 root 20 0 0 0 S 0.0 0.0 0: 46. 51 ksoftirqd/0
4 root 20 0 0 0 S 0.0 0.0 0: 38. 53 kworker/0: 0
6 root RT 0 0 0 S 0.0 0.0. 00 migration/0

We can see that the PID of test is 1997

Then, view the cpu usage of the thread in test.

Use the top-H-p 1997 command

Root @ bkjia :~ # Top-H-p 1997

Top-17:19:47 up 15 days, 34 min, 4 users, load average: 1.02, 1.06, 1.06
Tasks: 8 total, 1 running, 7 sleeping, 0 stopped, 0 zombie
Cpu (s): 99.3% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.3% si, 0.3% st
Mem: 1017924 k total, 887500 k used, 130424 k free, 85928 k buffers
Swap: 0 k total, 0 k used, 0 k free, 351280 k cached

Pid user pr ni virt res shr s % CPU % mem time + COMMAND
2005 root 20 0 358 m 71 m 3208 R 99.3 7.2 83: 44.47 test
2001 root 20 0 358 m 71 m 3208 S 0.3 0: 08. 66 test
2004 root 20 0 358 m 71 m 3208 S 0.3 0: 19. 61 test
1997 root 20 0 358 m 71 m 3208 S 0.0 0: 26. 24 test
1999 root 20 0 358 m 71 m 3208 S 0.0 0: 00. 06 test
2000 root 20 0 358 m 71 m 3208 S 0.0 0: 00. 00 test
2002 root 20 0 358 m 71 m 3208 S 0.0 0: 05. 86 Server Listen (2
2003 root 20 0 358 m 71 m 3208 S 0.0 0: 00. 00 Server Accept (2

The maximum CPU usage of a thread is 2005.

Use the gdb icdn 2005 command

Root @ bkjia :~ # Gdb icdn 2005.
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2. 1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3 +: gnu gpl version 3 or later This is free software: you are free to change and redistribute it.
There is no warranty, to the extent permitted by law. Type "show copying"
And "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu ".
For bug reporting instructions, please see:
<Http://bugs.launchpad.net/gdb-linaro/>...
Icdn: No such file or directory.
Attaching to process 2005

Warning: process 2005 is a cloned process
Reading symbols from/home/bkjia/test/snmptrapd... done.
Reading symbols from/usr/lib/libnetsnmp. so.30...... done.
Loaded symbols for/usr/lib/libnetsnmp. so.30
Reading symbols from/lib/x86_64-linux-gnu/libpthread. so.0. (no debugging symbols found)... done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1 ".
Loaded symbols for/lib/x86_64-linux-gnu/libpthread. so.0
Reading symbols from/usr/lib/x86_64-linux-gnu/libmysqlclient. so.18... (no debugging symbols found)... done.
Loaded symbols for/usr/lib/x86_64-linux-gnu/libmysqlclient. so.18
Reading symbols from/usr/local/lib/libstd. so... (no debugging symbols found)... done.
Loaded symbols for/usr/local/lib/libstd. so
Reading symbols from/lib/x86_64-linux-gnu/libc. so.6... (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/libc. so.6
Reading symbols from/lib/x86_64-linux-gnu/librt. so.1. .. (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/librt. so.1
Reading symbols from/lib/x86_64-linux-gnu/libcrypto. so.1.0.0... (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/libcrypto. so.1.0.0
Reading symbols from/lib64/ld-linux-x86-64.so.2... (no debugging symbols found)... done.
Loaded symbols for/lib64/ld-linux-x86-64.so.2
Reading symbols from/lib/x86_64-linux-gnu/libz. so.1. .. (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/libz. so.1
Reading symbols from/lib/x86_64-linux-gnu/libdl. so.2. .. (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/libdl. so.2
Reading symbols from/lib/x86_64-linux-gnu/libm. so.6... (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/libm. so.6
Reading symbols from/usr/local/lib/libudt. so... (no debugging symbols found)... done.
Loaded symbols for/usr/local/lib/libudt. so
Reading symbols from/usr/lib/x86_64-linux-gnu/libstdc ++. so.6... (no debugging symbols found)... done.
Loaded symbols for/usr/lib/x86_64-linux-gnu/libstdc ++. so.6
Reading symbols from/lib/x86_64-linux-gnu/libgcc_s.so.1... (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from/lib/x86_64-linux-gnu/libnss_files.so.2... (no debugging symbols found)... done.
Loaded symbols for/lib/x86_64-linux-gnu/libnss_files.so.2
Execute_search () at test. c: 310

From the last line, we can see that thread 2005 is running the execute_search function. Then we can view the function and find that there is such

 

While (g_search_state = 1)
{
Tmp = g_data_list-> phead-> next;

If (tmp)
{
Pthread_mutex_lock (& g_mysql_mutex );
Res. result = search_mysql (tmp-> sn, & (res. machine ));
Pthread_mutex_unlock (& g_mysql_mutex );

Snprintf (res. sn, sizeof (res. sn), "% s", tmp-> sn );
}
}

Since tmp is NULL for a long time, it becomes an while infinite loop, and the infinite loop occupies the memory, so the usleep (50000) is added below, and the problem is solved.

This article permanently updates the link address: