Analysis and review of CPU usage for each thread in the process of Linux

We often set up a game server in the new service, sometimes to stress test, so how to see, generally we will use the top command to see the CPU and memory consumption of each process, to obtain our process ID, Then we might look at each thread ID in the Pstack command and what the corresponding thread is doing now, analyzing multiple sets of data to get a solution that slows down the performance of the server. such as the group data:

[Email protected] bin]# Pstack 30222
Thread 9 (thread 0x7f729adc1700 (LWP 30251)):
#0 0x00007f72a429b720 in sem_wait () from/lib64/libpthread.so.0
#1 0x0000000000ac5eb6 in Semaphore::d Own () ()
#2 0X0000000000AC5CAC in Queue::get () ()
#3 0x00000000009a583f in DBManager::p rocessupdate (queue*) ()
#4 0X00000000009A4BFB in Dbupdatethread (void*) ()
#5 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#6 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 8 (thread 0x7f727c579700 (LWP 30252)):
#0 0x00007f72a429b720 in sem_wait () from/lib64/libpthread.so.0
#1 0x0000000000ac5eb6 in Semaphore::d Own () ()
#2 0X0000000000AC5CAC in Queue::get () ()
#3 0x00000000009a5799 in DBManager::p rocessquery (queue*) ()
#4 0x00000000009a4c3a in Dbquerythread (void*) ()
#5 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#6 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 7 (thread 0x7f7257fff700 (LWP 30253)):
#0 0X00007F72A42997BB in [email protected] @GLIBC_2.3.2 () from/lib64/libpthread.so.0
#1 0x00007f72a549ee08 in utils::thread::condition_impl::timed_wait (int) () from/usr/local/utils-0.0.1/lib/ libutils.so.0.0.1
#2 0x00007f72a549ebd3 in utils::thread::condition::timed_wait (int) () from/usr/local/utils-0.0.1/lib/ libutils.so.0.0.1
#3 0x00000000009d5f57 in Utils::messagequeue<fightinfo*>::p op () ()
#4 0x00000000009d5557 in Fightreport::svc () ()
#5 0x00007f72a5494b45 in Utils::task_base::thread_proc (void*) () from/usr/local/utils-0.0.1/lib/libutils.so.0.0.1
#6 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#7 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 6 (thread 0x7f72397b7700 (LWP 30254)):
#0 0X00007F72A4588FC3 in poll () from/lib64/libc.so.6
#1 0x00007f72a0fbded4 in __libc_res_nsend () from/lib64/libresolv.so.2
#2 0x00007f72a0fba76a in __libc_res_nquery () from/lib64/libresolv.so.2
#3 0x00007f72a0fbad29 in __libc_res_nquerydomain () from/lib64/libresolv.so.2
#4 0x00007f72a0fbb9cf in __libc_res_nsearch () from/lib64/libresolv.so.2
#5 0x00007f729adc37a7 in _nss_dns_gethostbyname4_r () from/lib64/libnss_dns.so.2
#6 0x00007f72a457a4c3 in Gaih_inet () from/lib64/libc.so.6
#7 0x00007f72a457cb20 in getaddrinfo () from/lib64/libc.so.6
#8 0x00007f72a56fc782 in CURL_GETADDRINFO_EX () from/usr/lib64/libcurl.so.4
#9 0x00007f72a56f1d42 in Curl_getaddrinfo () from/usr/lib64/libcurl.so.4
#10 0x00007f72a56c9e77 in Curl_resolv () from/usr/lib64/libcurl.so.4
#11 0x00007f72a56ca138 in Curl_resolv_timeout () from/usr/lib64/libcurl.so.4
#12 0x00007f72a56d8d88 in?? () from/usr/lib64/libcurl.so.4
#13 0x00007f72a56ddb79 in?? () from/usr/lib64/libcurl.so.4
#14 0x00007f72a56de76e in Curl_connect () from/usr/lib64/libcurl.so.4
#15 0x00007f72a56e69b0 in Curl_perform () from/usr/lib64/libcurl.so.4
#16 0x0000000000ae6e3d in Httpclient::svc () ()
#17 0x00007f72a5494b45 in Utils::task_base::thread_proc (void*) () from/usr/local/utils-0.0.1/lib/libutils.so.0.0.1
#18 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#19 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 5 (thread 0x7f721af6f700 (LWP 30255)):
#0 0x00007f72a455691d in Nanosleep () from/lib64/libc.so.6
#1 0x000000000098cb8a in Sleep (unsigned long) ()
#2 0x000000000098b87d in Dynresource::svc () ()
#3 0x00007f72a5494b45 in Utils::task_base::thread_proc (void*) () from/usr/local/utils-0.0.1/lib/libutils.so.0.0.1
#4 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#5 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 4 (thread 0x7f71fc727700 (LWP 30256)):
#0 0x00007f72a455691d in Nanosleep () from/lib64/libc.so.6
#1 0x000000000098cb8a in Sleep (unsigned long) ()
#2 0x0000000000a61516 in Playeroplogthread::svc () ()
#3 0x00007f72a5494b45 in Utils::task_base::thread_proc (void*) () from/usr/local/utils-0.0.1/lib/libutils.so.0.0.1
#4 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#5 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 3 (thread 0x7f71ddedf700 (LWP 30257)):
#0 0x00007f72a4592c73 in epoll_wait () from/lib64/libc.so.6
#1 0x00007f72a51f334f in Epoll_reactor::run_reactor_event_loop () () from/usr/local/net_manager-0.0.2/lib/libnet_ manager.so.0.0.2
#2 0x00007f72a51f2523 in Net_thread::svc () () from/usr/local/net_manager-0.0.2/lib/libnet_manager.so.0.0.2
#3 0x00007f72a5494b45 in Utils::task_base::thread_proc (void*) () from/usr/local/utils-0.0.1/lib/libutils.so.0.0.1
#4 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#5 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 2 (thread 0x7f71bf697700 (LWP 30258)):
#0 0x00007f72a4592c73 in epoll_wait () from/lib64/libc.so.6
#1 0x00007f72a51f334f in Epoll_reactor::run_reactor_event_loop () () from/usr/local/net_manager-0.0.2/lib/libnet_ manager.so.0.0.2
#2 0x00007f72a51f2523 in Net_thread::svc () () from/usr/local/net_manager-0.0.2/lib/libnet_manager.so.0.0.2
#3 0x00007f72a5494b45 in Utils::task_base::thread_proc (void*) () from/usr/local/utils-0.0.1/lib/libutils.so.0.0.1
#4 0x00007f72a4295851 in Start_thread () from/lib64/libpthread.so.0
#5 0x00007f72a459267d in Clone () from/lib64/libc.so.6
Thread 1 (thread 0x7f72a60ae7e0 (LWP 30222)):
#0 0x00007f72a4584c95 in _xstat () from/lib64/libc.so.6
#1 0x00007f72a45483e0 in __tzfile_read () from/lib64/libc.so.6
#2 0x00007f72a4547864 in tzset_internal () from/lib64/libc.so.6
#3 0x00007f72a4547b20 in Tzset () from/lib64/libc.so.6
#4 0x00007f72a4546699 in timelocal () from/lib64/libc.so.6
#5 0x0000000000b0b08d in Achieve::getremaintime (achievetemplate*) ()
#6 0x0000000000b115ca in Achieve::update () ()
#7 0x0000000000a197ce in Player::update () ()
#8 0x0000000000b1b272 in Playermng::tick () ()
#9 0x0000000000a73105 in Gameserver::frametick (unsigned int) ()
#10 0x0000000000a6ff80 in Gameserver::run () ()
#11 0x0000000000a773a1 in Main ()

[Email protected] gameserver]# Ps-elo pid,lwp,pcpu | grep 30222
30222 30222 31.4
30222 30251 0.0
30222 30252 0.0
30222 30253 0.0
30222 30254 0.0
30222 30255 0.0
30222 30256 1.2
30222 30257 1.2
30222 30258 1.0

Multiple sets of data show that our main logical thread really occupies a high CPU, found that it really achieve::update () did too much useless judgment, and can reduce the number of loop entry, so I am here to optimize

Then I used the IPTRAF command to see the network traffic:

First item: IP traffic Monitoring
Second item: General view network card traffic status. View only the total traffic for each NIC
The third item: Check the network card traffic status in detail. such as by Tcp,udp,arp and other protocols to view

Select all interfaces to view all NIC interfaces

The interface is divided into two parts, the upper part can show in detail which IP is connected with, how many packets are sent, what is the real-time traffic, the next part, can display UDP and other information.

Shows all connection traffic to the local server

In addition one has to mention another command SAR:

SAR has many uses, if you want to monitor network traffic, use the following command-line method:

Sar-n DEV interval Count

Where interval is the statistical interval, in seconds, Count is counted a total of several times, and if the constant statistic is 0 until Ctrl + C is interrupted, the count is executed to exit.

For example: Sar-n DEV 1 4

For example: Sar-n DEV 10 0

Iface:lan interface
RXPCK/S: Packets Received per second
TXPCK/S: Packets Sent per second
rxbyt/s: Number of bytes received per second
txbyt/s: Number of bytes sent per second

Sar-n Dev-u 1 10

07:22:01 PM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
07:22:02 PM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:22:02 PM eth0 2.47 0.00 0.11 0.00 0.00 0.00 0.00
07:22:02 PM eth1 80.25 22.22 3.53 3.52 0.00 0.00 0.00


07:22:02 PM CPU%user%nice%system%iowait%steal%idle
07:22:03 PM All 1.85 0.00 1.06 0.00 0.00 97.09


07:22:02 PM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
07:22:03 PM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:22:03 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:22:03 PM eth1 66.67 19.75 2.95 3.22 0.00 0.00 0.00


07:22:03 PM CPU%user%nice%system%iowait%steal%idle
07:22:04 PM All 2.12 0.00 1.06 0.00 0.00 96.82


07:22:03 PM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
07:22:04 PM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:22:04 PM eth0 7.32 4.88 0.65 0.38 0.00 0.00 0.00
07:22:04 PM eth1 85.37 28.05 4.18 3.88 0.00 0.00 0.00


07:22:04 PM CPU%user%nice%system%iowait%steal%idle
07:22:05 PM All 3.96 0.00 2.64 0.00 0.00 93.40


07:22:04 PM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
07:22:05 PM lo 412.50 412.50 104.44 104.44 0.00 0.00 0.00
07:22:05 PM eth0 13.75 12.50 1.32 1.26 0.00 0.00 0.00
07:22:05 PM eth1 903.75 707.50 174.12 62.42 0.00 0.00 0.00


07:22:05 PM CPU%user%nice%system%iowait%steal%idle
07:22:06 PM All 5.25 0.00 4.46 2.10 0.00 88.19


07:22:05 PM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
07:22:06 PM lo 849.41 849.41 154.54 154.54 0.00 0.00 0.00
07:22:06 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:22:06 PM eth1 1287.06 1337.65 84.48 461.76 0.00 0.00 0.00


07:22:06 PM CPU%user%nice%system%iowait%steal%idle
07:22:07 PM All 1.85 0.00 1.06 0.00 0.00 97.09


07:22:06 PM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
07:22:07 PM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:22:07 PM eth0 4.94 0.00 0.22 0.00 0.00 0.00 0.00
07:22:07 PM eth1 97.53 19.75 4.32 3.18 0.00 0.00 0.00


07:22:07 PM CPU%user%nice%system%iowait%steal%idle
07:22:08 PM All 1.58 0.00 1.32 0.00 0.00 97.11


07:22:07 PM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
07:22:08 PM lo 4.94 4.94 0.31 0.31 0.00 0.00 0.00
07:22:08 PM eth0 1.23 0.00 0.06 0.00 0.00 0.00 0.00
07:22:08 PM eth1 67.90 24.69 3.02 3.94 0.00 0.00 0.00


Average:cpu%user%nice%system%iowait%steal%idle
Average:all 3.63 0.00 2.55 0.84 0.00 92.98


Average:iface rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s
Average:lo 212.61 212.61 50.54 50.54 0.00 0.00 0.00
Average:eth0 42.79 41.70 3.27 41.84 0.00 0.00 0.00
Average:eth1 492.48 432.48 53.63 110.40 0.00 0.00 0.00

Analysis and review of CPU usage for each thread in the process of Linux