Locating and analysis of high CPU usage of one server

Background: Performance monitoring shows that the CPU usage of a single core of the online server has reached 100%, which is caused by one of our core services. Thanks to the fact that our service processes are undertaken by multiple identical worker (thread) scheduling tasks, in addition to the high CPU usage, the service is not affected. The last time we found the I/O-eating criminal, this time we were looking for special agents lurking in the group, which was even more thrilling!

(Note: As the problem has been solved, we can only explain the detection process below)
System Environment

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/47/10/wKiom1P2jSPi5DkFAABnC85tHrE367.jpg "Title =" wkiol1pzhbih3dckaabqlzu8nxs262.jpg "alt =" wkiom1p2jspi5dkfaabnc85thre367.jpg "/>

UseTopThe command can easily locate who occupies the highest CPU.

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/47/12/wKioL1P2jkyjZ9IxAAD-573FdUw718.jpg "Title =" top.jpg "alt =" wKioL1P2jkyjZ9IxAAD-573FdUw718.jpg "/>

Taking one of our business processes (imserver) as an example, why is this product a leader? Because this is a multi-threaded process, we need to know that the smallest unit of CPU usage is actually a thread, so it must be caused by one or more of the many threads occupying too much CPU.Top-H-P PIDCommand to view the CPU usage of each thread in the process

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/47/10/wKiom1P2jUjhAyrCAADwfi2JRbA140.jpg "Title =" topt.jpg "alt =" wkiom1p2jujhayrcaadwfi2jrba140.jpg "/>

As shown in, we can see that the thread CPU usage with ID 5887 is the highest, assuming that it causes the CPU usage of 100%. Well, now we only need to find the CPU he stole. Although the boy's mouth is strict, we have a complete interrogation process, not afraid of him. FirstStrace-t-r-c-P PIDCommand

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/47/12/wKioL1P2jnShkhXWAADNFEXns_k080.jpg "Title =" strace.png "alt =" wkiol1p2jnshkhxwaadnfexns_k080.jpg "/>

It is used to view system calls and elapsed time, but not necessarily the CPU occupied by system calls. It may also be user-mode.

Let's proceedPstack PIDLaunch

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/47/10/wKiom1P2jXbC1YV4AAd46Mg4Im8527.jpg "Title =" pstack.jpg "alt =" wkiom1p2jxbc1yv4aad46mg4im8527.jpg "/>

We can see the call stack of each thread, find the thread that has found the highest CPU usage, and then look at its call stack. It is easy to see which logic leads to the busy loop, then locate the code, fix the bug, and retrieve the stolen CPU.

Postscript: Actually, as a programmer, I feel that the greatest pleasure is not to write a program, but to find some "high-end" bugs. Maybe it is the same as some policemen are obsessed with detecting cases, this is the love of technology.

This article from "forever friends" blog, please be sure to keep this source http://yaocoder.blog.51cto.com/2668309/1543352