Linux CPU Usage 100% troubleshooting

A number of Tomcat instances have been deployed on a server, namely several vertically segmented Java site services, and several Java microservices, which suddenly receive operational CPU exception alarms.

Q: How to locate which service process is causing CPU overload, which thread is causing CPU overload, which code causes CPU overload?

Step one, find the most CPU-consuming process

Tools: Top

Method:

• Execute top-c, display the process run information list

• Type P (uppercase p) and the process is sorted by CPU usage

Icon:

For example, the CPU-consuming process PID is 10765

Step two: Find the thread that consumes the most CPU

Tools: Top

Method:

• TOP-HP 10765, displaying a list of thread run information for a process

• Type P (uppercase p), and threads are sorted by CPU usage

Icon:

For example, within process 10765, the CPU-consuming thread pid is 10804

Step three: Convert the thread PID to 16 binary

Tools: printf

Method: printf "%x" 10804

Icon:

For example, 10804 corresponds to the 16 binary is 0x2a34, of course, this step can be used calculator.

The reason to convert to 16 is because the thread ID in the stack is represented by the 16 binary.

Step four: Look at the stack and find out what the thread is doing

Tool: Pstack/jstack/grep

Method: Jstack 10765 | grep ' 0x2a34 '-c5--color

• Print process Stack

• Filter to get thread stack by thread ID

Icon:

For example, find the thread name "AsyncLogger-1" for threads that consume high CPU, and see the stack where the thread is executing code.

Linux CPU Usage 100% troubleshooting