On-line service cpu100% problem Quick Location Combat

This is a creation in Article, where the information may have evolved or changed.

Functional issues , through the log, one-step debugging is relatively good positioning.

Performance problem , such as online server cpu100%, how to find the relevant services, how to locate the problem code, more Test technology People's foundation.

The Home Architecture Department, Operations and Maintenance department,the Technical Department of the rapid operation of the first online service CPU Troubleshooting the actual Practice , students feedback has been harvested, Special will be practical training questions and answers published, hope to everyone also helpful.

Topic

A number of Tomcat instances have been deployed on a server , namely several vertically segmented Java site Services, and several Java microservices, which suddenly receive operational CPU exception alarms.

Q: How to locate which service process is causing the CPU to overload, which thread is causing the CPU to overload, which piece of code causes CPU overload?

Step One, find the most CPU-consuming The process

Tools :Top

Method :

• Execution top-c to display a list of process run information

• type p ( uppercase p), the process is sorted by CPU usage

Diagram :

For example, the most consumption CPU 's Process PID is 10765

Step Two: Find the most CPU consuming the thread

Tools :Top

Method :

• top-hp 10765 that displays a list of thread run information for a process

• type p ( uppercase p), threads are sorted by CPU usage

Diagram :

For example, within the process 10765 , the CPU -consuming thread PID is 10804

Step three: Thread PID conversion to the system

Tools :printf

Method :printf "%x\n" 10804

Diagram :

For example,10804 corresponding to the system is 0x2a34, of course, this step can use a calculator .

The reason to convert to a system is that the thread ID is represented in the stack with a binary notation.

Step four: Look at the stack and find out what the thread is doing

Tools :pstack/jstack/grep

Method :jstack 10765 | grep ' 0x2a34 '-c5--color

• Print process Stack

• filter to get thread stack by thread ID

Diagram :

For example, to find the consumption The thread name "AsyncLogger-1" corresponds to the thread of the high CPU, and the stack where the thread is executing code is seen.

Hope that the regular online CPU troubleshooting students can help, if there is better practice, also welcome to share.

If you want to impress, please be sure to practice on the line .

If there is a harvest, help turn .

Related articles:

Common online Operations Linux command combat

Linux Trace Common command on line issues

One minute awk is enough

One minute sed is enough