Linux performance tools that programmers should know

Objective

In the actual development, sometimes receive some service monitoring alarm, such as CPU high, memory high, this time, we will log on to the server for troubleshooting. This blog will cover this area of knowledge: Linux performance tools.

One-time online troubleshooting simulation

Background: After a steady run of service, the CPU suddenly soared high.

With the top command, you can confirm which process is causing the CPU to soar (perhaps false positives). ）。

You can see that the PID in the figure is a 2816 process and the CPU usage is very high.

Use TOP-HP 2816来 to observe the threads under the process. The figure shows that 2825 of this thread CPU is very high.

Here python is very convenient to convert the decimal thread ID into 16 binary, why do you want to do so?

This is because the 16-binary nid is used in the next thread dump file.

In practice, we should take advantage of jstack pid multiple dump several times, because the thread has a state transition, so multiple dump is good for crawling to the thread more information.

In the diagram, you can observe that a thread has been locked, running, delayed, and another thread has been waiting for this lock. At this point, you can go to see the code to analyze why the lock is delayed release reasons.

Performance monitoring Tool Top

In the above case, the use of top, and in practice, top of the amount of information is very large, detailed analysis here.

First line:

It involves 2 times, one is the system time, and the other is the time of the machine running. "We should focus on the time the machine is running, why?" Sometimes, restarting the machine can cause a lot of problems, you know! 】

How many users are logged into the system? "More information can be found through Who/w/history"

What is the meaning of the 3 load values?

Represents the load situation of 1min,5min,15min machine, how to determine the load size? In combination with the CPU's number of cores, for example, the machine is a 4-core CPU, then if the load value exceeds 4, it means that the load is heavy! "Press 1 under top to see the number of CPUs"

The above information can actually be obtained by uptime command.

Second line:

The main point is how many tasks are in total, and the focus should be on the number of zombie status tasks.

Third line:

is mainly CPU information.

Us/sy, that is, the user process and the system process using the CPU's proportion.

NI, Nice, indicates that the proportion of processes that have been adjusted for thread priority should not be large.

The ID, which indicates idle, WA indicates the time of the resource wait, such as when the service has a lot of logs under the instantaneous high traffic, then this value will be high because it consumes resources.

Hi, hard interrupt, is usually caused by peripherals, if Hi high, then it means that the peripheral hardware level problems. Si indicates a soft interrupt.

St, that is, steel, if the host is virtual, there will be this St information, that is, the virtual machine from the host to obtain the CPU time slice of the percentage.

Lines IV and fifth:

Here are basically 2 conceptual things: buffer and cache.

What is the main buffer? Should be the data to be processed, mainly to deal with the problem of speed mismatch between 2 systems. The cache, in general, should be the caching of the resulting data, such as loading some information from the DB for querying.

Swap partition, is to use the hard disk to do a portion of the cache, if swap is very frequent, that is, memory is not enough!

List Description:

PID process ID, user users, PR priority, VIRT virtual memory, RES resident memory, SHR shared memory

It should be noted here that res represents the memory that the process actually occupies, not the size of the requested memory. That is, the physical size of the memory occupied by the current process is res-shr.

Linux performance tools that programmers should know