Linux Performance monitoring Tools

Tag: Style color IO works with data sp on C

CPU Chapter:
CPU is also a kind of hardware resources, and any other hardware devices need to drive and management programs to use, we can take the kernel process scheduling as a
CPU management program, used to manage and allocate CPU resources, reasonable scheduling process to preempt the CPU, and decide which process to use the CPU, which process should wait

To monitor and understand CPU performance you need to know some basic knowledge of the operating system, such as: interrupts, process scheduling, process context switching, operational
Queues, and so on. Here Vpsee use an example to briefly introduce these concepts and their relationship, the CPU is very innocent, is a hard working wage earners, every time every
Have a job to do (process, thread) and have a work list (can run a queue), by the Boss (process scheduling) to decide what he should do, he
Need to communicate with the boss in order to get the boss's ideas and adjust their work in time (context switch), part of the work will need to report to the boss in time
(interrupts), so the worker (CPU), in addition to doing their own work, there is a lot of time and energy spent on communication and reporting.

Process scheduling assigns different priorities to different resources, with the highest priority being hardware interrupts, followed by kernel (System) processes, and finally user processes.
Each CPU maintains a running queue to hold those threads that can be run. The thread is either in a sleep state (blocked is waiting for IO) or
In the operational state, if the current load of the CPU is too high and the new request is constant, there will be a situation that the process is not able to handle temporarily, this time will not
The thread is not temporarily placed in the operational queue.

• How many jobs are accepted and completed by the wage earners and reported to the Boss (interrupted);
• The employee communicates with the boss, negotiates the work progress of each work (context switch);
• The work list of the wage earners are all full (can run the queue);
• Wage earners how efficient, is not lazy (CPU utilization)

Bottom line

Previous Linux Performance monitoring: The introduction of performance monitoring before the need to know the bottom line, then monitor the CPU performance of the bottom line is what? Usually we expect our system to reach the following goals:

• CPU utilization, if the CPU has 100% utilization, then a balance should be reached: 65%-70% User time,30%-35% System time,0%-5% Idle time;
• Context switching, context switching should be linked to CPU utilization, if you can maintain the above balance of CPU utilization, a lot of context switching is acceptable;
• Can run the queue, each can run a queue should not have more than 1-3 threads (per processor), such as: Dual-processor system can not run more than 6 threads in a running queue

Vmstat

Vmstat is a small tool that looks at the overall performance of the system, works well even in very heavy situations, and can capture continuous performance data at time intervals.

$ vmstat 1procs-----------memory-------------Swap-------io------System-------CPU------R  B   swpd   free< C4/>buff  cache   si   so    bi    bo   in   CS US sy ID WA St 2  1    140 2787980 336304 3531996< C14/>0    0     0 (   1166 5033  3  3) 0 0 1    2788296 336304 3531996  0    0     0     0 1194 5605  3  3 0  0  1    2788436 336304 3531996  0    0     0     0 1249 8036  5 4 (0 0 1)    2782688 336304 3531996  0    0     0     0 1333 7792  6  6  0 3  1    2779292 336304 3531992  0    0     0    28 1323 7087  4  5  0

Parameter description:

• R, the number of threads that can run the queue, all of which are operational, except that the CPU is temporarily unavailable;
• b, the number of processes being blocked, waiting for IO requests;
• In, number of interrupts processed
• CS, number of context switches being made on the system
• US, percent of CPU consumed by the user
• SYS, the percentage of cores and interrupts consuming CPU
• WA, all running threads are blocked waiting for IO, when the percentage of CPU idle
• Id,cpu percent of total idle

Give two realistic examples to actually analyze:

$ vmstat 1procs-----------memory-------------Swap-------io------System-------CPU------R  B   swpd   free< C3/>buff  cache   si   so    bi    bo   in   CS US sy ID WA St 4  0    140 2915476 341288 3951700< C13/>0    0     0     0 1057  523 bayi  0  0  0 4  0    2915724 341296 3951700  0< C24/>0     0     0 1048  546  0  0  0 4 0    2915848 341296 3951700  0    0< C35/>0     0 1044  514  0  0  0 4  0    2915848 341296 3951700  0    0     0< C46/>24 1044  564  0  0  0 4  0    2915848 341296 3951700  0    0     0     0 1060  546  0  0  0

From the above data you can see the points:

1. Interrupts (in) is very high, the context switch (CS) is relatively low, indicating that the CPU has been constantly requesting resources;
2. User Time (US) has remained above 80%, and the context switch is lower (CS), indicating that a process may have been hogging the CPU;
3. The run queue (R) is just 4.

$ vmstat 1procs-----------memory-------------Swap-------io------System-------CPU------R  B   swpd   free< C3/>buff  cache   si   so    bi    bo   in   CS US sy ID wa st14  0    140 2904316 341912 3952308<    0     0   460 1106 9593  1  0  017 0 c13/>0    2903492 341912 3951780  0    0     0     0 1037 9614  1  0  020  0    2902016 341912 3952000  0    0     0     0 1046 9739  1  0  017  0    2903904 341912 3951888  0    0     0    1044 9879 PNS  0  0  016  0    2904580 341912 3952108  0    0     0     0 1055 9808  1  0  0

From the above data you can see the points:

1. Context Switch (CS) is much higher than interrupts (in), which means the kernel has to switch processes back and forth;
2. Further observation shows that system time (SY) is very high and user time (US) is low, plus a high-frequency context switch (CS), indicating that a large number of system calls are being invoked by the running application;
3. Run Queue (R) is above 14 threads, according to the hardware configuration of this test machine (quad core), should be kept within 12.

Linux Performance monitoring Tools