Meaning of cpu usage in linux top Command

Meaning of cpu usage in linux top Command
In fact, there are many articles on this part on the Internet. Why is this article still available? There are several reasons why it is the motivation of my translation. First, although concepts and content are old-fashioned, but they are all very thorough and comprehensive. second, combining theory with practice, the case analysis is good. third, it is not fancy. The tools and commands used are the most basic and helpful for actual operations. however, I am not very familiar with the original text. Most of the translations are based on my understanding of the original text. You can also go to OSCAN to find the original text. If there is any major difference, I would like to send a message to you. I am very grateful! 1.0 performance monitoring introduction performance optimization is to find bottlenecks in system processing and remove these processes. Most administrators believe that performance optimization can be achieved by reading related "cook book, some kernel configurations can solve the problem simply, but are not suitable for each environment. performance optimization is actually a balanced definition of the various OS subsystems. These subsystems include: CPU Memory IO Network these subsystems are mutually dependent, and any high load will cause problems in other subsystems. for example: A large number of page call requests cause congestion in the memory queue. The large throughput of the NIC may lead to more CPU overhead, a large amount of CPU overhead, and more memory is used to request a large number of disk writes from the memory. the request may cause more CPU and IO problems. Therefore, we need to optimize a system, finding the bottleneck is the key. Although it seems that a sub-system has a problem, it may be caused by another sub-system. 1.1 determine the application type based on the need to understand where to start to optimize the bottleneck. The first important point is to understand and Analyzes the features of the current system. Most systems run in two types of applications: IO Bound: Application in this category, generally, it is a high-load memory usage and storage system. This actually represents an IO application, which is a process of massive data processing. i/O applications do not initiate more requests to the CPU or network (unless network storage hardware such as NAS ). i/O applications usually use CPU resources to generate IO requests and enter the sleep status of kernel scheduling. usually database software (: mysql, oracle, etc.) is considered to be the application type in the IO category. CPU Bound: Applications in this category are generally high-load CPU usage. the Application of CPU is a process of batch processing of CPU requests and mathematical computing. generally, web server, mail server, and other types of services are considered as CPU applications. 1.2 determine the baseline to collect statistics on the system utilization, generally with the Administrator's experience and The purpose of the system is determined. The only thing to be clear is what the system optimization hopes to achieve, and what aspects need optimization, and what are the reference values? Therefore, a baseline is established. The statistical data must be the available system performance status value to compare the unavailable Performance Status values. in the following example, a baseline snapshot of system performance is used to compare the system performance snapshots at high loads. # vmstat 1 procs memory swap io system cpu r B swpd free buff cache si so bi bo in cs us sy wa id 1 0 138592 17932 126272 214244 0 0 1 18 109 19 2 1 1 96 0 0 138592 17932 126272 0 0 0 0 214244 0 0 0 1 0 0 0 0 105 138592 17932 0 0 0 126272 62 40 14 0 45 0 0 214244 198 138592 0 0 0 0 117 49 0 0 0 100 0 138592 17924 126272 0 0 0 214244 176 220 3 4 13 80 0 938 138592 17924 126272 0 0 0 0 214244 358 8 17 0 75 1 0 138592 17924 126272 214244 0 0 0 0 368 4 24 0 72 0 0 1447 138592 17924 126272 0 0 0 214244 9 12 0 79 # vmstat 1 procs memory swap io system cpu r B swpd free buff cache si so bi bo in cs us sy wa id 2 0 145940 17752 118600 215592 0 1 18 109 19 2 1 1 96 2 0 145940 15856 118604 215652 0 0 0 468 789 108 86 14 0 0 3 0 146208 13884 118600 214640 0 360 360 71 91 9 0 0 2 0 498 146388 13764 118600 0 213788 340 340 41 41 87 13 0 0 2 0 147092 13788 118600 0 212452 0 740 1324 61 620 92 8 0 0 2 0 147360 13848 118600 0 211580 720 720 41 41 96 4 0 0 2 0 690 147912 13744 118192 0 720 0 720 605 44 95 5 0 0 2 0 148452 13900 118192 209260 0 372 372 639 45 81 19 0 0 2 149132 13692 117824 208412 372 372 457 47 90 10 0 0, as shown in the first result above, the last column (id) indicates the idle time. We can see that the CPU idle time is between 79% and 100% in baseline statistics. the second result shows that the system is in the 100% usage and there is no idle time. from this comparison, we can determine whether the CPU usage should be optimized. 2.0 install monitoring tools most * nix systems have a bunch of standard monitoring commands. these commands are part of * nix from the very beginning. in Linux, other monitoring tools are provided through basic installation packages and additional packages. Most of these installation packages are available in various Linux versions. although there are other open-source and third-party monitoring software, this document only discusses monitoring tools based on the Linux version released. this chapter describes the tools used to monitor system performance. tool Description Base Repository vmstat all purpose performance tool yes mpstat provides statistics per CPU no yes sar all purpose performance monitoring tool no yes iostat provides disk statistics no yes netstat provides network statistics yes dstat monitoring statistics aggregator no in most distributions iptraffic routing dashboard no yes netperf Network bandwidth tool no In some distributions ethtool reports on Ethernet interface configuration yes iperf Network bandwidth tool no yes tcptrace Packet analysis tool no yes 3.0 CPU introduction CPU usage mainly depends on what resources are trying to access. the kernel scheduler will be responsible for scheduling two types of resources: threads (single or multi-channel) and interruptions. the scheduler defines different priorities of different resources. the following lists are arranged in a descending order of priority: Interrupts-the device notifies the kernel that they process data. for example, when a network adapter device delivers a network packet or a hardware device provides an IO request. kernel (System) Processes (annotation: Kernel processing process)-All Kernel processing Processes are control priorities. user Processes (: User process)-This section involves "userland ". all software programs run in this user space. this is at a low priority in the kernel scheduling mechanism. from the above, we can see how the kernel manages different resources. there are also several key topics to introduce. The following sections will introduce context, run queues and utilization ). 3.1 Context switching most modern processors can run a process (single thread) or thread. multi-channel hyper-threading processors can run multiple threads. however, the Linux kernel uses the dual-core chip of each processor as an independent processor. for example, a Linux kernel system is displayed as two independent processors on a dual-core processor. A standard Linux kernel can run 50 to 50,000 processing threads. when there is only one CPU, the kernel will schedule and balance each process thread. each thread is assigned a time limit for overhead in the processor. A thread can either obtain the time limit or obtain some threads with a higher priority (such as hardware interruption). Threads with a higher priority will be replaced from the region to the processor queue. the Conversion Relationship of this thread is the context switch we mentioned. during each kernel context switch, resources are used to shut down threads in the CPU register and put in the queue. there are more context switches in the system, and the kernel will get more work under the scheduling management of the processor. 3.2 each CPU in the running queue maintains a thread's running queue. in theory, the scheduler should continuously run and execute threads. the process thread is either in sleep or running state. if the CPU subsystem is under high load, it means that the kernel scheduling will not be able to respond to system requests in a timely manner. as a result, processes in the running status are congested in the running queue. when the running queue is getting bigger and bigger, the process thread will spend more time to get it to be executed. the popular term is "load", which provides the detailed status of the current running queue. system load refers to the combination of a few threads in the CPU queue and the number of processes in the queue. if a dual-core system executes two threads and four other threads are in the running queue, load should be 6. the load averages shown in the top program refer to the load conditions within 1, 5, 15 minutes. 3.3 CPU usage is the percentage of CPU usage. the most important measurement method for evaluating the system is the CPU utilization. most performance monitoring tools have the following types of CPU utilization: User Time (user process Time)-Percentage of CPU overhead Time of processes executed in User space. system Time. wait IO (: IO request Wait time)-Percentage of CPU overhead idle time occupied by all process threads being blocked and waiting for an IO request to be completed. idle (Idle)-percentage of time a completely Idle process can be sold in a CPU processor. 4.0 CPU performance monitoring understand the relationship between running queues, utilization, and context switching on how to optimize CPU performance. as mentioned earlier, performance is relative to baseline data. in some systems, the expected performance includes: Run Queues-each processor should Run a queue of no more than 1-3 threads. for example, a dual-core processor should run a queue of no more than 6 threads. CPU Utiliation-if a CPU is fully used, the balanced ratio between utilization classification should be 65%-70% User Time 30%-35% System Time 0%-5% Idle Time Context Switches-the number of Context Switches is directly related to the CPU usage, if the CPU usage remains balanced, a large number of context switches are normal. many Linux tools can obtain these status values, namely, vmstat and top. 4.1 The vmstat tool provides a low-cost system performance observation method. because vmstat itself is a low-cost tool, on a very high-load server, you need to view and monitor the health of the system. In the control window, you can still use vmstat to output results. this tool runs in two modes: average and sample. the sample mode measures the status value by specifying the interval. this mode is helpful for understanding the performance under continuous load. the following is an example of one-second interval between running vmstat: # vmstat 1 procs ----------- memory ---------- --- swap -- ----- io ---- system -- ---- cpu ---- r B swpd free buff cache si so bi bo in cs us sy id wa 0 0 104300 16800 72200 0 0 5 26 7 14 4 1 95 0 0 0 104300 16800 95328 0 0 0 0 24 72200 64 1 1 98 0 0 0 1021 104300 16800 0 0 0 95328 59 1 1 98 0 Table 1: the vmstat CPU statistics Field Description r The amount of threads in the run queue. these are threads that are runnable, but the CPU is not available to execute them. number of threads in the current running queue. indicates that the thread is running, but the CPU has not been executed. B This is the number of processes blocked and waiting on IO requests to finish. the number of times that the current process is blocked and waits for the completion of the IO request in This is the number of interrupts being processed. number of currently interrupted processes cs This is the number of context switches currently happening on the system. the number of context switches in the current kernel system. us This is the percentage of user CPU utilization. CPU usage percentage sys This is the percentage of kernel and interrupts utilization. percentage of kernel and interrupt utilization wa This is the percentage of idle processor time due to the fact that ALL runnable threads are blocked waiting on IO. the percentage id of all runtime threads blocked waiting for IO requests. This is the percentage of time that the CPU is completely idle. percentage of CPU idle time 4.2 Case Study: continuous CPU utilization in this example, this system is fully utilized # vmstat 1 procs memory swap io system cpu r B swpd free buff cache si so bi bo in cs us sy wa id 3 0 206564 15092 80336 176080 0 0 0 0 0 0 0 718 26 81 19 0 0 2 0 206564 14772 80336 176120 0 0 0 0 758 23 96 4 0 0 1 0 206564 14208 80336 0 0 0 0 176136 820 4 0 0 0 0 0 0 0 0 0 0 0 206956 13884 79180 175964 0 412 2680 1008 207348 80 93 7 0 0 2 0 14448 78800 175576 412 0 412 763 207348 70 84 16 0 0 2 0 15756 78800 175424 0 0 0 0 0 0 0 0 0 0 874 25 89 11 0 0 1 0 207348 16368 78800 0 0 0 0 175596 86 14 0 0 0 1 0 940 207348 16600 0 0 0 0 78800 27 95 3 0 3 0 207348 16976 78548 175876 0 0 2508 969 35 93 7 0 0 0 207348 16216 78548 0 0 0 0 175704 36 93 6 0 1 4 0 874 207348 16424 0 0 0 0 0 0 0 0 0 850 26 77 23 0 0 2 0 207348 17496 78556 175840 0 0 0 0 736 23 83 17 0 0 0 0 0 207348 17680 78556 0 0 0 0 175868 21 91 8 0 1 according to Observation value, we can draw the following conclusions: 1. There are a large number of interruptions (in) and a small number of context switches (cs ). this means that a single process is generating requests to hardware devices. 2. It is further shown that the user time (us) of a single application is usually 85% or more. considering a small number of context switches, this application should still be processed in the processor. 3. The running queue is still within the acceptable performance range. Two of them are beyond the permitted limits. 4.3 Case Study: overload Scheduling in this example, context switching in kernel scheduling is saturated # vmstat 1 procs memory swap io system cpu r B swpd free buff cache si so bi bo in cs us sy wa id 2 1 207740 98476 81344 180972 0 0 0 2496 0 900 2883 4 12 57 27 0 1 207740 96448 83304 0 180984 1968 328 810 8 9 83 0 0 2559 207740 94404 85348 0 180984 2044 829 9 6 78 7 0 1 207740 92576 87176 180984 0 0 1828 689 3 9 78 10 2 0 2088 207740 91300 0 0 88452 0 180984 1276 7 6 83 4 3 1 565 2182 207740 0 0 0 1176 0 551 2219 2 7 91 0 4 2 207740 89240 90512 180984 0 880 520 443 22 10 67 0 5 3 907 207740 88056 91680 0 180984 1168 628 12 11 77 0 4 2 207740 86852 92880 180984 0 0 1200 654 6 7 87 0 6 1 1505 207740 85736 93996 0 0 180984 1116 5 10 85 0 1 526 1512 207740 0 0 0 892 0 438 1556 6 4 90 0 according to the observed value, we can draw the following conclusions: 1. The number of context switches is higher than the number of interrupts, indicating that a considerable amount of time in the kernel overwrites the context switching thread. 2. A large number of context switches will result in unbalanced CPU utilization classification. obviously, the percentage of requests waiting for io (wa) is very high, and the percentage of user time is very low (us ). 3. Because the CPU is blocked on IO requests, a considerable number of runnable threads in the running queue are waiting for execution. 4.4 use of the mpstat tool if your system runs on a multi-processor chip, you can use the mpstat command to monitor each independent chip. the Linux kernel depends on the dual-core processor as 2 CPUs. Therefore, the dual-core processor reports that 4 CPUs are available. the CPU utilization statistical value provided by the mpstat command is roughly the same as that provided by vmstat, but mpstat can provide a statistical value based on a single processor. # mpstat-p all 1 Linux 2.4.21-20. ELsmp (localhost. localdomain) 05/23/2006 05:17:31 pm cpu % user % nice % system % idle intr/s 05:17:32 PM all 0.00 0.00 3.19 96.53 13.27 05:17:32 PM 0 0.00 0.00 0.00 100.00 05:17:32 PM 1 0.00 1.12 0.00 05:17:32 PM 2 0.00 0.00 0.00 100.00 05:17:32 PM 3 0.00 0.00 0.00 0.00 100.00 case study: in this example, the unused processing capacity is available for 4 CPU cores. two of the CPU processes run (CPU 0 and 1 ). the 3rd cores process all kernels and other system functions (CPU 3 ). 4th cores are in idle (CPU 2 ). using the top Command, we can see that three processes occupy almost the entire CPU core. # top-d 1 top-23:08:53 up, 3 users, load average: 0.91, 0.37, 0.13 Tasks: 190 total, 4 running, 186 sleeping, 0 stopped, 0 zombie Cpu (s): 75.2% us, 0.2% sy, 0.0% ni, 24.5% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 2074736 k total, 448684 k used, 1626052 k free, 73756 k buffers Swap: 4192956 k total, 0 k used, 4192956 k free, 259044 k cached pid user pr ni virt res shr s % CPU % mem time + COMMAND 15957 nobody 25 0 2776 280 224 R 100 20.5 0: 25. 48 php 15959 mysql 25 0 2256 280 R 224 100. 78 mysqld 15960 apache 25 0 2416 280 R 224 100 0: 11. 20 httpd 15901 root 16 0 2780 1092 R 1 800. 59 top 1 root 16 0 1780 660 572 S 0 0.0. 64 init # mpstat-p all 1 Linux 2.4.21-20. ELsmp (localhost. localdomain) 05/23/2006 05:17:31 pm cpu % user % nice % system % idle intr/s 05:17:32 PM all 81.52 0.00 18.48 21.17 130.58 05:17:32 PM 0 83.67 0.00 17.35 0.00 05:17:32 PM 1 115.31 80.61 0.00 05:17:32 PM 2 0.00 0.00 16.33 84.66 2.01 05:17:32 PM 3 79.59 0.00 21.43 0.00 05:17:32 pm cpu % user % nice % system % idle intr/s 05:17:33 PM all 0.00 85.86 0.00 14.14 25.00 05:17:33 PM 0 88.66 0.00 12.37 0.00 116.49 05:17:33 PM 1 80.41 0.00 19.59 0.00 05:17:33 PM 2 0.00 0.00 0.00 0.00 100.00 05:17:33 PM 3 0.00 83.51 0.00 16.49 0.00 05:17:33 pm cpu % user % nice % system % idle intr/s 05:17:34 PM all 82.74 0.00 17.26 25.00 05:17:34 PM 0 115.31 85.71 0.00 13.27 05:17:34 PM 1 0.00 115.31 78.57 0.00 21.43 05:17:34 PM 2 0.00 0.00 0.00 0.00 05:17:34 PM 3 0.00 100.00 0.00 9.18 0.00 0.00 05:17:34 pm cpu % user % nice % system % idle intr/s 05:17:35 PM all 87.50 0.00 12.50 25.00 115.31 05:17:35 PM 0 91.84 0.00 8.16 05:17:35 PM 1 0.00 114.29 90.82 0.00 1.02 05:17:35 PM 2 0.00 0.00 0.00 100.00 0.00 05:17:35 PM 3 81.63 0.00 15.31 0.00 0.00 you can also use the ps command to view this column by using this column, check which process occupies the CPU. # while:; do ps-eo pid, ni, pri, pcpu, psr, comm | grep 'mysqld '; sleep 1; done pid ni pri % cpu psr command 15775 0 15 86.0 3 mysqld pid ni pri % cpu psr command 15775 0 14 94.0 3 mysqld pid ni pri % cpu psr command 15775 0 14 96.6 3 mysqld pid ni pri % cpu psr command 15775 0 14 98.0 3 mysqld pid ni pri % cpu psr command 15775 0 14 98.8 3 mysqld pid ni pri % cpu psr command 15775 0 14 99.3 3 mysqld 4.6 conclusion CPU performance monitoring consists of the following parts: 1. Check the running queue of the system and determine not to exceed the limit of three runable threads for each processor. 2. determine that the user/system ratio in CPU utilization is maintained at 70/30 3. When the CPU overhead is more time in system mode, it indicates that the CPU overhead has been overloaded and the priority should be re-scheduled 4, when I/O processing increases, application processing in the CPU category will be affected.