How to diagnose the performance of Linux in 60,000 milliseconds

Reproduced in: http://www.itxuexiwang.com/a/liunxjishu/2016/0225/168.html?145648414060,000 Performance diagnostics for Linux in milliseconds

When you log on to a Linux server in order to solve a performance problem: What should you check in the first minute?

At Netflix, we have a huge EC2 Linux cloud, as well as a number of performance analysis tools to monitor and diagnose its performance. These include Atlas for cloud monitoring, and vectors for on-demand instance analysis. While these tools can help us solve most problems, we still sometimes need to log on to a server instance and run some standard Linux performance tools.

In this article, the Netflix performance Engineering team will explain to you what you should do in the first 60 seconds of the best performance analysis on the command line, using the standard Linux tools you should be able to get.

First 60 seconds: overview

By running the following 10 commands, you can get a rough idea of the processes and resource usage that the system is running within 60 seconds. By looking at the error messages and the resource saturation of these commands (they are all easy to understand), you can then optimize the resources. Saturation refers to the load of a resource that exceeds the limit it can handle. Once saturated, it is usually exposed to the length or wait time of the request queue.

UPTIMEDMESG | Tailvmstat 1mpstat-p all 1pidstat 1iostat-xz 1free-msar-n DEV 1sar-n tcp,etcp 1top

Some of these commands require pre-installation of the Sysstat package. The information presented by these commands can help you implement the use method, a method for locating performance bottlenecks, such as checking usage, saturation, and error information for various resources such as CPU, memory, disk, and so on. In addition to locating the problem, you can use these commands to troubleshoot some of the possible problems that may be causing the problem, to help you narrow the scope of the check, and to indicate the direction for the next check.

The following sections briefly describe these commands as an example of executing these commands in a production environment. For more information on how these tools are used, refer to their man documentation.

1. Uptime

$ uptime 23:51:26 up 21:31, 1 user, load average:30.02, 26.43, 19.02#p# page Header #e#

This is a quick way to see the average load on the system, indicating how many tasks (processes) are running in the system. In a Linux system, these numbers contain processes that need to run in the CPU and that are waiting for I/O (usually disk I/O). It's just a rough display of the system load, a little bit of a look. You'll also need additional tools to learn more about the situation.

These three numbers show the results of the total load average of the system in one minute, five minutes, and 15 minutes compressed by the exponential ratio. From this we can see how the load of the system changes over time. For example, if you are checking a question and then see that the value of 1 minutes corresponds to a value that is much less than 15 minutes, then it may be that the problem has passed and you have not been able to observe it in time.

In this example, the system load increases over time because the load value of the last minute exceeds 30, while the average load of 15 minutes is only 19. Such a significant gap includes many meanings, such as CPU load. To further confirm, run the Vmstat or Mpstat command, which refer to the 3rd and 4th chapters later.

2. DMESG | Tail

$ DMESG | TAIL[1880957.563150] Perl invoked Oom-killer:gfp_mask=0x280da, order=0, oom_score_adj=0[...] [1880957.563400] out of Memory:kill process 18694 (perl) score 246 or sacrifice child[1880957.563408] killed process 1869 4 (Perl) total-vm:1972392kb, ANON-RSS:1953348KB, file-rss:0kb[2320864.954447] tcp:possible SYN flooding on port 7001.  Dropping request. Check SNMP counters.

This command explicitly shows the last 10 system messages if they exist. Look for errors that can cause performance problems. The above example contains the Oom-killer, and TCP discards a request.

Don't miss this step! The DMESG command is always worth a try.

3. Vmstat 1

$ vmstat 1procs ---------Memory---------- ---swap-- -----io---- -system-- - -----CPU----- r  b swpd   free   buff  cache   si   so     bi    bo   in   cs us sy id wa st34  0     0 200889792  73708 591828    0    0     0    &N bsp;5    6   10 96  1  3  0  032  0    0 200889920   73708 591860    0    0     0   592 13284 4282  98  1  1  0  032  0    0 200890112  73708 591860    0  & nbsp 0     0     0 9501 2154 99  1  0  0  032  0    0 200889568  73712 591856    0&nbsp   0     0    48 11900 2459 99  0  0  0  032  0& nbsp   0 200890208  73712 591860    0    0     0     0& nbsp;15898 4840 98  1  1  0  0^c#p# pagination title #e#

Vmstat (8) is the abbreviation for Virtual memory statistics, which is a common tool (created for BSD decades ago). It prints a statistical summary of a critical server on each line.

The Vmstat command specifies a parameter of 1 to run to print a statistical summary of each second. (This version of Vmstat) outputs the first row of those columns, which is explicitly the average since the boot, not the value of the previous second. Now, we skip the first line unless you want to understand and remember each column.

Check these columns:

• The number of processes that are running and waiting to run in R:CPU. It provides a better signal than the average load to determine whether the CPU is saturated because it does not contain I/O. Explanation: The value of "R" is greater than the number of CPUs, which means that it is saturated.

• Free: An explicitly-available memory in kilobytes. If you have a large number of digits, you have enough free memory. The "free-m" command, which is the seventh command below, provides a better indication of the state of the free memory.

• Si, So:swap-ins and swap-outs. If they are not 0, it means that you are running out of memory.

• US, SY, ID, WA, ST: These are the average CPU decomposition times for all CPUs. They are user time, system time (kernel), idle (idle), waiting for I/O (wait), and elapsed time (stolen) (by other visitors, or using Xen, the guest's own independent drive domain).

The CPU decomposition time will confirm whether the CPU is busy by user time and system time. The time to wait for I/O is constant, indicating a disk bottleneck, which is CPU idle because the task is blocked waiting for the pending disk I/O. You can think of waiting I/O as another form of CPU idle, which gives a clue as to why the CPU is idle.

System time is important for I/O processing. An average system time above 20% can be worth further discussion: Perhaps the kernel is too inefficient to handle I/O.

In the example above, CPU time is almost entirely at the user level, indicating that the application is taking up too much CPU time. The average CPU usage rate is also above 90%. This is not necessarily a problem; Check the saturation in the "R" column.

4. Mpstat-p all 1

$ mpstat -p all 1linux 3.13.0-49-generic  (titanclusters-xxxxx)   07/14/2015   _x86_64_  (32 CPU) 07:38:49 pm  cpu   %usr %nice   %sys % iowait   %irq %soft %steal %guest %gnice %idle07:38:50 pm  All   98.47   0.00   0.75    0.00   0.00   0.00    0.00& nbsp   0.00    0.00   0.7807:38:50 PM    0  96.04   0.00    2.97    0.00   0.00   0.00    0.00    0.00    0.00  &nbs p;0.9907:38:50 pm    1  97.00   0.00   1.00    0.00   0.00    0.00    0.00    0.00    0.00   2.0007:38:50 PM    2& nbsp 98.00   0.00   1.00    0.00&NBsp  0.00   0.00    0.00    0.00    0.00   1.0007:38:50 pm     3  96.97   0.00   0.00    0.00   0.00   0.00    0.00    0.00    0.00   3.03[...] #p # pagination Title #e#

This command prints the CPU decomposition time for each CPU, which can be used to check for an unbalanced usage. A single CPU is busy and represents an application that is running a single thread.

5. Pidstat 1

$ pidstat 1linux 3.13.0-49-generic  (titanclusters-xxxxx)   07/14/2015    _x86_ 64_    (32 CPU) 07:41:02 pm   UID       PID   %usr % system %guest   %cpu   CPU  command07:41:03 pm     0    & nbsp    9    0.00    0.94    0.00    0.94     1  rcuos/00  7:41:03 pm     0      4214    5.66    5.66    0.00   11.32    15  mesos-slave07:41:03 pm     0      4354    0.94    0.94    0.00    1.89     8  java07:41:03 pm  & nbsp  0      6521 1596.23    1.89    0.00 1598.11    27  java07:41:03 pm     0&NBSp     6564 1571.70    7.55    0.00 1579.25    28  java07:41:03  PM 60004     60154    0.94    4.72    0.00    5.66& nbsp    9  pidstat07:41:03 pm   UID       PID   %usr % system %guest   %cpu   CPU  command07:41:04 pm     0    & nbsp 4214    6.00    2.00    0.00    8.00    15  MESOS-SLAVE07:41:04&NB Sp pm     0      6521 1590.00    1.00    0.00 1591.00  & nbsp 27  java07:41:04 pm     0      6564 1573.00   10.00    0.00 1583.00    28  java07:41:04 pm   108      6718    1.00    0.00    0.00    1.00     0  snmp-pass07:41:04 pm 60004     60154    1.00    4.00    0.00    5.00     9  pidstat^c

The Pidstat command is a bit like the top command for each process's statistical summary, but cycles through a scrolling statistical summary to replace the top brush screen. It can be used for real-time viewing, as well as what you see (copy and paste) into your survey records.

The above example shows that two Java processes are consuming CPU. %cpu This column is the total of all CPUs, and 1591% means that the Java process consumes nearly 16 CPUs.

6. IOSTAT-XZ 1

$ iostat -xz 1linux 3.13.0-49-generic  (titanclusters-xxxxx)   07/14/2015  _x86 _64_  (32 CPU) avg-cpu: %user   %nice %system %iowait %steal   %  idle          73.96    0.00    3.73    0.03    0.06   22.21Device:   rrqm/s   wrqm/s     r/s     w/s    rkb/s    wkb/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %utilxvda        0.00     0.23    0.21    0.18     4 .52     2.08    34.37     0.00    9.98   13.80    5 .42   2.44   0.09xvdb        0.01     0.00    1.02     8.94   127.97   598.53   145.79     0.00    0.43    1.78    0.28   0.25    0.25xvdc        0.01     0.00    1.02    8.86   127.79&N Bsp  595.94   146.50     0.00    0.45    1.82    0.30  & nbsp;0.27   0.26dm-0        0.00     0.00    0.69    2.3 2    10.47    31.69    28.01     0.01    3.23    0.71&NBS P   3.98   0.13   0.04dm-1        0.00     0.00    0. 00    0.94     0.01     3.78     8.00     0.33& nbsp 345.84    0.04  346.81   0.01   0.00dm-2        0.00     0.00    0.09    0.07     1.35     0.36    22.50     0.00  &N Bsp 2.55    0.23    5.62   1.78   0.03[...] ^c#p# Pagination Title #e#

This is a great tool for viewing block device (disk) situations, whether it's for workloads or performance. To view a column:

• R/S, w/s, rkb/s, wkb/s: These represent the number of reads per second, number of writes, read KB, and KB written by the device. These are used to describe the workload. Performance problems may only be due to excessive load being exerted.

• Await: Average I/O elapsed time in milliseconds. This is the actual time that the application consumes because it includes the queue time and processing time. A larger average time than expected may mean saturation of the device, or a problem with the device.

• Avgqu-sz: The average number of requests made to the device. A value greater than 1 indicates that it is saturated (although the device can process the request in parallel, especially a virtual device consisting of multiple disks.) ）

• %util: Equipment utilization. This value is a percentage that shows how busy the device is at work per second. A value greater than 60% usually indicates a poor performance (as can be seen from the await), although it depends on the device itself. A value close to 100% usually means that it is saturated.

If the storage device is a logical disk device that targets many back-end disks, 100% utilization may simply mean that some I/O is currently being processed, however, the backend disk may be far from saturated and may be able to handle more work.

Keep in mind that poor disk I/O performance is not necessarily a program issue. Many technologies are usually asynchronous I/O, so that applications are not blocked and subject to delays (for example, read-ahead, and write buffering).

7. Free-m

$ free-m Total used free shared buffers cachedmem:245998 24545 221453 541-/+ buffers/cache:23944 222053swap:0 0 0

The two columns on the right are explicit:

• Buffers: Buffer cache for block device I/O.

• Cached: The page cache for the file system.

We just want to check that these are not close to 0 size, which can result in higher disk I/O (using Iostat acknowledgement), and worse performance. The example above looks good, and each column has many M sizes.

-/+ Buffers/cache provides more accurate memory usage than the first line. Linux uses memory that is temporarily unused as a cache and is reassigned to it as soon as the application needs it. So some of the memory that is used as a cache is actually an idle memory. To explain this, someone even built a website: linuxatemyram. #p # pagination Title #e#

This becomes even more confusing if you install ZFS on Linux, because ZFS has its own file system cache that does not count in Free-m. Sometimes it is found that the system does not have much free memory available, but in fact the memory is in the ZFS cache.

8. Sar-n DEV 1

$ sar -n dev 1linux 3.13.0-49-generic  (titanclusters-xxxxx)   07/14/2015      _x86_64_    (32 CPU) 12:16:48 am     IFACE   rxpck/s& nbsp  txpck/s    rxkb/s    txkb/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil12:16:49 AM      eth0  18763.00   5032.00  20686.42    478.30      0.00      0.00      0.00      0.0012:16:49 AM         lo     14.00     14.00      1.36    &NB Sp 1.36      0.00      0.00      0.00      0.0012:16:49 AM&N Bsp  docker0      0.00      0.00      0.00      0.00  &N Bsp   0.00      0.00      0.00      0.0012:16:49 am     IFACE   rxpck/s  & nbsp;txpck/s    rxkb/s    txkb/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil12:16:50 AM      eth0  19763.00   5101.00  21999.10    482.56      0.00      0.00      0.00      0.0012:16:50 AM         lo     20.00     20.00      3.25    &NB Sp 3.25      0.00      0.00      0.00      0.0012:16:50 AM&N Bsp  docker0      0.00      0.00      0.00      0.00  &N Bsp   0.00      0.00      0.00      0.00^C

This tool can be used to check the throughput of the network interface: RXKB/S and txkb/s, and whether the limit is reached. In the above example, the eth0 receives the traffic to reach 22MBYTES/S, which is 176mbits/sec (limit is 1gbit/sec)

The version we used also provided the%ifutil as an indicator of the device usage (maximum received and sent). We can also use Brendan's nicstat tool to measure this value. As Nicstat,sar shows, this value is difficult to obtain precisely, in this case, it does not work in normal (0.00).

9. Sar-n Tcp,etcp 1

$ sar -n tcp,etcp 1linux 3.13.0-49-generic  (titanclusters-xxxxx)   07/14/2015     _x86_64_    (32 CPU) 12:17:19 am  active/s passive/s    ISEG/S     oseg/s12:17:20 am      1.00      0.00  10233.00  18846.0012:17:19 am  atmptf/s  estres/s retrans/s isegerr/s   orsts/s12:17:20  AM      0.00      0.00      0.00      0.00      0.0012:17:20 am  active/s passive/s    iseg/s    OSEG/S12:17:21 AM       1.00      0.00   8359.00   6039.0012:17:20 AM  ATMPTF /s  estres/s retrans/s isegerr/s   orsts/s12:17:21 AM      0.00      0.00      0.00      0.00    &NBSp 0.00^c#p# Pagination Title #e#

This is a summary view of some of the key TCP metrics. These include:

• ACTIVE/S: Number of locally initiated TCP connections per second (for example, via Connect ()).

• PASSIVE/S: The number of TCP connections initiated remotely per second (for example, via accept ()).

• RETRANS/S: The number of TCP retransmissions per second.

The number of connections for active and passive is often useful for describing a rough measurement of server load: The number of newly accepted connections (passive), and the number of downstream connections (active). It can be understood that the active connection is external, and the passive connection is internal, although strictly not completely correct (for example, a localhost-to-localhost connection).

Retransmission is a symptom of a network and server problem. It may be caused by an unreliable network (for example, a public network), or it may be due to server overloading and packet loss. The example above shows that there is only one new TCP connection per second.

Ten. Top

$ toptop - 00:15:40 up 21:56,  1 user,  load average: 31.09,  29.87, 29.92Tasks: 871 total,   1 running, 868 sleeping,   0 stopped,   2 zombie%cpu (s): 96.8 us,  0.4 sy,  0.0 ni,   2.7 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 stkib mem:  25190241+total, 24921688 used, 22698073+free,    60448 bufferskib swap:        0 total,        0 used,        0 free.  & nbsp;554208 cached mem   PID USER      pr  ni    virt    res    shr s %cpu %mem     time+ command 20248 root       20   0  0.227t 0.012t  18748 s  3090  5.2  29812:58 java  4213 root      20   0 2722544  64640  44232 s  23.5  0.0 233:35.37 mesos-slave 66128 titancl+  20    0   24344   2332   1172 R   1.0  0.0    0:00.07 top  5235 root      20   0 38.227g 547004  49996  S   0.7  0.2   2:02.74 java  4299 root      20   0 20.015g 2.682g  16836 s   0.3  1.1  33:14.42 java     1 root      20   0   33620   2920   1496 s    0.0  0.0   0:03.82 init     2 root      20  & nbsp;0       0      0      0 s   0.0  0.0   0:00.02 kthreadd     3 root      20   0       0      0      0 s   0.0  0.0  &NB sp;0:05.35 ksoftirqd/0     5 root       0 -20      and nbsp;0      0      0 s   0.0  0.0   0:00.00 kworker/ 0:0h     6 root      20   0       0      0       0 s   0.0  0.0   0:06.94 kworker/u256:0     8& nbsp;root      20   0       0      0      0&NBS P s   0.0  0.0   2:38.05 rcu_sched#p# page title #e#

The top command contains a lot of metrics that we've checked before. It is easy to execute it to see the results compared to the previous command output are very different, which indicates that the load is variable.

One drawback of top is that it is hard to see the trend of data changing over time. The scrolling output provided by Vmstat and Pidstat will be clearer. If you do not pause the output fast enough (Ctrl-s pauses, Ctrl-q continues), some intermittent problem clues may also be lost due to a clear screen.

Follow-up analysis

There are more commands and methods that can be used for more in-depth analysis. View Brendan's Linux Performance tool tutorials at Velocity 2015, which contains more than 40 commands, covering the observable??? Benchmark management, tuning, static performance tuning, analysis, and tracking.

It is one of our hobbies to deal with the reliability and performance of the system in the whole network scale. If you want to join us in dealing with this challenge, we are recruiting!

How to diagnose the performance of Linux in 60,000 milliseconds