Diagnose the performance of Linux servers in one second

Performance diagnostics for Linux in 60,000 milliseconds

When you log on to a Linux server in order to solve a performance problem: What should you check in the first minute?

At Netflix, we have a huge EC2 Linux cloud, as well as a number of performance analysis tools to monitor and diagnose its performance. These include Atlas for cloud monitoring, and vectors for on-demand instance analysis. While these tools can help us solve most problems, we still sometimes need to log on to a server instance and run some standard Linux performance tools.

In this article, the Netflix performance Engineering team will explain to you what you should do in the first 60 seconds of the best performance analysis on the command line, using the standard Linux tools you should be able to get.

First 60 seconds: overview

By running the following 10 commands, you can get a rough idea of the processes and resource usage that the system is running within 60 seconds. By looking at the error messages and the resource saturation of these commands (they are all easy to understand), you can then optimize the resources. Saturation refers to the load of a resource that exceeds the limit it can handle. Once saturated, it is usually exposed to the length or wait time of the request queue.

Uptime

DMESG | Tail

Vmstat 1

Mpstat-p all 1

Pidstat 1

IOSTAT-XZ 1

Free-m

Sar-n DEV 1

Sar-n tcp,etcp 1

Top

Some of these commands require pre-installation of the Sysstat package. The information presented by these commands can help you implement the use method, a method for locating performance bottlenecks, such as checking usage, saturation, and error information for various resources such as CPU, memory, disk, and so on. In addition to locating the problem, you can use these commands to troubleshoot some of the possible problems that may be causing the problem, to help you narrow the scope of the check, and to indicate the direction for the next check.

The following sections briefly describe these commands as an example of executing these commands in a production environment. For more information on how these tools are used, refer to their man documentation.

1. Uptime

$ uptime

23:51:26 up 21:31, 1 user, load average:30.02, 26.43, 19.02

This is a quick way to see the average load on the system, indicating how many tasks (processes) are running in the system. In a Linux system, these numbers contain processes that need to run in the CPU and that are waiting for I/O (usually disk I/O). It's just a rough display of the system load, a little bit of a look. You'll also need additional tools to learn more about the situation.

These three numbers show the results of the total load average of the system in one minute, five minutes, and 15 minutes compressed by the exponential ratio. From this we can see how the load of the system changes over time. For example, if you are checking a question and then see that the value of 1 minutes corresponds to a value that is much less than 15 minutes, then it may be that the problem has passed and you have not been able to observe it in time.

In this example, the system load increases over time because the load value of the last minute exceeds 30, while the average load of 15 minutes is only 19. Such a significant gap includes many meanings, such as CPU load. To further confirm, run the Vmstat or Mpstat command, which refer to the 3rd and 4th chapters later.

2. DMESG | Tail

$ DMESG | TAIL[1880957.563150] Perl invoked Oom-killer:gfp_mask=0x280da, order=0, oom_score_adj=0

[...]

[1880957.563400] out of Memory:kill process 18694 (perl) score 246 or sacrifice child

[1880957.563408] killed process 18694 (Perl) total-vm:1972392kb, ANON-RSS:1953348KB, file-rss:0kb

[2320864.954447] tcp:possible SYN flooding on port 7001.  Dropping request. Check SNMP counters.

This command explicitly shows the last 10 system messages if they exist. Look for errors that can cause performance problems. The above example contains the Oom-killer, and TCP discards a request.

Don't miss this step! The DMESG command is always worth a try.

3. Vmstat 1

$ vmstat 1procs---------memory-------------Swap-------io-----system--------CPU-----

R b swpd free buff cache si so bi bo in CS us sy ID WA St

34 0 0 200889792 73708 591828 0 0 0 5 6 10 96 1 3 0 0

32 0 0 200889920 73708 591860 0 0 0 592 13284 4282 98 1 1 0 0

32 0 0 200890112 73708 591860 0 0 0 0 9501 2154 99 1 0 0 0

32 0 0 200889568 73712 591856 0 0 0 48 11900 2459 99 0 0 0 0

32 0 0 200890208 73712 591860 0 0 0 0 15898 4840 98 1 1 0 0

^c

Vmstat (8) is the abbreviation for Virtual memory statistics, which is a common tool (created for BSD decades ago). It prints a statistical summary of a critical server on each line.

The Vmstat command specifies a parameter of 1 to run to print a statistical summary of each second. (This version of Vmstat) outputs the first row of those columns, which is explicitly the average since the boot, not the value of the previous second. Now, we skip the first line unless you want to understand and remember each column.

Check these columns:

• The number of processes that are running and waiting to run in R:CPU. It provides a better signal than the average load to determine whether the CPU is saturated because it does not contain I/O. Explanation: The value of "R" is greater than the number of CPUs, which means that it is saturated.

• Free: An explicitly-available memory in kilobytes. If you have a large number of digits, you have enough free memory. The "free-m" command, which is the seventh command below, provides a better indication of the state of the free memory.

• Si, So:swap-ins and swap-outs. If they are not 0, it means that you are running out of memory.

• US, SY, ID, WA, ST: These are the average CPU decomposition times for all CPUs. They are user time, system time (kernel), idle (idle), waiting for I/O (wait), and elapsed time (stolen) (by other visitors, or using Xen, the guest's own independent drive domain).

The CPU decomposition time will confirm whether the CPU is busy by user time and system time. The time to wait for I/O is constant, indicating a disk bottleneck, which is CPU idle because the task is blocked waiting for the pending disk I/O. You can think of waiting I/O as another form of CPU idle, which gives a clue as to why the CPU is idle.

System time is important for I/O processing. An average system time above 20% can be worth further discussion: Perhaps the kernel is too inefficient to handle I/O.

In the example above, CPU time is almost entirely at the user level, indicating that the application is taking up too much CPU time. The average CPU usage rate is also above 90%. This is not necessarily a problem; Check the saturation in the "R" column.

4. Mpstat-p all 1

$ mpstat-p All 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (CPU)

07:38:49 PM CPU%usr%nice%sys%iowait%irq%soft%steal%guest%gnice%idle

07:38:50 PM All 98.47 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.78

07:38:50 PM 0 96.04 0.00 2.97 0.00 0.00 0.00 0.00 0.00 0.00 0.99

07:38:50 PM 1 97.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00

07:38:50 PM 2 98.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00

07:38:50 PM 3 96.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.03

[...]

This command prints the CPU decomposition time for each CPU, which can be used to check for an unbalanced usage. A single CPU is busy and represents an application that is running a single thread.

5. Pidstat 1

$ pidstat 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (CPU)

07:41:02 PM UID PID%usr%system%guest%cpu CPU Command

07:41:03 PM 0 9 0.00 0.94 0.00 0.94 1 rcuos/0

07:41:03 PM 0 4214 5.66 5.66 0.00 11.32

07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8 Java

07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 Java

07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 Java

07:41:03 PM 60004 60154 0.94 4.72 0.00 5.66 9 Pidstat

07:41:03 PM UID PID%usr%system%guest%cpu CPU Command

07:41:04 PM 0 4214 6.00 2.00 0.00 8.00

07:41:04 PM 0 6521 1590.00 1.00 0.00 1591.00 Java

07:41:04 PM 0 6564 1573.00 10.00 0.00 1583.00 Java

07:41:04 PM 108 6718 1.00 0.00 0.00 1.00 0 Snmp-pass

07:41:04 PM 60004 60154 1.00 4.00 0.00 5.00 9 Pidstat

^c

The Pidstat command is a bit like the top command for each process's statistical summary, but cycles through a scrolling statistical summary to replace the top brush screen. It can be used for real-time viewing, as well as what you see (copy and paste) into your survey records.

The above example shows that two Java processes are consuming CPU. %cpu This column is the total of all CPUs, and 1591% means that the Java process consumes nearly 16 CPUs.

6. IOSTAT-XZ 1

$ iostat-xz 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (CPU)

AVG-CPU:%user%nice%system%iowait%steal%idle

73.96 0.00 3.73 0.03 0.06 22.21

device:rrqm/s wrqm/s r/s w/s rkb/s wkb/s avgrq-sz avgqu-sz await r_await w_await SVCTM%util

Xvda 0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09

Xvdb 0.01 0.00 1.02 8.94 127.97 598.53 145.79 0.00 0.43 1.78 0.28 0.25 0.25

XVDC 0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26

DM-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04

Dm-1 0.00 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.01 0.00

Dm-2 0.00 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03

[...]

^c

This is a great tool for viewing block device (disk) situations, whether it's for workloads or performance. To view a column:

• R/S, w/s, rkb/s, wkb/s: These represent the number of reads per second, number of writes, read KB, and KB written by the device. These are used to describe the workload. Performance problems may only be due to excessive load being exerted.

• Await: Average I/O elapsed time in milliseconds. This is the actual time that the application consumes because it includes the queue time and processing time. A larger average time than expected may mean saturation of the device, or a problem with the device.

• Avgqu-sz: The average number of requests made to the device. A value greater than 1 indicates that it is saturated (although the device can process the request in parallel, especially a virtual device consisting of multiple disks.) ）

• %util: Equipment utilization. This value is a percentage that shows how busy the device is at work per second. A value greater than 60% usually indicates a poor performance (as can be seen from the await), although it depends on the device itself. A value close to 100% usually means that it is saturated.

If the storage device is a logical disk device that targets many back-end disks, 100% utilization may simply mean that some I/O is currently being processed, however, the backend disk may be far from saturated and may be able to handle more work.

Keep in mind that poor disk I/O performance is not necessarily a program issue. Many technologies are usually asynchronous I/O, so that applications are not blocked and subject to delays (for example, read-ahead, and write buffering).

7. Free-m

$ free-m

Total used free shared buffers Cached

mem:245998 24545 221453 83 59 541

-/+ buffers/cache:23944 222053

swap:0 0 0

The two columns on the right are explicit:

• Buffers: Buffer cache for block device I/O.

• Cached: The page cache for the file system.

We just want to check that these are not close to 0 size, which can result in higher disk I/O (using Iostat acknowledgement), and worse performance. The example above looks good, and each column has many M sizes.

-/+ Buffers/cache provides more accurate memory usage than the first line. Linux uses memory that is temporarily unused as a cache and is reassigned to it as soon as the application needs it. So some of the memory that is used as a cache is actually an idle memory. To explain this, someone even built a website: linuxatemyram.

This becomes even more confusing if you install ZFS on Linux, because ZFS has its own file system cache that does not count in Free-m. Sometimes it is found that the system does not have much free memory available, but in fact the memory is in the ZFS cache.

8. Sar-n DEV 1

$ sar-n DEV 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (CPU)

12:16:48 AM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s%ifutil

12:16:49 AM eth0 18763.00 5032.00 20686.42 478.30 0.00 0.00 0.00 0.00

12:16:49 AM Lo 14.00 14.00 1.36 1.36 0.00 0.00 0.00 0.00

12:16:49 AM DOCKER0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

12:16:49 AM IFACE rxpck/s txpck/s rxkb/s txkb/s rxcmp/s txcmp/s rxmcst/s%ifutil

12:16:50 AM eth0 19763.00 5101.00 21999.10 482.56 0.00 0.00 0.00 0.00

12:16:50 AM Lo 20.00 20.00 3.25 3.25 0.00 0.00 0.00 0.00

12:16:50 AM DOCKER0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

^c

This tool can be used to check the throughput of the network interface: RXKB/S and txkb/s, and whether the limit is reached. In the above example, the eth0 receives the traffic to reach 22MBYTES/S, which is 176mbits/sec (limit is 1gbit/sec)

The version we used also provided the%ifutil as an indicator of the device usage (maximum received and sent). We can also use Brendan's nicstat tool to measure this value. As Nicstat,sar shows, this value is difficult to obtain precisely, in this case, it does not work in normal (0.00).

9. Sar-n Tcp,etcp 1

$ sar-n tcp,etcp 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (CPU)

12:17:19 AM active/s passive/s iseg/s oseg/s

12:17:20 AM 1.00 0.00 10233.00 18846.00

12:17:19 AM atmptf/s estres/s retrans/s isegerr/s orsts/s

12:17:20 AM 0.00 0.00 0.00 0.00 0.00

12:17:20 AM active/s passive/s iseg/s oseg/s

12:17:21 AM 1.00 0.00 8359.00 6039.00

12:17:20 AM atmptf/s estres/s retrans/s isegerr/s orsts/s

12:17:21 AM 0.00 0.00 0.00 0.00 0.00

^c

This is a summary view of some of the key TCP metrics. These include:

• ACTIVE/S: Number of locally initiated TCP connections per second (for example, via Connect ()).

• PASSIVE/S: The number of TCP connections initiated remotely per second (for example, via accept ()).

• RETRANS/S: The number of TCP retransmissions per second.

The number of connections for active and passive is often useful for describing a rough measurement of server load: The number of newly accepted connections (passive), and the number of downstream connections (active). It can be understood that the active connection is external, and the passive connection is internal, although strictly not completely correct (for example, a localhost-to-localhost connection).

Retransmission is a symptom of a network and server problem. It may be caused by an unreliable network (for example, a public network), or it may be due to server overloading and packet loss. The example above shows that there is only one new TCP connection per second.

Ten. Top

$ toptop-00:15:40 up 21:56, 1 user, load average:31.09, 29.87, 29.92

tasks:871 Total, 1 running, 868 sleeping, 0 stopped, 2 zombie

%CPU (s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 ID, 0.1 wa, 0.0 hi, 0.0 si, 0.0th

KiB mem:25190241+total, 24921688 used, 22698073+free, 60448 buffers

KiB swap:0 Total, 0 used, 0 free. 554208 Cached Mem

PID USER PR NI VIRT RES SHR S%cpu%MEM time+ COMMAND

20248 Root 0 0.227t 0.012t 18748 S 3090 5.2 29812:58 java

4213 Root 0 2722544 64640 44232 S 23.5 0.0 233:35.37 Mesos-slave

66128 titancl+ 0 24344 2332 1172 R 1.0 0.0 0:00.07 Top

5235 Root 0 38.227g 547004 49996 S 0.7 0.2 2:02.74 java

4299 Root 0 20.015g 2.682g 16836 S 0.3 1.1 33:14.42 Java

1 root 0 33620 2920 1496 S 0.0 0.0 0:03.82 Init

2 Root 0 0 0 0 S 0.0 0.0 0:00.02 Kthreadd

3 Root 0 0 0 0 S 0.0 0.0 0:05.35 ksoftirqd/0

5 root 0-20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0h

6 Root 0 0 0 0 S 0.0 0.0 0:06.94 kworker/u256:0

8 Root 0 0 0 0 S 0.0 0.0 2:38.05 rcu_sched

The top command contains a lot of metrics that we've checked before. It is easy to execute it to see the results compared to the previous command output are very different, which indicates that the load is variable.

One drawback of top is that it is hard to see the trend of data changing over time. The scrolling output provided by Vmstat and Pidstat will be clearer. If you do not pause the output fast enough (Ctrl-s pauses, Ctrl-q continues), some intermittent problem clues may also be lost due to a clear screen.

Follow-up analysis

There are more commands and methods that can be used for more in-depth analysis. View Brendan's Linux Performance tool tutorials at Velocity 2015, which contains more than 40 commands covering observability, benchmarking, tuning, static performance tuning, analysis, and tracking.

Diagnose the performance of Linux servers in one second