Linux performance testing and monitoring sar commands

Linux performance testing and monitoring sar commands

Sar (System Activity Reporter System Activity Report) is one of the most comprehensive System performance analysis tools on Linux. It can report System activities from multiple parties, including: file read/write, system call usage, disk I/O, CPU efficiency, memory usage, process activity, and IPC-related activity. This document describes the sar commands in CentOS6.3 x64.

Common sar command formats

Sar [options] [-A] [-o file] t [n]

Where:

T indicates the sampling interval, and n indicates the number of samples. The default value is 1;

-O file: stores the command results in binary format. file is the file name.

Options is the command line option. Common options for sar commands are as follows:

-A: total of all reports

-U: outputs CPU usage statistics

-V: outputs statistics of inode, files, and other kernel tables.

-D: outputs the activity information of each block device.

-R: Output memory and swap space statistics

-B: displays the I/O and transfer rate statistics.

-A: file read/write status

-C: output process statistics, number of processes created per second

-R: Output Memory Page statistics

-Y: terminal device activity

-W: Output System exchange activity information

1. CPU resource monitoring

For example, sample once every 10 seconds, sample three times in a row, observe the CPU usage, and save the sample result in binary format to the file test in the current directory. You need to enter the following command:

Sar-u-o test 10 3

The screen is shown as follows:

17:06:16 CPU % user % nice % system % iowait % steal % idle

17:06:26 all 0.00 0.00 0.20 0.00 0.00

17:06:36 all 0.00 0.00 0.20 0.00 0.00

17:06:46 all 0.00 0.00 0.10 0.00 0.00

Average: all 0.00 0.00 0.17 0.00 0.00

Output description:

CPU: all indicates that the statistical information is the average value of all CPUs.

% User: displays the percentage of total CPU usage at the user level (application.

% Nice: displays the percentage of the total CPU usage for nice operations at the user level.

% System: Percentage of total CPU usage at the core level (kernel.

% Iowait: displays the percentage of CPU usage for I/O operations.

% Steal: the percentage of virtual CPUs waiting for hypervisor to provide services for another virtual process.

% Idle: displays the percentage of CPU consumed by idle time.

1. If % iowait is too high, it indicates that the hard disk has an I/O bottleneck.

2. If the value of % idle is high but the system response is slow, it may be that the CPU waits for memory allocation. In this case, the memory capacity should be increased.

3. If the value of % idle is lower than 1, the CPU processing capability of the system is relatively low, indicating that the most important resource to be solved in the system is the CPU.

To view the content in the binary file test, enter the following sar command:

Sar-u-f test

2. inode, file, and other kernel Table Monitoring

For example, if you sample the table every 10 seconds and sample the table three times in a row to observe the status of the core table, enter the following command:

Sar-v 10 3

The screen is shown as follows:

17:10:49 dentunusd file-nr inode-nr pty-nr

17:10:59 6301 5664 12037 4

17:11:09 6301 5664 12037 4

17:11:19 6301 5664 12037 4

Average: 6301 5664 12037 4

Output description:

Dentunusd: number of unused entries in the directory Cache

File-nr: the number of file handles used.

Inode-nr: inode handle

Pty-nr: Number of pty used

3. Memory and swap space monitoring

For example, sampling is performed once every 10 seconds, and sampling is performed three times in a row. Monitoring memory paging:

Sar-r 10 3

The screen is shown as follows:

Output description:

Kbmemfree: this value is basically the same as the free value in the free command, so it does not include the buffer and cache space.

Kbmemused: this value is basically the same as the used value in the free command, so it includes the buffer and cache space.

% Memused: this value is a percentage of kbmemused and total memory (excluding swap.

Kbbuffers and kbcached: These two values are buffer and cache in the free command.

Kbcommit: Ensure the memory required by the current system, that is, the memory required to ensure no overflow (RAM + swap ).

% Commit: this value is a percentage of kbcommit and total memory (including swap.

4. Memory paging monitoring

For example, sampling is performed once every 10 seconds, and sampling is performed three times in a row. Monitoring memory paging:

Sar-B 10 3

The screen is shown as follows:

Output description:

Pgpgin/s: the number of bytes per second from disk or SWAP to memory (KB)

Pgpgout/s: the number of bytes per second from memory replacement to disk or SWAP (KB)

Fault/s: number of missing pages generated by the system per second, that is, the sum of the Primary and Secondary missing pages (major + minor)

Majflt/s: Number of Primary missing pages per second.

Pgfree/s: Number of pages per second in the idle queue

Pgscank/s: Number of pages scanned by kswapd per second

Pgscand/s: number of directly scanned pages per second

Pgsteal/s: the number of pages that are cleared from the cache every second to meet memory requirements

% Vmeff: Percentage of pages cleared per second (pgsteal) to total scanning pages (pgscank + pgscand)

5. I/O and Transfer Rate Monitoring

For example, if you sample data every 10 seconds and sample data three times in a row to report the usage of the buffer, you need to enter the following command:

Sar-B 10 3

The screen is shown as follows:

18:51:05 tps rtps wtps bread/s bwrtn/s

18:51:15 0.00 0.00 0.00 0.00 0.00

18:51:25 1.92 0.00 1.92 0.00 22.65

18:51:35 0.00 0.00 0.00 0.00 0.00

Average: 0.64 0.00 0.64 0.00 7.59

Output description:

Tps: the total amount of I/O transferred by physical devices per second

Rtps: Total amount of data read from physical devices per second

Wtps: Total amount of data written to the physical device per second

Bread/s: the amount of data read from the physical device per second. Unit: block/s.

Bwrtn/s: the volume of data written to the physical device per second. Unit: block/s.

6. Process queue length and average load Status Monitoring

For example, sample once every 10 seconds and sample three times in a row to monitor the length of the process queue and the average load status:

Sar-q 10 3

The screen is shown as follows:

19:25:50 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15

19:26:00 0 259 0.00 0.00 0.00

19:26:10 0 259 0.00 0.00 0.00

19:26:20 0 259 0.00 0.00 0.00

Average: 0 259 0.00 0.00

Output description:

Runq-sz: length of the running Queue (number of processes awaiting running)

Plist-sz: Number of processes and threads in the Process List

Ldavg-1: average System load for the last minute (System load average)

Ldavg-5: average system load for the past 5 minutes

Ldavg-15: average system load for the past 15 minutes

7. Monitoring of system exchange activity information

For example, if sampling is performed once every 10 seconds and three consecutive sampling times, the monitoring system will exchange activity information:

Sar-W 10 3

The screen is shown as follows:

19:39:50 pswpin/s pswpout/s

19:40:00 0.00 0.00

19:40:10 0.00 0.00

19:40:20 0.00 0.00

Average: 0.00 0.00

Output description:

Pswpin/s: Number of swap pages per second

Pswpout/s: Number of swap pages generated by the system per second

8. device usage monitoring

For example, if you want to sample the device every 10 seconds for three consecutive times and report the device usage, enter the following command:

# Sar-d 10 3-p

The screen is shown as follows:

17: 45: 54DEVtpsrd_sec/swr_sec/savgrq-szavgqu-szawaitsvctm % util

17: 46: 04cd00. 000.000.000.000.000.000.000.000.000.00

17: 46: 04sda0. 000.000.000.000.000.000.000.000.00

17: 46: 04vg_livedvd-lv_root0.000.000.000.000.000.000.000.000.00

17: 46: 04vg_livedvd-lv_swap0.000.000.000.000.000.000.000.000.00

Where:

Parameter-p can print out the sda, hdc and other disk device names, if you do not have the parameter-p, the device node may be a dev8-0, dev22-0

Tps: the number of I/O requests from the physical disk per second. Multiple logical requests are merged into one I/O disk request. The size of one transmission is uncertain.
Rd_sec/s: Number of read sectors per second.

Wr_sec/s: Number of write sectors per second.

Avgrq-sz: average data size (slice) of each device I/O operation ).

Avgqu-sz: average length of the disk Request queue.

Await: Average time consumed by each request, from the request disk operation to the system processing, including the request queue waiting time, in milliseconds (1 second = 1000 milliseconds ).

Svctm: average time for the system to process each request, excluding the time consumed in the Request queue.

% Util: Percentage of I/O requests to CPU. The larger the ratio, the more saturated the request.

1. When avgqu-sz is low, the utilization rate of the device is high.

2. When the value of % util is close to 1%, the bandwidth of the device is full.

To determine the system bottleneck, you sometimes need to combine several sar command options.

Suspected CPU bottlenecks, which can be viewed by sar-u and sar-q.

Suspected memory bottlenecks, which can be viewed by sar-B, sar-r, and sar-W.

I/O bottlenecks are suspected, which can be viewed by sar-B, sar-u, and sar-d.