A tutorial on the use of SAR command monitoring system in Linux

The SAR (System Activity Reporter report) is one of the most comprehensive performance analysis tools available on Linux, and can be used to report the activities of the system in many ways.
Includes read and write files, usage of system calls, disk I/O, CPU efficiency, memory usage, process activity, and IPC-related activities.
This paper mainly takes CentOS 6.3 x64 system as an example to introduce SAR command.
1, the SAR command commonly used format
SAR [Options] [-A] [-o file] t [n]
which
T is the sampling interval, n is the sampling number, and the default value is 1;
The-o file indicates that the command results are stored in a binary format in a file, which is the file name.
The options are command-line options, and the SAR commands are commonly used as follows:

-A: Sum of all reports
-P: Set CPU
-U: Output CPU Usage statistics
-V: Output statistics for inode, file, and other kernel tables
-D: Output activity information for each block device
-N: Reporting Network conditions
-Q: Reporting queue length and load information
-R: Statistical information for output memory and swap space
-B: Display statistics for I/O and transfer rates
-A: File read/write status
-C: Output process statistics, number of processes created per second
-r: Output memory page statistics
-Y: Terminal equipment activity
-W: Output system Exchange activity information
2. CPU Resource Monitoring
For example, sampling every 10 seconds, sampling 3 consecutive times, observing CPU usage, and storing the sample results in binary form in the file Sys_info of the current directory, type the following command:

The code is as follows:

Sar-u-O sys_info 10 3

The screen appears as follows:
17:06:16 CPU%user%nice%system%iowait%steal
17:06:26 All 0.00 0.00 0.20 0.00 0.00 99.80
17:06:36 All 0.00 0.00 0.20 0.00 0.00 99.80
17:06:46 All 0.00 0.00 0.10 0.00 0.00 99.90
Average:all 0.00 0.00 0.17 0.00 0.00 99.83
Output Item Description:
Cpu:all indicates that the statistic is the average of all CPUs.
%user: Displays the percentage of total CPU time that was run at the user level (application).
%nice: The percentage of total CPU time that is displayed at the user level for nice operations.
%system: The percentage of total CPU time used at the core level (kernel) run.
%iowait: Displays the percentage of total CPU time that is used to wait for I/O operations.
%steal: hypervisor (hypervisor) provides services to another virtual process and waits for the percentage of virtual CPUs.
%idle: Displays the percentage of total CPU time that CPU idle time consumes.
Note:
(1). If the value of%iowait is too high, the hard disk has I/O bottleneck
(2). If the%idle value is high but the system response is slow, it is possible that the CPU is waiting to allocate memory, should increase the memory capacity
(3). If the value of%idle is less than 1, the CPU processing ability of the system is relatively low, which indicates that the most necessary resource in the system is CPU.
Because Sys_info is the content of binary files, when you look at Sys_info with Cat are all garbled, oh, do not worry, SAR prepared for you-f filename options, you only use-f set to read the information stored files, you can clearly read out the information.

Like what

The code is as follows:

Sar-f Sys_info

If my CPU is a multi-core processor, can SAR know the running information of a certain kernel?
No problem at all. There is an option-p, which is designed to be used for multi-core processors.
When the-P option is not set with the SAR command, the SAR gives a macroscopic report based on all the cores, which is the average value.
If the-P option is used to specify a kernel, specific performance information is given for that individual kernel.
When the-P-all is used, SAR gives its specific performance information based on each kernel, and then gives a general performance information.

For example, I have a Xeon processor CPU, is a dual-core CPU, see the use of-P:

The code is as follows:

Sar-p all 1 1

Linux 2.6.9 10/16/2009

10:59:38 PM CPU%user%nice%system%iowait%idle
10:59:39 PM All 2.12 0.00 2.87 0.00 95.01
10:59:39 PM 0 0.00 0.00 1.98 0.00 98.02
10:59:39 PM 1 9.00 0.00 7.00 0.00 84.00
SAR will give performance information based on each of the processor's cores. When we want to view the information for the 6th kernel, the output is as follows:

The code is as follows:

Sar-p 0 1 1

3. Inode, file and other kernel table monitoring
For example, sampling every 10 seconds, sampling 3 consecutive times, and observing the status of the core table, type the following command:

The code is as follows:

Sar-v 10 3

The screen appears as follows:
17:10:49 DENTUNUSD File-nr Inode-nr pty-nr
17:10:59 6301 5664 12037 4
17:11:09 6301 5664 12037 4
17:11:19 6301 5664 12037 4
average:6301 5664 12037 4
Output Item Description:
DENTUNUSD: Number of entries not used in the directory cache
FILE-NR: Number of uses for file handles (files handle)
INODE-NR: Number of uses of index node handle (inode handle)
PTY-NR: Number of Pty used
4. Memory and swap space monitoring
For example, sampling every 10 seconds, sampling 3 consecutive times, monitoring memory paging:

The code is as follows:

Sar-r 10 3

The

screen appears as follows:
10:02:52 PM kbmemfree kbmemused %memused kbbuffers  kbcached % Commit
10:03:02 pm   2289016   1632096     41.62    204860   1218352    333068      8.49
10:03:12 PM   2288388   1632724     41.64    204860   1218352     333068      8.49
10:03:22 pm   2288544   1632568      41.64    204860   1218352    333068       8.49
average:        2288649   1632463      41.63    204860   1218352    333068      8.49

Output Item Description:
Kbmemfree: This value is basically the same as the free value in the null command, so it does not include buffer and cache space.
Kbmemused: This value is basically consistent with the used value in the free command, so it includes buffer and cache space.
%memused: This value is a percentage of kbmemused and total memory (excluding swap).
Kbbuffers and kbcached: These two values are the buffer and cache in the free command.
Kbcommit: The memory needed to ensure that the current system is needed, that is, the memory (RAM+SWAP) required to make sure it does not overflow.
%commit: This value is a percentage of kbcommit and total memory, including swap.
5. Memory Paging Monitor
For example, sampling every 10 seconds, sampling 3 consecutive times, monitoring memory paging:

The code is as follows:

Sar-b 10 3

The screen appears as follows:

Output Item Description:
PGPGIN/S: The number of bytes per second that are replaced from disk or swap to memory (KB)
PGPGOUT/S: The number of bytes per second that are being replaced from memory to disk or swap (KB)
FAULT/S: The number of pages per second system generated, that is, the main page fault and the second missing pages (major + minor)
MAJFLT/S: The number of main pages that are generated per second.
PGFREE/S: Number of pages placed in the free queue per second
PGSCANK/S: Number of pages scanned by KSWAPD per second
PGSCAND/S: Number of pages scanned directly per second
PGSTEAL/S: The number of pages that are purged from the cache per second to meet memory needs
%vmeff: Percentage of pages cleared per second (pgsteal) as a total scan page (Pgscank+pgscand)
6. I/O and transmission rate monitoring
For example, sampling every 10 seconds, sampling 3 consecutive times, and reporting the use of buffers, type the following command:

The code is as follows:

Sar-b 10 3

The screen appears as follows:
18:51:05 TPs Rtps Wtps bread/s bwrtn/s
18:51:15 0.00 0.00 0.00 0.00 0.00
18:51:25 1.92 0.00 1.92 0.00 22.65
18:51:35 0.00 0.00 0.00 0.00 0.00
average:0.64 0.00 0.64 0.00 7.59
Output Item Description:
TPS: Total I/O transfer of physical devices per second
Rtps: Total amount of data read from physical devices per second
Wtps: The total amount of data written to the physical device per second
BREAD/S: The amount of data read from the physical device per second, in block/s
BWRTN/S: The amount of data written to the physical device per second, in block/s
7. Process Queue Length and average load state monitoring
For example, sampling every 10 seconds, sampling 3 consecutive times, monitoring process queue length and average load status:

The code is as follows:

Sar-q 10 3

The screen appears as follows:
19:25:50 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
19:26:00 0 259 0.00 0.00 0.00
19:26:10 0 259 0.00 0.00 0.00
19:26:20 0 259 0.00 0.00 0.00
Average:0 259 0.00 0.00 0.00
Output Item Description:
Runq-sz: Length of Run queue (number of processes waiting to run)
Plist-sz: Number of processes (processes) and threads (threads) in the process list
Ldavg-1: System load average for last 1 minutes
LDAVG-5: Average system load of the last 5 minutes
LDAVG-15: Average system load of the last 15 minutes
8. System Exchange activity Information monitoring
For example, sampling every 10 seconds, sampling 3 consecutive times, monitoring system exchange activity information:

The code is as follows:

Sar-w 10 3

The screen appears as follows:
19:39:50 PSWPIN/S pswpout/s
19:40:00 0.00 0.00
19:40:10 0.00 0.00
19:40:20 0.00 0.00
average:0.00 0.00
Output Item Description:
PSWPIN/S: Number of swap pages (Swap page) exchanged in system per second
PSWPOUT/S: Number of swap pages (Swap page) swapped out per second
9. Monitoring of equipment usage
For example, sampling every 10 seconds, sampling 3 consecutive times, and reporting device usage, type the following command:

The code is as follows:

# sar-d 10 3–p

The screen appears as follows:
17:45:54 DEV TPs rd_sec/s wr_sec/s Avgrq-sz Avgqu-sz await SVCTM
17:46:04 scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 SDA 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 vg_livedvd-lv_root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 Vg_livedvd-lv_swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
which
The parameter-P can print out the name of a disk device such as SDA,HDC, and if you do not use the parameter-p, the device node may be dev8-0,dev22-0
TPS: Number of times per second from physical disk I/O. Multiple logical requests are merged into one I/O disk request, and the size of one transmission is indeterminate.
RD_SEC/S: Number of sectors read per second.
WR_SEC/S: Number of Write sectors per second.
Avgrq-sz: The average data size (sector) per device I/O operation.
Avgqu-sz: The average length of the disk request queue.
Await: The average elapsed time per request, including the request queue wait time, in milliseconds (1 seconds =1000 milliseconds), from the request disk operation to the system completion process.
SVCTM: The average time that the system processes each request, excluding the time that is consumed in the request queue.
%UTIL:I/O request as a percentage of the CPU, the greater the ratio, the more saturated the description.
(1). When the value of Avgqu-sz is low, the utilization of the device is high.
(2). When the value of the%util is close to 100%, the device bandwidth is fully occupied.
10, to determine the system bottleneck problem, sometimes need several SAR command options to combine
Suspected CPU bottlenecks, available sar-u and sar-q, etc. to view
Suspected memory bottlenecks can be viewed with sar-b, Sar-r, and Sar-w.
Suspected I/O bottlenecks, available sar-b, sar-u, and sar-d to view

11, the use of SAR to do a background monitoring program, can report the performance of the machine real-time. Can SAR run in the background?
With the Linux background character on it, do not forget to redirect the standard output Oh, the method is this:

The code is as follows:

Sar-o monitor.res interval Count >/dev/null 2>&1 &

Remember to replace interval and count with the intervals and times you want. So all the performance information is stored in binary format data file monitor.res. It is OK to use-f when reading.

12, SAR analysis of network card traffic

The code is as follows:

sar-n {DEV | Edev | NFS | NFSD | Sock | All}

SAR provides six different syntax options for displaying network information. -N option uses 6 different switches: DEV | Edev | NFS | NFSD | Sock | All. DEV Displays network interface information, Edev displays statistics about network errors, NFS client Information for NFS statistics activities, NFSD statistics Server for NFS, sock displays socket information, all displays all 5 switches. They can be used individually or together.
If you use the DEV keyword, then the SAR will report information related to the network device, such as Lo,eth0 or eth1, for example

The code is as follows:

#sar-N DEV 2 10

Linux 2.6.18-53.el5pae (Localhost.localdomain) 03/29/2009
01:39:40 AM iface rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
01:39:42 AM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:39:42 AM eth1 131.34 104.98 119704.48 36110.45 0.00 0.00 0.00
01:39:42 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:39:42 AM iface rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
01:39:44 AM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:39:44 AM eth1 168.00 165.50 114496.50 83938.50 0.00 0.00 0.00
01:39:44 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Iface:lan interface, name of network device
RXPCK/S: Packets Received per second
TXPCK/S: Packets Sent per second
rxbyt/s: Number of bytes received per second
txbyt/s: Number of bytes sent per second
RXCMP/S: Compressed packets received per second
TXCMP/S: Compressed packets sent per second
RXMCST/S: Multicast packets received per second
Using the Edev keyword, the failure condition is reported against the network device, for example:

The code is as follows:

#sar-N Edev 2 10

Linux 2.6.18-53.el5pae (Localhost.localdomain) 03/29/2009
01:42:18 AM iface rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s
01:42:20 AM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:42:20 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:42:20 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Name of the Iface:lan interface network device
RXERR/S: Bad packets received per second
TXERR/S: Bad packets sent per second
COLL/S: Number of conflicts per second
RXDROP/S: The number of received packets dropped per second because the buffer is full
TXDROP/S: The number of sent packets dropped per second because the buffer is full
TXCARR/S: Number of carrier errors per second when sending packets
RXFRAM/S: Number of frame alignment errors received for packets per second
RXFIFO/S: The number of FIFO errors per second received packets
TXFIFO/S: The number of FIFO errors per second sent packets
With the sock keyword, the socket connection is reported:

The code is as follows:

#sar-N Sock 2 10

Linux 2.6.18-53.el5pae (localhost.localdomain) 03/29/2009
01:44:32 AM totsck tcpsck udpsck rawsck ip-frag
01:4 4:34 AM 243 9 8 0 0
01:44:36 am 242 9 7 0 0
01:44:38 am 238 9 7 0 0
01:44:40 am 238 9 7 0 0
Totsck: Used sockets total Quantity
Tcpsck: Number of TCP sockets used
Udpsck: Number of UDP sockets used
Rawsck: Number of raw sockets used
Ip-frag: Number of IP segments used
If you use the full keyword, Equivalent to the above Dev, edev and sock synthesis of the three-person