Linux sar command for performance monitoring

 

We have already introduced the parsing and use of vmstat and top commands. Next we will learn a more important command SAR.
The SAR command can view the usage of a specific part of the system through parameters.

Common formats of SAR command lines:

SAR [Options] [-A] [-o file] T [N]

In the command line, the N and T parameters are combined to define the sampling interval and number of times. t indicates the sampling interval and is a required parameter. N indicates the number of samples and is optional, the default value is 1.-o file indicates that the command results are saved in binary format,

File is not a keyword here, but a file name. Options is the command line option, and there are many options for SAR commands. below, only common options are listed:

-A: total of all reports.
-U: CPU usage
-V: process, I node, file, and lock table status.
-D: hard disk usage report.
-R: Memory pages and hard disk blocks that are not in use.
-G: serial port I/O.
-B: Buffer usage.
-A: file read/write status.
-C: System Call status.
-R: Process activity.
-Y: terminal device activity.
-W: system exchange activity.
-N: records network usage

Default Monitoring: SAR 5 5 // CPU and iowait statistical status
(1) SAR-B 5 5 // Io Transfer Rate
(2) SAR-B 5 5 // page switching Rate
(3) SAR-C 5 5 // process creation rate
(4) SAR-D 5 5 // active information of Block devices
(5) SAR-N Dev 5 5 // status information of network devices
(6) SAR-N sock 5 // sock usage
(7) SAR-n all 5 5 // all network status information
(8) SAR-P all 5 5 // The Usage Status and iowait statistical status of each CPU
(9) SAR-Q 5 5 // queue length (number of processes awaiting running) and Load Status
(10) SAR-R 5 5 // memory and swap space usage
(11) SAR-R 5 5 // memory statistics (Memory Page allocation and release, Memory Page used by the system as buffer per second, Memory Page cached per second)
(12) SAR-U 5 5 // CPU usage and iowait information (same as default monitoring)
(13) SAR-V 5 5 // inode, file and other kernel tablesd status information
(14) SAR-W 5 5 // Number of context switches per second
(15) SAR-W 5 5 // statistical information of swap switching (monitoring status is the same as Si so of iostat)
(16) SAR-x 2906 5 5 // display the statistical information of a specified process (2906), including: errors Caused by processes, user-level and system-level user CPU usage, and on which CPU is running
(17) Activity Status of the SAR-y 5 5 // tty device
(18) output to file (-O) and read record information (-f)

Example 1:Oracle @ Oracle[/Home/Oracle] Sar-u 1 0-e 16:00:00> data.txt // records the usage of CPU every 1second. Data is saved to the data.txt file. (The-e parameter indicates the end time. Note that the time format must be hh: mm: Ss)

Linux 2.6.18-194. EL5 (Oracle)

At 02:20:28 CPU % USER % nice % System % iowait % steal % idle
02:20:29 all 0.00 0.00 0.00 0.00 0.00 100.00
02:20:30 all 0.00 0.00 0.00 0.00 0.00 100.00
02:20:31 all 0.00 0.00 0.00 0.00 0.00 100.00
02:20:32 all 0.00 0.00 0.00 0.00 0.00 100.00
02:20:33 all 0.00 0.00 0.00 0.00 0.00 100.00
02:20:34 all 0.00 0.00 0.25 0.00 0.00 99.75
02:20:35 all 0.00 0.00 0.00 0.00 0.00 100.00
02:20:36 all 0.00 0.00 0.00 0.00 0.00 100.00
02:20:37 all 0.25 0.00 0.00 0.00 0.00 99.75
02:20:38 all 0.00 0.00 0.00 0.00 0.00 100.00
Average: All 0.02 0.00 0.02 0.00 0.00

When no parameter is used, the system prints the CPU usage by default.

% USER: Percentage of CPU time in user mode.
% SYSTEM: Percentage of CPU time in system mode.
% Iowait: Percentage of CPU waiting for input/output completion time.
% Idle: Percentage of idle CPU time.
In all the displays, we should pay attention to % iowait and % idle. The value of % iowait is too high, indicating that the hard disk has an I/O bottleneck, and the value of % idle is high, indicating that the CPU is idle, if the % idle value is high but the system response is slow, it may be that the CPU is waiting for memory allocation. In this case, the memory capacity should be increased. If the value of % idle is lower than 10, the CPU processing capability of the system is relatively low, indicating that the most important resource to be solved in the system is the CPU.

In a multi-CPU Linux system, the sar command can also break down this information for each CPU. The following command is used: Sar-u-p all 5

Oracle @ Oracle [/home/Oracle] Sar-u-p all 5 2
Linux 2.6.18-194. EL5 (Oracle)

At 02:41:20 CPU % USER % nice % System % iowait % steal % idle
02:41:25 all 0.00 0.00 0.10 0.00 0.00 99.90
02:41:25 PM 0 0.00 0.00 0.20 0.00 0.00
02:41:25 pm 1 0.00 0.00 0.00 0.00 0.00
02:41:25 PM 2 0.20 0.00 0.20 0.00 0.00
02:41:25 PM 3 0.00 0.00 0.00 0.00 0.00

At 02:41:25 CPU % USER % nice % System % iowait % steal % idle
02:41:30 all 0.00 0.00 0.05 0.00 0.00 99.95
02:41:30 PM 0 0.00 0.00 0.00 0.00 0.00
02:41:30 pm 1 0.00 0.00 0.00 0.00 0.00
02:41:30 PM 2 0.00 0.00 0.00 0.00 0.00
02:41:30 PM 3 0.00 0.00 0.00 0.00 0.00

Example 2: run the SAR-v t n command.
Oracle @ Oracle [/home/Oracle] Sar-V 30 5 // process, I node, file, and lock Table Status
Linux 2.6.18-194. EL5 (Oracle)

02:28:45 dentunusd file-SZ inode-SZ super-SZ % super-SZ dquot-SZ % dquot-SZ rtsig-SZ % rtsig-SZ
02:29:15 PM 8675 9690 7119 0 0.00 0 0.00 0 0.00
02:29:45 PM 8676 9690 7119 0 0.00 0 0.00 0 0.00
02:30:15 PM 8677 9690 7119 0 0.00 0 0.00 0 0.00
02:30:45 PM 8684 9690 7126 0 0.00 0 0.00 0 0.00
02:31:15 PM 8685 9690 7126 0 0.00 0 0.00 0 0.00
Average: 8679 9690 7122 0 0.00 0 0.00 0 0.00

Dentunusd: number of entries not used in the buffer directory.
File-Nr: number of file handles used by the system.
Inode-Nr: Number of index nodes used.
Pty-Nr: Number of Pty used.

Example 3: run the SAR-d t n command.

Oracle @ Oracle [/home/Oracle] Sar-d 30 2 // view device usage
Linux 2.6.18-194. EL5 (Oracle)

02:30:33 Dev TPS rd_sec/s wr_sec/s avgrq-SZ avgqu-SZ await svctm % util
02:31:03 dev8-0 47.95 12099.97 58.38 253.56 0.08 1.74 1.03 4.95
02:31:03 dev8-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:31:03 dev8-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:31:03 dev8-3 47.95 12099.97 58.38 253.56 0.08 1.74 1.03 4.95

02:31:03 Dev TPS rd_sec/s wr_sec/s avgrq-SZ avgqu-SZ await svctm % util
02:31:33 dev8-0 38.67 9648.00 62.93 251.14 0.07 1.75 1.01 3.89
02:31:33 dev8-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:31:33 dev8-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:31:33 dev8-3 38.67 9648.00 62.93 251.14 0.07 1.75 1.01 3.89

Dev disk device
With the parameter-P can print the SDA, HDC and other disk device names, if you do not have the parameter-P, the device node may be a dev8-0, dev22-0
TPS: the number of I/O requests from the physical disk per second. Multiple logical requests are merged into one I/O disk request. The size of one transmission is uncertain.
Rd_sec/S: Number of read sectors per second.
Wr_sec/S: Number of write sectors per second.
Avgrq-SZ: average data size (slice) of each device I/O operation ).
Avgqu-SZ: average length of the disk Request queue.
Await: Average time consumed by each request, from the request disk operation to the system processing, including the request queue waiting time, in milliseconds (1 second = 1000 milliseconds ).
Svctm: average time for the system to process each request, excluding the time consumed in the Request queue.
% Util: Percentage of I/O requests to the CPU. The higher the ratio, the more full it is.

Example 4: Use the command Sar-N T N

Oracle @ Oracle [/home/Oracle] Sar-n all 5 2
Linux 2.6.18-194. EL5 (Oracle)

02:52:49 iface rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
02:52:54 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:54 eth0 127.15 112.57 38894.41 20819.16 0.00 0.00 0.00
02:52:54 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:54 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:52:49 iface rxerr/s txerr/s Coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
02:52:54 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:54 eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:54 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:54 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:52:49 call/s retrans/s Read/s write/s access/s getatt/s
02:52:54 0.00 0.00 0.00 0.00 0.00

02:52:49 scall/s badcall/s packet/s UDP/s TCP/s hit/s Miss/s sread/s swrite/s saccess/s sgetatt/s
02:52:54 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:52:49 totsck tcpsck udpsck rawsck IP-Frag
02:52:54 PM 274 99 41 0 0

02:52:54 iface rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
02:52:59 lo 0.40 0.40 20.00 20.00 0.00 0.00 0.00
02:52:59 eth0 139.00 120.60 47988.20 22587.00 0.00 0.00 0.00
02:52:59 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:59 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:52:54 iface rxerr/s txerr/s Coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
02:52:59 lo 0.00 0.00 0.00 0.00 0.00 0.0 0 0.00 0.00 0.00
02:52:59 eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:59 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:52:59 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:52:54 call/s retrans/s Read/s write/s access/s getatt/s
02:52:59 0.00 0.00 0.00 0.00 0.00

02:52:54 scall/s badcall/s packet/s UDP/s TCP/s hit/s Miss/s sread/s swrite/s saccess/s sgetatt/s
02:52:59 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

02:52:54 totsck tcpsck udpsck rawsck IP-Frag
02:52:59 PM 274 99 41 0 0

Average: iface rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
Average: Lo 0.20 0.20 9.99 9.99 0.00 0.00
Average: eth0 133.07 116.58 43436.76 21702.20 0.00 0.00
Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00
Average: sit0 0.00 0.00 0.00 0.00 0.00 0.00

Average: iface rxerr/s txerr/s Coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
Average: Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: Call/s retrans/s Read/s write/s access/s getatt/s
Average: 0.00 0.00 0.00 0.00 0.00

Average: scall/s badcall/s packet/s UDP/s TCP/s hit/s Miss/s sread/s swrite/s saccess/s sgetatt/s
Average: 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: totsck tcpsck udpsck rawsck IP-Frag
Average: 274 99 41 0 0

SAR provides four different syntax options to display network information. The-N option uses four different switches: Dev, EDEV, sock, and full. Dev displays network interface information, EDEV displays statistics about network errors, and sock displays socket information,

Full display all three switches. They can be used independently or together.

Field description
Iface LAN interface
Rxpck/s packets received per second
Txpck/s packets sent per second
Rxbyt/s the number of bytes received per second
Txbyt/s number of bytes sent per second
Rxcmp/s compressed data packets received per second
Txcmp/s compressed data packets sent per second
Multicast packet received by rxmcst/s per second
 
The subsequent commands are explained only for each command, without testing.

Example 5: Sar-R 5 5 // memory and swap space usage
Kbmemfree: this value is basically the same as the free value in the free command, so it does not include the buffer and cache space.
Kbmemused: this value is basically the same as the used value in the free command, so it includes the buffer and cache space.
% Memused: this value is a percentage of kbmemused and total memory (excluding swap.
Kbbuffers and kbcached: These two values are buffer and cache in the free command.
Kbcommit: Ensure the memory required by the current system, that is, the memory required to ensure no overflow (RAM + swap ).
% Commit: this value is a percentage of kbcommit and total memory (including swap.

Example 6: Sar-B 5 5 // page switching Rate
Pgpgin/s: the number of bytes per second from disk or swap to memory (KB)
Pgpgout/s: the number of bytes per second from memory replacement to disk or swap (KB)
Fault/S: number of missing pages generated by the system per second, that is, the sum of the Primary and Secondary missing pages (Major + minor)
Majflt/S: Number of Primary missing pages per second.
Pgfree/S: Number of pages per second in the idle queue
Pgscank/S: Number of pages scanned by kswapd per second
Pgscand/S: number of directly scanned pages per second
Pgsteal/s: the number of pages that are cleared from the cache every second to meet memory requirements
% Vmeff: Percentage of pages cleared per second (pgsteal) to total scanning pages (pgscank + pgscand)

Example 7: Sar-Q 5 5 // queue length (number of processes awaiting running) and Load Status
Runq-SZ: Number of running or ready Processes
Plist-SZ: Total number of processes (including threads ).
Ldavg-1: load from the last minute.
Ldavg-5: load for the last five minutes.
Ldavg-15: load for the last 10 minutes.
Data of average load and queue comes from/proc/loadavg

Example 8: Sar-n nfs 5 5 // NFS client monitoring
Call/S: Successful RPC calls per second increase the call/s value, for example, one read/write operation on NFS.
Retrans/s: the number of RPC retransmissions per second. For example, a timeout occurs due to server issues. In this case, the client needs to re-transmit the data.
Read/s: The number of times read from the NFS server per second.
Write/s: the number of writes to the NFS server per second.
Access/s: The number of times NFS is accessed per second, for example, copying files from the NFS server.
Getatt/s: the number of times that NFS Server File Attributes are obtained per second, such as LS-L/nfsserver/. If nfsserver has 300 files, 300 such requests are generated.

Example 9: Sar-B 5 5 // Io Transfer Rate
TPS: the number of I/O requests from the physical disk per second. Multiple logical requests are merged into one I/O disk request. The size of one transmission is uncertain.
Rtps: Number of read requests per second
Wtps: Number of write requests per second
Bread/S: number of data blocks read from the disk per second (in blocks 1 block = 512b, kernel after 2.4)
Bwrtn/S: number of data blocks written to the disk per second (in blocks 1 block = 512b, kernel after 2.4)
TPS = (rtps + wtps)

SAR can also monitor non-real-time data and run it in the specified directory through cron cycles.
For example, we want to view the memory resources from to on January 1, 27th day of this month.
Sa27 is July 22, 27th day of this month. The specific time can be specified through-s (start) and-E (end.
Sar-F/var/log/SA/sa27-s 00:00:00-e 23:00:00-R