System performance monitoring Tools-SAR

Yum Installation: Yum install Sysstat (SAR is the front-end display tool for background process SADC, after installing a package called "Sysstat", SADC automatically collects reports from the kernel and saves them)

SAR (System activity Reporter report) is one of the most comprehensive system performance analysis tools available on Linux, which can be used to report the activities of the systems in many ways, including: Read and write of files, usage of system calls, disk I/O, CPU efficiency, Memory usage, process activity, and IPC-related activities.
Common formats:
SAR [Options] [-A] [-o file] t [n]
which
T is the sampling interval, n is the number of samples, the default value is 1;
-o file means that the command result is stored in a binary format in a file, and file is a filename.
Options is a command-line option, and the SAR command is commonly used as follows:

-A: Sum of all reports
-U: Output statistics about CPU usage
-V: Output statistics for inode, files, and other kernel tables
-D: Output activity information for each block device
-r: Output memory and swap space statistics
-B: Display statistics for I/O and transfer rates
-A: File read and write status
-C: Output process statistics, number of processes created per second
-r: Output memory page statistics
-Y: terminal equipment activity situation
-W: Output system Exchange activity information

1) CPU Resource monitoring
Sample every 10 seconds, 3 consecutive samples, observe the CPU usage, and the sample results in binary form into the file test in the current directory, you need to type the following command:
Sar-u-O Test 10 3

17:06:16 CPU%user%nice%system%iowait%steal%idle
17:06:26 All 0.00 0.00 0.20 0.00 0.00 99.80
17:06:36 All 0.00 0.00 0.20 0.00 0.00 99.80
17:06:46 All 0.00 0.00 0.10 0.00 0.00 99.90
Average:all 0.00 0.00 0.17 0.00 0.00 99.83

Cpu:all represents the average of the statistics for all CPUs.
%user: Displays the percentage of total CPU time that is running at the user level (application).
%nice: The percentage of total CPU time that is displayed at the user level for the nice operation.
%system: The percentage of total CPU time that is used to run at the core level (kernel).
%iowait: Displays the percentage of total CPU time that is used to wait for I/O operations.
%steal: The percentage of the hypervisor (hypervisor) that waits for a virtual CPU to serve another virtual process.
%idle: Shows the percentage of CPU idle time that takes up the total CPU time.

A, if the value of%iowait is too high, indicates that the hard disk has I/O bottleneck
b, if the value of the%idle is high but the system response is slow, it is possible that the CPU waits to allocate memory, should increase the memory capacity at this time
C, if the value of%idle is consistently less than 1, the system's CPU processing power is relatively low, indicating that the most necessary resources in the system is the CPU.
D. If you want to view the contents of a binary file test, type the following SAR command: sar-u-F Test

2) inode, file, and other kernel table monitoring
Sample every 10 seconds, 3 consecutive samples, observe the status of the core table, you need to type the following command:
Sar-v 10 3

17:10:49 DENTUNUSD File-nr Inode-nr pty-nr
17:10:59 6301 5664 12037 4
17:11:09 6301 5664 12037 4
17:11:19 6301 5664 12037 4
average:6301 5664 12037 4

DENTUNUSD: Number of unused entries in the directory cache
FILE-NR: The number of file handles (files handle) used
INODE-NR: Number of uses of the index node handle (inode handle)
PTY-NR: Number of Pty used

3) memory and swap space monitoring
Sample every 10 seconds, 3 consecutive samples, monitoring memory paging:
Sar-r 10 3

Kbmemfree: This value is basically the same as the free value in the command, so it does not include buffer and cache space.
Kbmemused: This value is basically the same as the used value in the free command, so it includes buffer and cache space.
%memused: This value is a percentage of kbmemused and total memory (excluding swap).
Kbbuffers and kbcached: These two values are the buffer and cache in the free command.
Kbcommit: Ensure that the current system requires memory (RAM+SWAP) in order to ensure that it does not overflow.
%commit: This value is a percentage of kbcommit and total memory (including swap).

4) Memory Paging monitoring
Sample every 10 seconds, 3 consecutive samples, monitoring memory paging:
Sar-b 10 3

PGPGIN/S: Indicates the number of bytes per second from disk or swap to memory (KB)
PGPGOUT/S: Indicates the number of bytes per second from memory to disk or swap (KB)
FAULT/S: The number of pages per second that the system generates, that is, the sum of the primary and secondary page faults (major + minor)
MAJFLT/S: The number of main pages generated per second.
PGFREE/S: Number of pages per second that are placed in the free queue
PGSCANK/S: Number of pages scanned per second by KSWAPD
PGSCAND/S: Number of pages scanned directly per second
PGSTEAL/S: The number of pages per second that are purged from the cache to meet memory requirements
%vmeff: Percentage of pages purged per second (pgsteal) in total scan page (Pgscank+pgscand)

5) I/O and transfer rate monitoring
Sample every 10 seconds, 3 consecutive samples, report the use of the buffer, you need to type the following command:
Sar-b 10 3

18:51:05 TPs Rtps Wtps bread/s bwrtn/s
18:51:15 0.00 0.00 0.00 0.00 0.00
18:51:25 1.92 0.00 1.92 0.00 22.65
18:51:35 0.00 0.00 0.00 0.00 0.00
average:0.64 0.00 0.64 0.00 7.59

TPS: Total I/O transfers per second for physical devices
Rtps: Total amount of data read from physical devices per second
Wtps: The total amount of data written to the physical device per second
BREAD/S: The amount of data read from the physical device per second, in blocks/s
BWRTN/S: The amount of data written to the physical device per second, in blocks/s

6) Process Queue length and average load status monitoring
Sample every 10 seconds, sample 3 times, monitor process queue length and average load status:
Sar-q 10 3

19:25:50 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
19:26:00 0 259 0.00 0.00 0.00
19:26:10 0 259 0.00 0.00 0.00
19:26:20 0 259 0.00 0.00 0.00
Average:0 259 0.00 0.00 0.00

Runq-sz: Length of the run queue (number of processes waiting to run)
Plist-sz: Number of processes (processes) and threads (threads) in the process list
Ldavg-1: Average system load for last 1 minutes (systems load average)
Ldavg-5: System average load over the last 5 minutes
Ldavg-15: System average load over the last 15 minutes

7) System Exchange activity information monitoring
Sampling every 10 seconds, sampling 3 times, monitoring system exchange activity information:
Sar-w 10 3

19:39:50 PSWPIN/S pswpout/s
19:40:00 0.00 0.00
19:40:10 0.00 0.00
19:40:20 0.00 0.00
average:0.00 0.00

PSWPIN/S: Number of swap pages per second system swapped in (Swap page)
PSWPOUT/S: Number of swap pages per second system swapped out (Swap page)

8) Monitoring of equipment usage
Sample every 10 seconds, 3 consecutive samples, report the use of the device, you need to type the following command:
Sar-d 3-p

17:45:54 DEV TPs rd_sec/s wr_sec/s avgrq-sz avgqu-sz await SVCTM%util
17:46:04 scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 SDA 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 vg_livedvd-lv_root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 Vg_livedvd-lv_swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Parameter-P can print out the name of the disk device such as SDA,HDC, if not the parameter-p, the device node may be dev8-0,dev22-0
TPS: Number of times per second from physical disk I/O. Multiple logical requests are merged into one I/O disk request, and the size of one transfer is indeterminate.
RD_SEC/S: Number of Read sectors per second.
WR_SEC/S: Number of Write sectors per second.
Avgrq-sz: The average data size (sector) per device I/O operation.
Avgqu-sz: The average length of the disk request queue, when the value of Avgqu-sz is low, the device utilization is higher
Await: The average elapsed time of each request, including the request queue wait time, in milliseconds (1 seconds =1000 milliseconds), from the request disk operation to the completion of the system processing.
SVCTM: The average time that the system processes each request, excluding the time consumed in the request queue.
%UTIL:I/O requests account for the percentage of CPU, the higher the ratio, the more saturated the description, when the value of%util is close to 100%, indicating that the device bandwidth is already full.

9) to determine the system bottleneck , sometimes it takes several SAR command options to combine
Suspected CPU bottlenecks, sar-u and sar-q can be used to view
Suspect memory bottlenecks and can be viewed with Sar-b, sar-r, sar-w, etc.
Suspected I/O bottlenecks and can be viewed with Sar-b, sar-u, sar-d, etc.

10) Analyze Network card traffic
sar-n {DEV | Edev | NFS | NFSD | SOCK | All}
The SAR provides six different syntax options to display network information. The-N option uses 6 different switches: DEV | Edev | NFS | NFSD | SOCK | All. The dev Displays the network interface information, Edev displays statistics about network errors, NFS Statistics active NFS Client information, NFSD statistics Server for NFS, sock displays socket information, all shows all 5 switches. They can be used alone or together.

A) Sar-n DEV 2 10
Linux 2.6.18-53.el5pae (Localhost.localdomain) 03/29/2009

01:39:40 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
01:39:42 AM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:39:42 AM eth1 131.34 104.98 119704.48 36110.45 0.00 0.00 0.00
01:39:42 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

01:39:42 AM IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
01:39:44 AM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:39:44 AM eth1 168.00 165.50 114496.50 83938.50 0.00 0.00 0.00
01:39:44 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Iface:lan interface
RXPCK/S: Packets Received per second
TXPCK/S: Packets Sent per second
rxbyt/s: Number of bytes received per second
txbyt/s: Number of bytes sent per second
RXCMP/S: Compressed packets received per second
TXCMP/S: Compressed packets sent per second
RXMCST/S: Multicast packets received per second

b) Sar-n Edev 2 10
Linux 2.6.18-53.el5pae (Localhost.localdomain) 03/29/2009

01:42:18 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
01:42:20 AM Lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:42:20 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:42:20 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Iface:lan interface
RXERR/S: Bad packets received per second
TXERR/S: Bad packets sent per second
COLL/S: Number of collisions per second
RXDROP/S: The number of received packets dropped per second because the buffer is full
TXDROP/S: The number of sent packets dropped per second because the buffer is full
TXCARR/S: The number of carrier errors per second when sending packets
RXFRAM/S: Number of frame alignment errors received per second for packets
RXFIFO/S: Number of errors received per second FIFO over-speed
TXFIFO/S: Number of errors sent packets per second FIFO over-speed

c) Sar-n SOCK 2 10
Linux 2.6.18-53.el5pae (Localhost.localdomain) 03/29/2009

01:44:32 AM totsck tcpsck udpsck rawsck Ip-frag
01:44:34 AM 243 9 8 0 0
01:44:36 AM 242 9 7 0 0
01:44:38 AM 238 9 7 0 0
01:44:40 AM 238 9 7 0 0

TOTSCK: Total number of sockets used
TCPSCK: Number of TCP sockets used
UDPSCK: Number of UDP sockets used
RAWSCK: Number of raw sockets used
Ip-frag: Number of IP segments used

System performance monitoring Tools-SAR