Iostat for Linux monitoring tools

Analysis of iostat results

[Root @ 20081006-1724 ~] # Iostat-X-d

Linux 2.6.9-78. elsmp (20081006-1724) 11/20/2009 AVG-CPU: % USER % nice % sys % iowait % idle
0.19 0.00 0.04 0.03 99.73 device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s RKb/s WKB/s avgrq-SZ avgqu-SZ await svctm % util
SDA 0.05 17.60 1.46 7.72 80.69 202.57 40.34 101.29 30.87 0.01 1.06 0.37 0.34
Sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 29.90 0.00 3.14 3.14 0.00
Sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16.25 0.00 1.51 1.30 0.00
Sda3 0.05 17.60 1.46 7.72 80.69 202.57 40.34 101.29 30.87 0.01 1.06 0.37 0.34
Dm-0 0.00 0.00 1.46 25.28 80.32 202.26 40.16 101.13 10.57 0.36 13.56 0.13 0.34
Dm-1 0.00 0.00 0.05 0.04 0.37 0.32 0.18 0.16 8.00 0.00 6.84 1.30 rrqm/S: Number of merge read operations per second. That is, Delta (rmerge)/s
Wrqm/S: Number of write operations performed on merge per second. That is, Delta (wmerge)/s
R/S: The number of read I/O devices per second. That is, Delta (Rio)/s
W/s: the number of write I/O devices completed per second. That is, Delta (WIO)/s
Rsec/S: Number of read sectors per second. That is, Delta (rsect)/s
Wsec/S: Number of write sectors per second. That is, Delta (wsect)/s
RKb/s: the number of bytes read per second. It is half of rsect/s because the size of each slice is 512 bytes.
WKB/s: the number of K bytes written per second. Half of wsect/s.
Avgrq-SZ: average data size (slice) of each device I/O operation ). That is, Delta (rsect + wsect)/DELTA (Rio + WIO)
Avgqu-SZ: Average I/O queue length. That is, Delta (aveq)/S/1000 (because aveq is measured in milliseconds ).
Await: average wait time (in milliseconds) for each device I/O operation ). That is, Delta (ruse + wuse)/DELTA (Rio + WIO)
Svctm: Average service time (in milliseconds) for each device I/O operation ). That is, Delta (use)/DELTA (Rio + WIO)
% Util: the percentage of time in one second is used for I/O operations, or the number of I/O queues in one second is not empty.
That is, Delta (use)/S/1000 (because the Unit of use is milliseconds)

If % util is close to 100%, it indicates that too many I/O requests are generated and the I/O system is fully loaded. This disk may have a bottleneck.

Important Parameters
% Util: The amount of time in one second for I/O operations, or the amount of time in one second for I/O queues is not empty.
Svctm: Average service time for each device I/O operation
Await: average wait time for each device I/O operation
Avgqu-SZ: Average I/O Queue Length

If % util is close to 100%, it indicates that there are too many I/O requests, the I/O system is fully loaded, and the disk may have a bottleneck. Generally, if % util is greater than 70%, the I/O pressure is relatively high, wait has a high read speed. you can also view the parameters B (number of processes waiting for resources) and Wa in combination with vmstat (percentage of CPU time occupied by Io wait, which is higher than 30% when I/O pressure is high ).
The size of await generally depends on the service time (svctm), the length of the I/O queue, and the mode in which I/O requests are sent. If svctm is close to await, it means that I/O has almost no waiting time. If await is much larger than svctm, it means that the I/O queue is too long and the response time of the application is slow.

Image metaphor
R/S + w/s is similar to the total number of payers
The average queue length (avgqu-sz) is similar to the number of average queues per unit time.
The average service time (svctm) is similar to the cashier's collection speed
The average wait time (await) is similar to the average wait time of each person
Average I/O data (avgrq-sz) is similar to the average number of items bought by each person
I/O operation rate (% util) is similar to the time ratio of someone waiting in queue before the cashier

Device Io operations: Total IO/S = R/S (read) + w/s (write) = 1.46 + 25.28 = 26.74
On average, each device I/O operation only takes 0.36 milliseconds, but now it takes 10.57 milliseconds to complete, because too many requests are sent (26.74 requests per second). If the request is sent at the same time, the average wait time can be calculated as follows:
Average wait time = time of a single I/O Server * (1 + 2 +... + total number of requests-1)/total number of requests
Many I/0 requests are sent per second, but the average queue is 4, indicating that these requests are evenly distributed and most of the requests are processed in a timely manner.
Generally, svctm is smaller than await (because the wait time of the simultaneously waiting request is calculated repeatedly ),
The size of svctm is generally related to disk performance, and the CPU/memory load will also affect it, with too many requests
It also indirectly leads to the increase of svctm. The size of await generally depends on the service time (svctm), the length of the I/O queue, and the mode in which I/O requests are sent. If svctm is close to await, it means that I/O has almost no waiting time. If await is much larger than svctm, it means that the I/O queue is too long and the response time of the application is slow, if the response time exceeds the allowable range, you can consider replacing a faster disk, adjusting the kernel elevator algorithm, optimizing the application, or upgrading the CPU.

The queue length (avgqu-sz) can also be used as an indicator to measure the system I/O load. However, because avgqu-SZ is based on the average per unit time, therefore, it cannot reflect the instantaneous I/O flood.