Linux wa% too high, iostat view IO status

You will see this line when you use the top command:

Translation: US: User state uses more CPU time than SY: The system state uses more CPU time than NI: The user-state CPU time compared to a nice-weighted process is allocated: Idle CPU time is longer than wa:cpu waiting for disk write completion hi: Hard Interrupt Time si: Soft interrupt consumption time ST: virtual machine steal time if A machine to see WA is particularly high, then the general description is disk IO problems, you can use Iostat and other commands to continue the detailed analysis.

1, install Iostat

Yum Install Sysstat

You can then use the Iostat command,

2, Getting Started with

Iostat-d-K 2

The parameter-D indicates that the device (disk) usage status is displayed; k Some columns that use block are forced to use kilobytes, and 2 indicates that the data appears refreshed every 2 seconds.

TPS: The number of transmissions per second of the device (indicate, transfers per second, were issued to the.). "One-time transfer" means "one-time I/O request". Multiple logical requests may be merged into "one I/O request". The size of the "one transfer" request is unknown. KB_READ/S: The amount of data read from the device (drive expressed) per second;
KB_WRTN/S: The amount of data written to the device (drive expressed) per second;
Kb_read: The total amount of data read, KB_WRTN: The amount of total data written, all of which are kilobytes.

Specifies that the monitored device name is SDA, and that the output of the command is exactly the same as the command above.

Iostat-d SDA 2

By default, all hard disk devices are monitored and are now designated to monitor SDA only.

3,-x parameter

Iostat also has a more common option- x, which will be used to display and IO-related extended data.

iostat-d-x-k 1 10

Meaning of the output information

RRQM/S: How much of this device-dependent read request is being merge per second (when the system call needs to read the data, the VFS sends the request to each FS, if FS finds that different read requests read the same block of data,
FS merges this request into merge);
WRQM/S: How much of this device-related write request is being merge per second. RSEC/S: Number of sectors read per second; wsec/: Number of sectors written per second. Rkb/s:the number of read requests that were issued to the device per second;wkb/s:the number of write requests that were I Ssued to the device per Second;avgrq-sz the average request sector size Avgqu-sz is the length of the average request queue.    There is no doubt that the shorter the queue, the better. Await: The average time (in milliseconds) of processing per IO request.         This can be understood as the response time of IO, generally the system IO response time should be less than 5ms, if greater than 10ms is relatively large. This time includes the queue time and service time, that is, in general, await is greater than SVCTM, their difference is smaller, then the shorter the queue time, conversely, the greater the difference, the longer the queue time, indicating that the system has a problem. SVCTM represents the average service time (in milliseconds) for each device I/O operation. If the value of SVCTM is close to await, indicating that there is little I/O waiting, disk performance is good, if the value of await is much higher than the value of SVCTM,
Indicates that the I/O queue waits too long,
Applications running on the system will become slower.
 %util: All processing io time, divided by total statistic time, in the statistical time. For example, if the statistic interval is 1 seconds, the device has 0.8 seconds in processing Io,
and 0.2 seconds is idle, then the device's%util = 0.8/1 = 80%,
So this parameter implies how busy the device is. 
。 Generally, if the parameter is 100%, the device is already running close to full load
(Of course, if it is a multi-disk, even if the%util is 100%, because of the concurrency of the disk, so disk use may not be the bottleneck) 。

4, common usage

Iostat-d-K 1         #查看TPS和吞吐量信息 (disk read and write speed in KB) iostat-d-M 2            #查看TPS和吞吐量信息 (disk read and write speed in MB) iostat-d-x-k 1      #查看设备使 Use Rate (%util), Response time (await) Iostat-c 1 #查看cpu状态

5, Case analysis

Iostat-d-K 1 | grep vdadevice:            TPs    kb_read/s    kb_wrtn/s    kb_read    kb_wrtnsda10            60.72        18.95        71.53  395637647 1493241908sda10           299.02      4266.67       129.41       4352        132sda10           483.84      4589.90      4117.17       4544       4076sda10           218.00      3360.00       100.00       3360        100sda10           546.00      8784.00       124.00       8784        124sda10           827.00     13232.00       136.00      13232        136

As seen above, the average number of disk transfers per second is about 400, and the disk reads about 5MB per second and writes about 1MB.

iostat-d-x-k 1Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkb/s wkb/s    Avgrq-sz Avgqu-sz   await  svctm  %utilsda          1.56  28.31  7.84 31.50   43.65    3.16    21.82     1.58     1.19     0.03    0.80   2.61  10.29SDA          1.98  24.75 419.80  6.93 13465.35  253.47  6732.67   126.73    32.15     2.00    4.70   2.00  85.25sda          3.06  41.84 444.90 54.08 14204.08 2048.98  7102.04  1024.49    32.57     2.10    4.21   1.85  92.24

You can see the average response time of the disk <5ms, and the disk usage is >80. The disk responds properly, but it is already busy.

You can see the average response time of the disk <5ms, and the disk usage is >90. The disk responds properly, but it is already busy.

Await: The  average time (in milliseconds) of processing per IO request. This can be understood as the response time of IO, generally the system IO response time should be less than 5ms, if greater than 10ms is relatively large

SVCTM    represents the average service time (in milliseconds) for each device I/O operation. If the value of SVCTM is close to await, it means that there is almost no I/O waiting, the disk is performing well,
If the value of the await is much higher than the value of SVCTM, the I/O queue waits too long for  the applications running on the system to become slower.

%util: All processing IO time in statistical time, divided by total statistic time

So this parameter implies how busy the device is.
。 Generally, if this parameter is 100% indicates that the device is already running close to full load (of course if it is a multi-disk, even if%util is 100% because of the concurrency of the disk, disk usage may not be the bottleneck).


You can also use the following command to display both CPU and disk usage

Linux wa% too high, iostat view IO status