Disk I/O is a high-resolution approach

Introduced

The sudden high disk IO is a problem frequently encountered by operations personnel, due to the large number of disk reads and writes, usually occurring in the database, but the various scenarios that occur. This article gives a few examples to illustrate the solution.

Body

Find out what the program is doing in a large number of read and write operations. You can view the history of disk IO through monitoring software (such as Zabbix) or tools such as atop.

This article assumes that the scenario occurs on a Xen virtual machine and uses Iostat to view the IO status

# IOSTAT-XDK 2

The output is as follows:

device:rrqm/s wrqm/s r/s w/s rkb/s wkb/s avgrq-sz avgqu-sz await r_await w_await SVCTM%util

SDA 0.00 24.50 0.00 13.50 0.00 141.00 20.89 1.71 126.67 0.00 126.67 39.70 53.60

SDB 0.00 1.50 0.00 12.00 0.00 132.00 22.00 1.42 108.33 0.00 108.33 83.33 100.00

Dm-0 0.00 0.00 0.00 12.50 0.00 12.50 2.00 2.36 188.48 0.00 188.48 11.84 14.80

Dm-1 0.00 0.00 0.00 6.00 0.00 24.00 8.00 0.76 126.00 0.00 126.00 33.33 20.00

Dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Dm-3 0.00 0.00 0.00 6.50 0.00 26.00 8.00 1.18 181.54 0.00 181.54 22.46 14.60

Dm-4 0.00 0.00 0.00 3.00 0.00 12.00 8.00 0.40 133.33 0.00 133.33 66.67 20.00

Dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Dm-7 0.00 0.00 0.00 3.50 0.00 23.50 13.43 0.21 60.57 0.00 60.57 33.71 11.80

dm-8               0.00     0.00    0.00     0.00     0.00     0.00      0.00     0.00    0.00    0.00     0.00   0.00   0.00

Dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

DM-10 0.00 0.00 0.00 4.00 0.00 28.00 14.00 0.47 117.50 0.00 117.50 78.50 31.40

dm-11              0.00     0.00    0.00     0.00     0.00     0.00      0.00     0.00    0.00    0.00     0.00   0.00   0.00

dm-12              0.00     0.00    0.00     0.00     0.00     0.00      0.00     0.00    0.00    0.00     0.00   0.00   0.00

DM-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

DM-14 0.00 0.00 0.00 2.50 0.00 15.00 12.00 0.32 126.40 0.00 126.40 94.40 23.60

Dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

DM-16 0.00 0.00 0.00 15.00 0.00 134.00 17.87 1.56 93.60 0.00 93.60 66.67 100.00

dm-17              0.00     0.00    0.00     0.00     0.00     0.00      0.00     0.00    0.00    0.00     0.00   0.00   0.00

DM-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

You can see that SDB and dm-16 use 100% of%util, and the following need to find out which logical volume uses 100% IO:

# Dmsetup LS

The output is as follows:

....... DOMOVG-SERVERNAME--DB_HDC (253, 16) .........

Once the corresponding logical volume is found, log on to the corresponding machine and use Dstat or Iostat to find out which process uses a lot of IO:

# Dstat--top-bio-adv

If the problem with the data is caused, it is usually caused by the following actions:

1 SCP,TAR,GZIP,CP

2 Backup

3 have a lot of slow queries

4 full-Scan SQL statements

There may be a problem with the battery (BBU) of the raid card, which can be used to view the raid status (e.g. MEGACLI) according to the corresponding server model.

This article from the "Linux and Network" blog, reproduced please contact the author!

Disk I/O is a high-resolution approach