How Linux tests IO performance (disk read/write speed)

These days to do the MySQL performance test, a huge company, find a few performance testing machines are very tangled, finally coordinated to two, IO performance is not known.
The database is an IO-intensive application, so evaluate the server's IO performance first to see if it matches the machine on the line.

I've always known that using the DD (device to device) command can simply test the disk's IO read and write speed, but not delve into it.
But this time to do performance testing of the relationship, need to get a relatively accurate value (before the test ate this aspect of the loss, insert a digression, performance testing must first confirm a good test environment. ）。

The common online method is to use the Hdparm and dd command to test, but the actual use of the problem, and test results always feel biased, the heart is not bottom.
So it is safe to study the next two commands, but also do some testing and analysis, simple summary.

usage:
1. Test IO Read
Hdparm-t--direct/dev/sda3

Io reads the above command test, but the Hdparm tool needs to be installed on its own and needs to be executed by the root user.

2. Test IO Write
Sync;/usr/bin/time-p bash-c "(dd If=/dev/zero of=test.dd bs=1000k count=20000;sync)"

The command line above is somewhat complex and is explained in detail:
(1) The Sync command is used to flush the file system's buffer, and the Sync command actually performs the data buffer written to disk in memory.
The next sync command is performed first to reduce the impact on subsequent tests. You can also use echo 3 >/proc/sys/vm/drop_caches to clear the cache.
(2) The time command is used to test the execution times of the command, and the shell is built with a temporal command, where we use the full path to specify the non-built command to use.
The output format for the-P option setting time is the POSIX default time format, in seconds, and the output form of time-p can be seen in the subsequent test section.
(3) The Bash command-C option is performed with the following string arguments as a bash script, which seems to be superfluous, as if direct execution is possible.
In fact, because the string contains two command lines, the time command needs to count the execution times of the two command lines.
(4) The parenthesis means that another child process to execute the script in parentheses, DD This command line here does not say, do not know Baidu bar.
The focus is on the sync command, because when DD exits, most of the content that this command line commits is in memory buffers (write cache), even if the machine's memory is much larger than the data you submit,
It is possible that the data is in memory, and then the sync is performed to write the memory data to the disk, otherwise it will be the test memory write speed, which is obviously not the result you want.
(5) DD test, need to add oflag=direct, measured is the real disk IO (has not been tested).
Test:
Some tests were done on a memory 64g,sas hard drive.
During the test, we need to pay attention to two points, disk read and write speed and IO usage, we use the Iopp and Iostat tools to observe respectively.

Test Write Speed:
Test write 20G data, the larger the amount of data, the test value should be more accurate.
# sync;/usr/bin/time-p Bash-c "(dd If=/dev/zero of=test.dd bs=1m count=20000)"
20000+0 Records in
20000+0 Records out
Real 92.87
User 0.00
SYS 18.08

The time to write 20000M is 92.87s, so the write speed of the disk is 215.35mb/sec.
With Iostat observation, the%util has been in the state of 100%, and the DD command write speed shown by Iopp is basically the same as the value calculated above. (Sampling time set to 5s)
# iostat-x 5
device:rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await SVCTM%util
SDA 0.00 0.00 0.20 864.00 1.60 449004.80 519.56 143.61 163.46 1.16 100.00

# Iopp-i-K 5
PID Rchar wchar syscr syscw rkb wkb cwkb command
32566 1033216 1033216 0 0 0 1034868 0 DD

Note: The current working directory of the command line is on the partition/dev/sda3, so test.dd This file is also created on this partition, the following read test will be used to this file.

test Read speed:
Make a comparison between DD and hdparm:
# hdparm-t--direct/dev/sda3
/dev/sda3:
Timing o_direct disk reads:2108 MB in 3.00 seconds = 702.34 mb/sec

The test result of the hdparm is that the 3s reads 2108MB and the reading speed is 702.34MB.

Because the Hdparm test time is too short, the iostat and Iopp sampling times are adjusted to 1s.
device:rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await SVCTM%util
SDA 0.00 0.00 2752.00 1.00 1409024.00 8.00 511.82 4.64 1.69 0.35 96.10

PID Rchar wchar syscr syscw rkb wkb cwkb command
32349 733184 0 0 0 733184 0 0 hdparm

Use the Test.dd file you just generated (size 20G) to do a read test
# echo 3 >/proc/sys/vm/drop_caches; /USR/BIN/TIME-P DD if=test.dd of=/dev/null bs=1m
20000+0 Records in
20000+0 Records out
Real 35.69
User 0.00
SYS 9.81

Read 20G, reading speed is 560.38mb/sec.

Read will be faster than write, Iostat and Iopp sampling time adjusted to every 3s.
device:rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await SVCTM%util
SDA 106.67 0.00 5851.00 0.33 1434256.00 2.67 245.12 1.62 0.28 0.16 93.60

PID Rchar wchar syscr syscw rkb wkb cwkb command
17004 1449984 1449984 1 1 1450760 0 0 DD

There is a difference between the read speed and the hdparm of DD test,
The read-write block size test is set through the BS option (the default is 512 bytes, the test uses 1M),
It can be seen that the measured speed of DD is related to the size of the read-write block, and may be affected by the process of Io reading and writing in the system.
Hdparm test principle did not do in-depth research, it may be and DD test methods differ, need to know this.

Overall, the actual test speed of Io is affected by a number of factors, including the way of reading and writing (random or sequential, hdparm and DD tests are sequential read and write), caching mechanism, test sampling and so on.
So it is not likely to get a definite value (same command line multiple tests are not the same, but the difference is smaller), the above method in the read test is recommended to use Hdparm.
Although the above data is biased, the IO performance of the machine can be generally analyzed. Just need to be clear, these test values are in what kind of environment to obtain.

How Linux tests IO performance (disk read/write speed)