Linux performance monitoring I/O

Article Title: I/O about Linux performance monitoring. Linux is a technology channel of the IT lab in China. Includes basic categories such as desktop applications, Linux system management, kernel research, embedded systems, and open source.

I believe you are familiar with Linux performance monitoring. For I/O, let's first analyze some specific situations. In these cases, I/O will become the bottleneck of the system. We will use tools such as top, vmstat, iostat, and sar. The output of each tool reflects the performance of the system in different aspects.

Linux performance monitoring 1: a large number of I/O operations at the same time

In this case, we will find that the percentage of wa time of the CPU increases, proving that most of the idle time of the system is waiting for I/O operations.

# Vmstat 1

Procs ----- memory ----- --- swap --- io ---- -- system -- cpu ----

R B swpd free buff cache si so bi bo in cs us sy id wa

3 2 0 55452 9236 1739020 0 0 9352 2580 20 24 0 57

2 3 0 53888 9232 1740836 0 0 14860 2642 8954 23 25 0 52

2 2 0 51856 9212 1742928 0 0 12688 2636 8487 23 25 0 52

From this output, we can see that 50% of the CPU time is waiting for I/O operations. We can also see that the system's bi value is very large, this proves that the system has a large number of I/O requests to read disk content into the memory.

There is no good tool to see which process is performing I/O read/write. But we can guess through the output of the top command

# Top-d 1

Top-19:45:07 up, 3 users, load average: 6.36, 5.87, 4.40

Tasks: 119 total, 3 running, 116 sleeping, 0 stopped, 0 zombie

Cpu (s): 5.9% us, 87.1% sy, 0.0% ni, 0.0% id, 5.9% wa, 1.0% hi, 0.0% si

Mem: 2075672 k total, 2022668 k used, 53004 k free, 7156 k buffers

Swap: 2031608 k total, 132 k used, 2031476 k free, 1709372 k cached

Pid user pr ni virt res shr s % CPU % mem time + nFLT COMMAND

3069 root 5-10 450 m 303 m 280 m S 61.5. 68 15.0 vmware-vmx

3016 root 5-10 447 m 300 m 280 m S 21.8. 83 14.8 vmware-vmx

3494 root 5-10 402 m 255 m 251 m S 3.0 12.6. 65 3829 vmware-vmx

3624 root 5-10 401 m 256 m 251 m S 1.0 0: 29. 92 12.6 vmware-vmx

Sort the top output by faults. We can see that vmware generates the most page faults. That is to say, it performs a lot of IO operations.

Linux performance monitoring 2: Too small MPs queue

Any I/O operation takes a certain amount of time, and the time is determined for the hard disk, it contains the disk rotation delay RD (rotation delay) and the head search time DS (disk seek ). RD is determined by the disk speed (RPM. RD is half the time required for a disk to rotate for one week. For example, RPM is 10000.

RPS = RPM/60 = 166

1/166 = 0.0006 = 6 ms disk rotation takes 6 milliseconds for a week

RD = 6 ms/2 = 3 ms

The average search time of a disk is 3 ms, and the average latency of data transmission is 2 ms. The average time of an I/O operation is as follows:

3 ms + 3 ms + 2 ms = 8 ms

IOPS = 1000/8 = 125 IOPS (IOPS) is 125. Therefore, for an ephemeral RPM disk, the I/O operations it can handle are in the IOPS range of 120 ~ In the range of 150. If the system's I/O request exceeds this value, the disk will become a bottleneck of the system.

For systems, there are two different types of I/O pressure, continuous I/O and random I/O.

Continuous I/O is often used in enterprise-level databases and requires continuous reading of a large amount of data. The performance of this system depends on the size and speed of Data Reading and moving. We use iostat for monitoring and we will find that rKB/s and wKB/s are very high.

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm % util

/Dev/sda 0.00 12891.43 0.00 105.71 0.00 106080.00 0.00 53040.00 1003.46 1099.43 3442.43 26.49 280.00

From the output, we can see w/s = 105, wKB/s = 53040. So 53040/105 = 505KB per I/O.

For random I/O systems, the performance focus is not on the size and speed of data retrieval and transmission, but on the disk IOPS. Such systems have small but large I/O requests, such as Web servers and Mail servers. Their performance mainly depends on the number of requests that can be processed per second:

# Iostat-x 1

Avg-cpu: % user % nice % sys % idle

2.04 0.00 97.96 0.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm % util

/Dev/sda 0.00 633.67 3.06 102.31 24.49 5281.63 12.24 2640.82 288.89 73.67 113.89 27.22 50.00

From the output, we can see w/s = 102, wKB/s = 2640. so 2640/102 = 23KB per I/O. therefore, for continuous I/O systems, we need to focus on the ability of the system to read a large amount of data, that is, KB per request. we pay attention to IOPS values for random I/O systems.