Factors that affect the performance of IO-intensive applications

If we have ever experienced performance problems in IO-intensive applications, we probably all know that the performance of the system will decrease with the increase in disk pressure. This factor is well known, but the reasons behind it may be less clear. This article will try to explain what is going on behind this process.

A typical scenario is that when the data is written to a file, it is first written to the memory area to retain the page cache, and the cached page holds data that has not yet been written, which is called dirty data, and after an event, the system kernel flushes the dirty data to the device queue that is persisted to the hard disk, depending on the IO policy. Once the data arrives in the queue, the rest is machine operation: The device drives the Read IO request, then rotates, finds and writes to the physical disk block location where the file resides. The first is to write the log file before the data is written to the actual file.

In a recent discussion with other engineers, an idea was made to reduce disk write latencies by disabling the logging file system. While this idea is correct, because this practice reduces disk operations once per disk write, the time it takes to write logs is negligible because the logs are not annually updated and the files written are in the same block. The benefits of using log files to recover from disk crashes are also far greater than just a slightly reduced latency write operation.

More importantly, the bottleneck of IO-intensive applications is not the log-writing step when the system flushes dirty data pages to disk. The throughput of the refresh is limited by the bandwidth of the device. A typical hard drive with a speed of 15K can achieve 120mb/seconds in the best case, while the actual bandwidth is even smaller at random io. To better illustrate the problem, assume that the system uses the default 30-second refresh policy of the Redhat Linux system, and that the app is written at 20M per second. Then after 30 seconds, the system will accumulate 600M of our data needs to be flushed to disk (assuming this has not crossed the dirty data page before). In Linux, the refresh operation is done through the Pdflush daemon. In the case of optimal sequential writing, it will use all Pdflush threads (default to 8 threads) and will require a full 5 seconds to flush dirty data from the cache page to the device queue. This 5-second side effect is twofold: the cache page area is always busy and the disk bandwidth is exhausted. One way to monitor the situation is to check the length of the disk queue. In Linux, it is primarily the "Writeback" value in/proc/meminfo or the "Avgqu-sz" value of the SAR (System activity Reporter).

The scenario becomes more complex when the JVM's garbage collector is cleaning up heap memory and is stuck in the kernel's busy refresh (kernel-busily-flushing) moment. Some GC events are Stop-the-world (STW) events, which require that all threads in the application be paused to achieve a secure state so that objects in the heap memory can be moved out. If an app's thread tries to write data while the kernel is busy flushing, the thread will block after the character is refreshed and cannot respond to a request to pause. This leads to a chain reaction: a busy disk blocks the write thread, which prolongs the time of the garbage collection (GC) pause, while pausing makes the app unresponsive. The problem can be seen through the garbage collection (GC) log, where there is a long stw pause, accompanied by a small amount of CPU time spent on "usr" and "sys" on disk activity.

And it gets worse when the system memory usage reaches a limit. By default, in Linux, the memory Exchange function is turned on by setting "Swappiness" to 60. With this setting, when the system memory pressure is high and the IO is busy, the system will actively swap the "idle" Memory pages to disk (yes, more disk IO) from the running process to make more space for the cached pages. The displaced process also needs to be swapped again, because when the process is active again, it needs to be swapped back to the memory page first. Conversely, if swappiness is disabled (setting/proc/sys/vm/swappiness to 0) to the memory pages of the process as much as possible in memory, then no choice, the kernel will flush dirty data to disk more frequently to free memory pages. Premature and frequent refreshes increase the pressure per second of Io, reducing disk read and write performance, which makes the original problem even worse.

So what are the mitigation strategies?

at the system level , for our specific IO load is usually checking the kernel's refresh strategy, usually tuning to optimal performance requires several rounds of testing. The recommended adjustment switches under Linux are:

• /proc/sys/vm/dirty_expire_centiseconds

• /proc/sys/vm/dirty_background_ratio

at the device configuration level , it helps to avoid single device queue blocking issues by storing frequently accessed files on different devices. Alternatively, using multiple sets of RAID1 will result in more device queues, which is better than using a single disk with only one device queue.

For example, 4 hard drives are shown, and 2 x RAID1 settings (left) provide two device queue access systems. Conversely, the other 4 device RAID10 settings (on the right) provide only one device queue.

If cost permits, consider upgrading to SSD, which has a bandwidth of 6 to 7 times times the size of the rotational disk bandwidth. The recommendation for SSDs is that SSDs occasionally need to do data compression. The effect of data compression on performance is not good, the discussion on this aspect is best to open a new article discussion. Also be aware of logical volume management (LVM). Using LVM will introduce an extra small delay relative to SAS disk access speed, but it can be negligible, but this delay becomes very noticeable when using SSD drives because the SSD speed itself is much faster than SAS.

at the application level , check any configuration to avoid double buffering of dirty data pages. For example, in MySQL we can turn on the Direct_io feature so that dirty data is only cached in MySQL's memory, not on the system cache page.

Last but not least, try to avoid unnecessary disk access.

2. This article is translated by the programmer architecture

4. This article is translated from Http://architects.dzone.com/articles/performance-impact-io?mz=110215-high-perf

6. Reprint Please be sure to indicate this article from: Programmer Architecture (No.:archleaner )

8. More articles please scan code:

Factors that affect the performance of IO-intensive applications