Analysis of buffer cache performance problem in Linux

1, the role of Buffer cache

To improve the IO performance of disk devices, we use memory as the cache for disk devices. When a user operates a disk device, the data is first written to memory, and then the dirty data in memory is refreshed to disk at timed intervals. This memory used as the disk data cache is the so-called buffer cache. In the previous Linux system, there is a very good buffer cache software layer, specifically responsible for the disk data caching. The file system is often architected on top of the disk device, and the VFS layer also provides file system level page cache to improve file system performance. This leads to two cache in the system and overlaps together, which is unnecessary and redundant. In order to solve this problem, the buffer cache software layer is weakened in the existing Linux system, and the page cache is integrated. Buffer cache and page cache using radix tree for maintenance, only when access to bare devices will use buffer cache, normal go file system IO will not use buffer cache.

We know that the page cache of the Ext3 file system is based on page size, so how big is the buffer cache block size? How does it affect performance? These two days I have done a lot of experiments on this problem on the Linux-2.6.23 platform, get some data results, and get the method of setting the cache block size from the source code analysis. The performance of this buffer cache is analyzed and explained for discussion.

2, Buffer Cache performance problems

2.1 Test Experiment

First let's do an experiment, on the Linux-2.6.23 platform, using the DD tool for a block device sequential write operation, you can use the following command format:

DD If=/dev/zero of=/dev/sda2 bs=<request_size> count=100

Using this command to test the IO performance of different request sizes (req_size) with different buffer cache block (blk_size) sizes, the test data shown in the following table can be obtained:

Table: Throughput under different buffer cache block size configurations

Make a performance comparison diagram of the data in the table, as shown in the following illustration:

As can be seen from the figure, the larger the cache block, the higher the IO performance is when the request size is smaller than the cache block size, but the performance has a significant leap after the request size is larger than the cache block size.

For example, when the buffer cache block size is configured to 2KB, the block performance of less than 2KB is basically about 19mb/s; When the buffer cache block size is configured 512B, the write performance of less than 512B remains in 5mb/s; when buffer When the cache block size is configured to 1024B, the write performance of less than 1KB is basically kept up and down 9.5mb/s. This shows that for less than the size of the cache block Small_write,buffer cache, its performance will be better, conversely, the worse performance, this is the role of buffer cache.

Observation found that once the request size is greater than or equal to the cache block size, performance increases dramatically, because the test tool IO pressure is large enough to quickly run out of disk performance. What is this for? In fact, when the request block comparison of hours, for the cache block is "local operation", this "local operation" will introduce buffer cache data read operations, and the data read operation and user write operations exist sequential relationship, which greatly affected the IO write performance. Therefore, when the request size is larger than the cache block, and can be aligned with the cache block, can take full advantage of the disk IO bandwidth, so it produces the above figure in the performance leap.

After seeing the test results in the above picture, we will think in the actual application, how do we choose the buffer cache block size? If the request size is 512B, it is clear that the buffer cache block set to 512 is more appropriate, if the request size is 256B, it is clear that the buffer cache block set to 2KB is more appropriate. Therefore, the individual thinks that the block size setting also needs to decide according to the actual application, the different application needs to set the different block size, this can make the overall performance achieves the best.