Cache memory consumption in Linux solution

In Linux systems, we often use the free command to view the state of the system's memory usage. On a RHEL6 system, the display of the free command is probably a state:

Here the default display unit is KB, my server is 128G of memory, so the numbers appear relatively large. This command is almost always a command for anyone who has ever used Linux, but the more such commands seem to really understand the fewer people (I mean the smaller the ratio). In general, the understanding of this command output can be divided into several levels:

1. Don't know. The first reaction of such people is: God, memory with a lot of, 70 more than G, but I almost do not run what big program AH? Why is that? Linux is good for memory!

2. I think I know it very well. Such a person generally evaluated will say: Well, according to my professional eyes, the memory is used about 17G, there is a lot of memory available. Buffers/cache occupy more, indicating that there are processes in the system have read and write files, but it does not matter, this part of the memory is when idle to use.

3. Really know. The reaction of this kind of person is that people feel the least understanding of Linux, their response is: "The free display is this, OK, I know." God horse? You ask me these memory enough, of course I do not know! How do I know how your program is written?

Based on the current technical documentation on the Web, I believe the vast majority of people who know a bit of Linux should be on the second level. It is generally believed that the memory space occupied by buffers and cached can be released as free space when the memory pressure is large. But is that really the case? Before we demonstrate this topic, let's briefly introduce what buffers and cached mean:

What is Buffer/cache?

Buffer and cache are two nouns that are used in computer technology, which can have different meanings in a non-context. In Linux memory management, the buffer here refers to the Linux memory: buffer cache. The cache here refers to the memory of Linux: Page cache. Translated into Chinese can be called buffer cache and page cache. Historically, one (buffer) was used as a cache for IO devices, while the other (cache) was used as a read cache for IO devices, where IO devices mainly referred to block device files and ordinary files on file systems. But now, the meaning of them is different. In the current kernel, page cache, as its name implies, is a cache of memory pages, which, in a way, can be managed using page cache as its cache if the memory is managed with page allocation. Of course, not all of the memory is managed by pages (page), and many are managed against blocks (block), which are used in buffer cache if the cache function is to be used. (from this point of view, is it better to rename the buffer cache called block cache?) However, not all blocks have a fixed length, the length of the block on the system is determined primarily by the block device used, and the page length is 4k on the X86 either 32-bit or 64-bit.

By understanding the differences between these two sets of caching systems, you can understand exactly what they can be used to do.

What is page cache?

Page cache is used primarily as a cache of file data on the file system, especially when the process has read/write operations on the file. If you think about it, as a system call that can map files to memory: Mmap is it natural that you should also use page cache? In the current system implementation, the page cache is also used as a caching device for other file types, so in fact page cache is responsible for most of the block device file caching work.

What is buffer cache

The buffer cache is designed to be used by a system that caches chunks of data when the system is read and written to a block device. This means that some operations on blocks are cached using buffer cache, such as when we are formatting the filesystem. In general, two cache systems are used together, such as when we write to a file, the contents of the page cache are changed, and the buffer cache can be used to mark the page as a different buffer, and to record which buffer was modified. This way, the kernel does not have to write the entire page back when it performs subsequent write-back (writeback) of the dirty data, only to write back the modified part.

How do I recycle the cache?

The Linux kernel triggers memory recycling when memory is about to run out, freeing up memory for use in memory-hungry processes. In general, the primary memory release in this operation comes from the release of Buffer/cache. In particular, more cache space is used. Since it is mainly used for caching, only when the memory is sufficient to speed up the process of file read and write speed, then in the case of large memory pressure, of course, it is necessary to empty the cache, as free space for the relevant process use. So in general, we think that Buffer/cache space can be released, and this understanding is correct.

But the job of clearing the cache is not without cost. Understand what the cache does. The cache must be guaranteed to be consistent with the data in the corresponding file in order to release the caches. so with the cache cleanup behavior, the system IO is generally high. because the kernel wants to compare the data in the cache with the data on the corresponding hard disk file, if the inconsistency needs to be written back, it can be recycled.

We can also use the following file to manually trigger the cache cleanup in addition to the memory being exhausted in the system:

[Email protected] ~]# cat/proc/sys/vm/drop_caches 1

The method is:

Echo 1 >/proc/sys/vm/drop_caches

Of course, this file can be set to a value of 1, 2, 3, respectively. The meanings they represent are:

echo 1 >/proc/sys/vm/drop_caches: Indicates clear Pagecache. Echo 2 >/proc/sys/vm/drop_caches: Represents the purge of objects in the slab allocator (including the catalog item cache and the Inode cache). The slab allocator is a mechanism for managing memory in the kernel, where many cache data implementations are pagecache. Echo 3 >/proc/sys/vm/drop_caches: Indicates that the cache objects in the Pagecache and slab allocator are cleared.

The following optimizations are followed:

Cache memory consumption in Linux solution