Swap and BufferCache mechanisms in Linux memory

Article Title: Swap and BufferCache mechanisms in Linux memory. Linux is a technology channel of the IT lab in China. Includes basic categories such as desktop applications, Linux system management, kernel research, embedded systems, and open source.

With the advent and Development of Linux and more applications and servers, we should be very clear about the various management mechanisms of Linux. A complete Linux system mainly includes storage management and memory management, the file system and process management are composed of several aspects, and some good articles have been posted before. Share with you! The following describes the Swap and Buffer Cache mechanisms.

Linux supports virtual memory. virtual memory refers to the use of disks as RAM extensions, so that the available memory size increases accordingly. The kernel will write the content of memory blocks that are not currently used to the hard disk, so that this memory can be used for other purposes. When raw content needs to be used, they are re-read into the memory. These operations are completely transparent to users. Programs Running in Linux only see a large amount of memory for use, and do not notice that some of them reside on the hard disk from time to time. Of course, reading and writing hard disks is much slower than using real memory directly (several thousand times slower), so the program will not run as fast as it has been running in the memory. The hard disk used as the virtual memory is called the Swap Space ).

In general, pages in the swap space are first swapped into the memory; if there is not enough physical memory to hold them, they will be swapped out (to other SWAP spaces ). If you do not have enough virtual memory to accommodate all these pages, Linux will fluctuate and become abnormal. However, after a long period of time, Linux will recover, but the system is no longer available.

Sometimes, although there is a lot of idle memory, there is still a lot of swap space in use. This situation may occur. For example, if there is a need for swap at a certain moment, but then a large process that occupies a lot of physical memory ends and releases the memory. The exchanged data is not automatically exchanged into the memory unless necessary. In this case, the physical memory remains idle for a period of time. There is nothing to worry about, but it doesn't matter if you know what it is.

Many operating systems use virtual memory. Because they only need swap space at run time to avoid using swap space at the same time, except for the swap space of the currently running operating system, others are a waste. So it will be more efficient for them to share a swap space.

Note that if several people use this system at the same time, they will all consume memory. However, if two people run a program at the same time, the total memory consumption is not doubled, because only one copy of the code page and shared library exists.

The swap space is often used in Linux to keep as much free physical memory as possible. Even if there is no need for memory, Linux will swap out memory pages temporarily unused. This avoids the time required to wait for the swap: When the disk is idle, the swap can be done in advance.

Swap space can be dispersed on several hard disks. This improves the performance of related disks and access modes.

Disk read/write is slow compared to memory accessed (real. In addition, it is common to read the same part of the disk multiple times in a relatively short period of time. For example, someone may first read an e-mail message, then read the message into the editor for a reply, and then copy the message to the folder, make the email program read it again. Or consider how many times the ls command will be used in a system with many users. By reading information from the disk only once and saving it in the memory, you can speed up all other reads except for the first read. This is called disk buffering, and the memory used for this purpose is called Buffer Cache ).

However, because the memory is a limited and inadequate resource, high-speed buffering cannot do a lot (it cannot accommodate all the data to be used ). When the buffer is full of data, the data that is not used for the longest time will be discarded to free up memory space for new data.

The disk Buffer technology is equally effective for disk write operations. On the one hand, data written to the disk is often quickly read (for example, the original code file is saved to a file and read by the compiler ), therefore, it is a good idea to buffer the data to be written. On the other hand, programs can speed up operation by placing data in a buffer instead of writing it to a disk immediately. Later, write operations can be completed in the background without delaying program execution.

Most operating systems have high-speed buffering (although different), but not all follow the above principles. Write-through: data is immediately written to the disk (of course, data is also cached ). If the write operation is performed later, the cache is called write-back ). Background writing is more effective than direct writing, but it is also prone to errors: if the machine crashes or suddenly loses power, the changed data in the buffer will be lost. If the data that has not been written contains important notes, this may even mean that the file system (if any) is incomplete.

For the above reasons, there are many log file systems. After the data is modified in the buffer zone, the file system records the modification information, so that even if the system loses power at this time, after the system is restarted, the data will be restored from the log to ensure that the data is not lost. Of course, these questions are no longer covered in this article.

For the above reason, do not turn off the power before using the appropriate shutdown process. The sync command flushes buffer, that is, forces all unwritten data to the disk, it can be used to determine that all write operations have been completed. In traditional UNIX systems, a program called update runs on the background and performs the sync operation every 30 seconds. Therefore, you do not need to use the sync command manually. In Linux, there is another background program, bdflush, which runs more frequently but is not a full synchronization operation, to avoid the disk freezing caused by frequent disk I/O operations.

In Linux, bdflush is started by update. There is usually no reason to worry about this, but if the bdflush process is dead for some reason, the kernel will warn you about it, and you have to start it manually (/sbin/update ).

Cache is not actually a buffer for files, but a buffer block. It is the smallest unit for disk I/O operations (in Linux, they are usually 1 kb ). In this way, the directory, the super block, the note data of other file systems, and the disk data of non-file systems can be buffered.

The buffer effect is mainly determined by its size. If the buffer is too small, it is useless:

It can only hold a little data, so when it is reused, all buffered data will be dumped. The actual size depends on the data read/write frequency and the frequency at which the same data is accessed. Only the experiment method can be used.

If the cache has a fixed size, the cache is too large or not, because it will make the idle memory too small and lead to swap operations (this is also slow ). To make the most effective use of the actual memory, Linux automatically uses all idle memory as the high-speed buffer. When the program requires more memory, it will automatically reduce the buffer size.

This is the general mechanism of Linux memory in general cases. Of course, the operating mechanism of Linux memory is much more complicated than this, but only by understanding this mechanism can we easily manage servers!