Linux Memory management mechanism

A physical memory and virtual memory

         We know that reading and writing data directly from physical memory is much faster than reading and writing data from a hard disk, so we want all of the data read and written to be done in memory, and memory is limited, which leads to the concept of physical and virtual memory.
Physical memory is the amount of memory provided by the system hardware, is the real memory, relative to the physical memory, under Linux there is a virtual memory concept, virtual memory is to meet the lack of physical memory of the proposed strategy, it is the use of disk space virtual out of a piece of logical memory, The disk space used as virtual memory is called swap space.
As an extension of physical memory, Linux will use the virtual memory of the swap partition when physical memory is low, in more detail, the kernel will write the memory block information that is temporarily unused to the swap space, so that the physical memory has been released, this memory can be used for other purposes, when the original content needs to be used, This information is re-read into the physical memory from the swap space.
Linux memory management is a paging access mechanism, in order to ensure that the physical memory can be fully utilized, the kernel will be in the physical memory of infrequently used data blocks automatically swapped into virtual memory, and the information often used to retain the physical memory.
To learn more about Linux memory operating mechanisms, you need to know the following:
First of all, the Linux system will occasionally do the paging operation to keep as much free physical memory as possible, even if there is nothing to do with memory, and Linux will swap out the memory pages that are temporarily unused. This avoids the time required to wait for the interchange.
Second, Linux for the page exchange is conditional, not all the pages are swapped to virtual memory when not in use, the Linux kernel based on the "most Frequently used" algorithm, only some infrequently used paging files to virtual memory, sometimes we will see a phenomenon: Linux physical memory and many , but the swap space is also used a lot. In fact, this is not surprising, for example, a very large memory of the process to run, the need to spend a lot of memory resources, there will be some infrequently used paging file is swapped into virtual memory, but later this memory-intensive process ended and freed a lot of memory, just The paging file that is swapped out does not automatically swap into physical memory, unless it is necessary, then the system physical memory will be idle a lot at the moment, while the swap space is also being used, there is a phenomenon just said. Don't worry about that, just know what's going on.
Finally, the Swap space page will be swapped to physical memory first, and if there is not enough physical memory to accommodate the pages, they will be swapped out immediately, so there may not be enough space in the virtual memory to store the swap pages, which will eventually lead to a fake crash, service exception, and so on. , Linux can recover itself over a period of time, but the restored system is largely unusable.
Therefore, it is very important to plan and design the use of Linux memory rationally.

Two monitoring of memory
As a Linux system administrator, monitoring memory usage status is very important, through monitoring to help understand the use of memory state, such as memory consumption is normal, memory shortage, and so on, the most commonly used to monitor memory commands have free, top, etc., the following is the output of a system free:

[[email protected] ~]$ free        total used         free      shared   buffers   cachedmem:         16402432    16360492      41940        0     465404   12714880-/+ buffers/cache:        3180208   13222224Swap:        8193108      8192844

We explain the meaning of each option in the output result:
First, the first line:
? Total: The overall size of the physical memory.
? used: How small is the physical memory already in use.
? Free: The idle physical memory value.
? shared: The memory value that multiple processes share.
? buffers/cached: The size of the disk cache.
Second line of Mem: represents physical memory usage.
Third row (-/+ buffers/cached): Represents the disk cache usage status.
Line four: Swap indicates the swap space memory usage state.
The memory state of the free command output can be viewed in two angles: one from the kernel point of view, one from the application layer perspective.

1. View the state of memory from the perspective of the kernel
Is that the kernel can now be directly assigned to, do not need additional operations, that is, the output of the above free command of the second mem value, it can be seen that this system physical memory 16G, free memory only 41940K, that is, 40M a little more, we do a calculation such as:
16402432-16360492=41940
In fact, the total physical memory minus the physical memory that has been used to get the free physical memory size, note that the available memory value of 41940 does not contain the memory size in the buffers and cached states.
If you think that the system is too small, you are wrong, in fact, the kernel completely control the use of memory, Linux will need to memory, or when the system is running progressively, the buffers and cached state memory into the Free State of memory for the system to use.

2. Use state of system memory from the perspective of the application layer
That is, the amount of memory that the application running on Linux can use, that is, the output of the third line "(-/+ buffers/cached)" of the free command, and you can see that the memory used by this system is 3180208K and the idle memory reaches 13222224K. Continue to do such a calculation:
41940+ (465404+12714880) =13222224
This equation shows that the physical memory values available to the application are the free value of the MEM item plus the sum of the buffers and cached values, that is, the value of this value includes the size of the buffers and cached items,
For applications, buffers/cached occupies a memory that is available because buffers/cached is designed to improve the performance of file reads, and buffers/cached is quickly recycled for application use when the application needs memory.

3. Similarities and differences of buffers and cached
In the Linux operating system, when the application needs to read the data in the file, the operating system allocates some memory, reads the data from the disk into the memory, and then distributes the data to the application, and when the data needs to be written to the file, the operating system allocates the memory to receive the user data first. The data is then written from memory to disk. However, if there is a large amount of data that needs to be read from disk to memory or written to disk by memory, the system's read and write performance becomes very low, since it is a time-consuming and resource-intensive process to read data from disk or write data to disk, in which case Linux introduces buffers and Cached mechanism.
Buffers and cached are memory operations that are used to save files and file attribute information that have been opened by the system, so that when the operating system needs to read some files, it will first look in the buffers and cached memory areas, and if found, read them directly to the application. If you do not find the data needed to read from disk, this is the operating system caching mechanism, through the cache, greatly improve the performance of the operating system. But the content of buffers and cached buffer is different.
Buffers is used to buffer the block device, it only records the file system metadata (metadata) and tracking in-flight pages, and cached is used to buffer the file. More commonly said: buffers is mainly used to store content in the directory, file attributes and permissions and so on. and cached is used directly to memorize the files and programs we have opened.
In order to verify our conclusion is correct, you can open a very large file by VI, look at the change of cached, and then again VI this file, feel how the speed of two times to open the similarities and differences, is not the second opening speed significantly faster than the first time?
Then execute the following command:

Find/*-name  *.conf

See if the value of the buffers changes, and then repeat the Find command to see how the two times the display speed is different.
Linux operating system memory operation Principle, is largely based on the needs of the server design, such as the system's buffering mechanism will be used to cache files and data in cached, Linux is always trying to cache more data and information, so that the need for this data can be directly from the memory Without the need for a lengthy disk operation, this design approach improves the overall performance of the system.

Use of swap in three-swap space
Although memory has become very inexpensive now, swap still has a lot of value, reasonable planning and use of the swap partition, is essential to the stable operation of the system. Linux can be used as a swap space using either a regular file in the file system or a separate partition. Linux also allows the use of multiple swap partitions or swap files.

1. Create Swap swap Space
The swap file required to create the swap space is an ordinary file, but creating a swap file differs from creating a normal file, which must be done through the DD command, which must reside on the local hard disk and not create swap files on the network file System (NFS). For example:

[[email protected] ~]# dd If=/dev/zero of=/data/swapfile bs=1024 count=6553665536+0 Records in65536+0 Records out

This creates a continuous space of the swap file, size of about 60M, about the DD command to do a simple narration:
if= the input file, or the device name.
of= output file or device name.
The ibs=bytes represents a read-in bytes byte at a time (that is, a block size of bytes bytes).
Obs=bytes represents a write-bytes byte at a time (that is, a block size of bytes bytes).
Bs=bytes, also sets the size of the read-write block, in bytes, which replaces IBS and OBS.
Count=blocks copies only blocks blocks.
Skip=blocks means that the blocks block is skipped from the beginning of the input file before copying begins.
Seek=blocks indicates that the blocks block is skipped from the beginning of the output file before copying begins. (usually only valid if the output file is a disk or tape)
The input device/dev/zero here represents an output device file that is always 0, which can be used as input to get all empty files.

2. Activating and using Swap
The device or file that is used as the swap space is first specified by the Mkswap command:

[[Email protected] ~] #mkswap  /data/swapfilesetting up swapspace version 1, size = 67104 kb[[email protected] backup]# Free       Total used       free     shared    buffers     cachedmem:       2066632    1998188      68444          0      26160    1588044-/+ buffers/cache:     383984    1682648Swap:      4088500     101036    3987464

From the above output, we have specified a 67104 KB swap space, and the new swap space is not yet used, the following is a brief introduction to the following Mkswap command, Mkswap the general use of the format:
mkswap [parameter] [device name or file] [swap size]
Parameters:
C: Before establishing the swap area, first check if there are any damaged blocks.
-v0: Establishes the old swap area, which is a preset value.
-V1: Establishment of a new exchange zone.
Swap size: Specifies the size of the swap area, in 1024 bytes.
After setting the swap partition, the swap is then activated via the Swapon command:

[[Email protected] ~]#/usr/sbin/swapon/data/swapfile[[email protected] backup]# free             total       used       free     shared    buffers     cachedmem:       2066632    1997668      68964          0      27404    1588880-/+ Buffers/cache:     381384    1685248Swap:      4154028     100976    4053052

With the free command you can see that the swap size has changed from 4088500k to 4154028k, the difference is about 60M, just equal to the size of a swap file we added, which indicates that the new swap partition is ready to use, but if the Linux reboot, Then the new swap space becomes unavailable, so you need to add the auto-load setting in/etc/fstab:
/data/swapfile None swap SW 0 0
Since then, Linux has been able to automatically load the swap partition after rebooting. In fact, Linux executes the "swapon-a" command during startup, which loads all the swap space listed in/etc/fstab.

3. Removing swap
Remove a swap space with Swapoff
[Email protected] ~]#/usr/sbin/swapoff/data/swapfile
In fact, you can also remove all the swap spaces defined in/etc/fstab by "Swapoff-a", where "swapoff-a" corresponds to the "swapon-a" mentioned above. After executing "swapoff-a", the free command output is as follows:

[[email protected] backup]# free       total used       free     shared    buffers     cachedmem:       2066632    2048724      17908          0      30352    1642748-/+ buffers/cache:     375624    1691008Swap:            0          0          0

Linux Memory management mechanism