Introduction to Linux memory management mechanism

Author: Technology Achievement dream

Link: http://ixdba.blog.51cto.com/2895551/541355

 

1. Physical memory and virtual memory

We know that reading and writing data directly from the physical memory is much faster than reading and writing data from the hard disk. Therefore, we hope that all data can be read and written in the memory, while the memory is limited, this introduces the concept of physical memory and virtual memory.
Physical memory is the memory size provided by the system hardware and is the real memory. Compared with physical memory, there is also a concept of virtual memory in Linux, virtual Memory is a strategy proposed to meet the shortage of physical memory. It is a logical memory virtualized by the disk space. The disk space used as the virtual memory is called the swap space ).
As an extension of the physical memory, Linux uses the virtual memory of swap partitions when the physical memory is insufficient. In more details, the kernel will write the memory block information that is not used for the moment to the swap space, in this way, the physical memory has been released, and this memory can be used for other purposes. When the original content is required, the information will be re-read from the swap space into the physical memory.
Linux memory management adopts a paging access mechanism. To ensure that the physical memory can be fully utilized, when appropriate, the kernel automatically exchanges infrequently used data blocks in the physical memory to the virtual memory, and retains frequently used information to the physical memory.
To learn more about the Linux memory running mechanism, you need to know the following aspects:
First, the Linux system will perform page switching from time to maintain as much free physical memory as possible. Even if there is no need for memory, Linux will swap out memory pages that are temporarily unavailable. This avoids waiting for the switching time.
Second, page switching in Linux is conditional. Not all pages are switched to virtual memory when not in use. Algorithm Only swap some infrequently used page files to the virtual memory. Sometimes we can see that there is still a lot of Linux physical memory, but the swap space is also used. In fact, this is not surprising. For example, a process that occupies a large amount of memory needs to consume a lot of memory resources, and some infrequently used page files will be exchanged to the virtual memory, however, when the process that occupies a lot of memory resources ends and releases a lot of memory, the page files that have just been exchanged will not be automatically exchanged into the physical memory unless necessary, now the physical memory of the system will be much idle and the swap space will be used. You don't have to worry about this. You just need to know what it is.
Finally, the pages in the swap space will be first exchanged to the physical memory during use. If there is not enough physical memory to accommodate these pages, they will be immediately exchanged, there may not be enough space in the virtual memory to store these swap pages, which will eventually lead to issues such as false crashes and service exceptions in Linux. Although Linux can recover itself within a period of time, however, the recovered system is basically unavailable.
Therefore, reasonable planning and design of Linux memory usage is very important.

2. Memory monitoring
As a Linux system administrator, it is very important to monitor the memory usage status. It helps you to understand the memory usage status, such as whether the memory usage is normal and whether the memory is in short supply, the most commonly used commands for monitoring memory include free and top. below is the output of a system free:
[Haixigov @ webserver ~] $ Free
Total used free shared buffers cached
Mem: 16402432 16360492 41940 0 465404 12714880
-/+ Buffers/cache: 3180208 13222224
Swap: 8193108 264 8192844
Let's explain the meaning of each option in the output result:
The first line is:
Total: the total size of the physical memory.
Used: how small the physical memory is.
Free: idle physical memory value.
Shared: memory value shared by multiple processes.
Buffers/cached: disk cache size.
Row 2 mem: indicates the physical memory usage.
Row 3 (-/+ buffers/cached): indicates the disk cache usage status.
Row 4: swap indicates the memory usage status of the swap space.
The memory status output by the free command can be viewed from two perspectives: one is from the kernel perspective and the other is from the application layer perspective.

1. view the memory status from the kernel perspective
That is, the kernel can be directly allocated to it. No additional operation is required, that is, the value of the mem entry in the second line of the free command output above. It can be seen that the physical memory of the system is 16 GB, the idle memory is only 41940 kb, that is, a little more than 40 MB. Let's do this calculation:
16402432-16360492 = 41940
In fact, the total physical memory minus the used physical memory gets the idle physical memory size. Note that the available memory value 41940 does not include the memory size in the buffers and cached states.
If you think that the idle memory of this system is too small, you will be wrong. In fact, the kernel fully controls the memory usage. When Linux needs the memory, or change the memory in the buffers and cached state to the Free State when the system is running for the system to use.

2. view the system memory usage status from the application layer perspective
This is the application running on Linux.ProgramThe memory size that can be used, that is, the output of the free command line 3 "(-/+ buffers/cached)". We can see that the memory used by this system is only 3180208 kb, when the idle memory reaches 13222224 kb, the following calculation continues:
41940 + (465404 + 12714880) = 13222224
Through this equation, we can see that the physical memory value available for the application is the free value of the mem item plus the sum of the buffers and cached values. That is to say, the free value includes the size of the buffers and cached items,
For applications, the memory occupied by buffers/cached is available, because buffers/cached is used to improve the file reading performance. When the application needs to use the memory, buffers/cached will be quickly recycled for use by the application.

3. Similarities and Differences between buffers and cached
In Linux, when an application needs to read data from a file, the operating system first allocates some memory to read data from the disk into the memory, then, the data is distributed to the application. When you need to write data to a file, the operating system first allocates memory to receive user data, and then writes the data from the memory to the disk. However, if a large amount of data needs to be read from the disk to the memory or written into the disk by the memory, the system's read/write performance becomes very low, because whether it is reading data from the disk, writing data to a disk is a very time-consuming and resource-consuming process. In this case, the buffers and cached mechanisms are introduced in Linux.
Both buffers and cached are in-memory operations to save the files that have been opened by the system and the file attribute information. In this way, when the operating system needs to read some files, it will first search in the buffers and cached memory areas, if the data is found, read the data directly and send it to the application. If the data is not found, the data is read from the disk. This is the operating system's cache mechanism, this greatly improves the performance of the operating system. However, the buffer content of buffers and cached is different.
Buffers is used to buffer Block devices. It only records metadata (metadata) and tracking in-flight pages of the file system, while cached is used to buffer files. More broadly speaking, buffers is mainly used to store contents in directories, file attributes, and permissions. Cached is used directly to remember the files and programs we opened.
To verify whether our conclusion is correct, you can open a very large file through VI to see the changes in cached, and then vi the file again, I feel the similarities and differences between the two open speeds, is the second open faster than the first one?
Run the following command:
Find/*-name *. conf
Check whether the buffers value has changed, and then execute the find command again to see if the display speed is different twice.
The memory operating principle of the Linux operating system is designed to a large extent based on the requirements of the server. For example, the system buffer mechanism caches frequently used files and data in cached, linux always tries to cache more data and information, so that you can directly retrieve the data from the memory when you need it again, without a long disk operation, this design improves the overall performance of the system.

Use of swap
Although the current memory has become very cheap, swap still has great application value. Reasonable planning and use of swap partitions are crucial to the stable operation of the system. In Linux, you can use a regular file in the file system or an independent partition as a swap space. At the same time, Linux allows multiple swap partitions or swap files.

1. Create a swap space
The swap file required for creating a swap space is a common file. However, unlike creating a swap file, the swap file must be created using the DD command, and the file must be located on the local hard disk, you cannot create swap files on the Network File System (NFS. For example:
[Root @ localhost ~] # Dd If =/dev/Zero of =/data/swapfile BS = 1024 COUNT = 65536
65536 + 0 records in
65536 + 0 records out
In this way, a swap file with continuous space is created, and the size is about 60 MB. The following is a brief description of the DD command:
If = input file or device name.
Of = Name of the output file or device.
IBS = bytes indicates that bytes are read at a time (that is, the size of a block is bytes ).
Obs = bytes indicates writing bytes at a time (that is, the size of a block is bytes ).
BS = bytes, and set the size of read/write blocks, in bytes. This parameter can replace IBS and obs.
Count = blocks only copies blocks.
Skip = blocks indicates that the blocks are skipped from the beginning of the input file and then copied.
Seek = blocks indicates that the blocks are skipped from the beginning of the output file and then copied. (Usually only valid when the output file is a disk or tape)
The input device/dev/zero indicates a device file whose output is always 0. You can use it as the input to obtain all files that are empty.

2. Activate and use swap
First, use the mkswap command to specify the device or file used as the swap space:
[Root @ localhost ~] # Mkswap/data/swapfile
Setting up swapspace version 1, size = 67104 KB
[Root @ localhost backup] # Free
Total used free shared buffers cached
Mem: 2066632 1998188 68444 0 26160 1588044
-/+ Buffers/cache: 383984 1682648
Swap: 4088500 101036 3987464
The above output shows that we have specified a 67104 kb swap space, and the new swap space has not yet been used. The following describes the mkswap command. The general format of mkswap is as follows:
Mkswap [parameter] [device name or file] [swap zone size]
Parameters:
-C: Check whether damaged blocks exist before creating a swap zone.
-V0: creates an old-style swap zone, which is the default value.
-V1: Create a new exchange zone.
Swap size: the size of the SWAp, in 1024 bytes.
After setting the swap partition, run the Swapon command to activate swap:
[Root @ localhost ~] #/Usr/sbin/Swapon/data/swapfile
[Root @ localhost backup] # Free
Total used free shared buffers cached
Mem: 2066632 1997668 68964 0 27404 1588880
-/+ Buffers/cache: 381384 1685248
Swap: 4154028 100976 4053052
The free command shows that the swap size has changed from 4088500k to 4154028 K, and the difference is about 60 m, which is equal to the size of an added swap file, this indicates that the newly added swap partition can be used, but if Linux is restarted, the newly added swap space will become unavailable. Therefore, you need to add the automatic loading settings in/etc/fstab:
/Data/swapfile none swap SW 0 0
So far, Linux can automatically load swap partitions after restart. In fact, Linux will execute the "Swapon-a" command during startup. This command will load all the swap spaces listed in/etc/fstab.

3. Remove swap
Use swapoff to remove a swap space
[root @ localhost ~] #/Usr/sbin/swapoff/data/swapfile
you can also use "swapoff-a" to remove all swap spaces defined in/etc/fstab, "swapoff-a" corresponds to the "Swapon-a" mentioned above. After "swapoff-a" is executed, the free command output is as follows:
[root @ localhost backup] # Free
total used free shared buffers cached
mem: 2066632 2048724 17908 0 30352 1642748
-/+ buffers/cache: 375624 1691008
swap: 0 0 0

 

Complete!