Linux Memory management mechanism

first, about the computer CPU , Cache , memory, hard disk relationships and differences.

1, the CPU is also called the Central processor (cpu,central processing Unit) is an ultra-large-scale integrated circuit,

is the computing core (core) and control Unit of a computer. Its function is mainly to explain the computer instruction and processing the data in the computer software. The central processing Unit consists of three core components, an operator, a controller, and a bus, which are mainly composed of arithmetic logic units (ALU) and registers (RS).

2,cache is a high-speed buffer memory, is located between the CPU and the main memory of a small but high-speed memory. Because the CPU speed is much higher than the main memory, the CPU accesses the data directly from the memory to wait for a certain period of time, the cache holds the CPU just used or recycled part of the data, when the CPU re-use the part of the data can be directly called from the cache, which reduces the CPU waiting time, Improve the efficiency of the system. The cache is also divided into one-level cache (L1 cache) and level two cache (L2 cache), L1 cache is integrated within the CPU, L2 cache is usually soldered to the motherboard, and is now integrated into the CPU, with a common capacity of 256KB or 512KB L2 Cache.

3, memory is one of the important parts of the computer, it is a bridge to communicate with the CPU. All programs in the computer run in memory, so the performance of the memory affects the computer very much. Memory is also known as internal memory, which is used to temporarily store data in the CPU, as well as with external memory such as the hard disk. Memory contains a very wide range, generally divided into read-only memory (ROM), random memory (RAM) and cache memory (caches).

4, hard disk is one of the main computer storage media, consisting of one or more discs made of aluminum or glass. Hard drives are solid-state drives (SSD disks, new hard drives), mechanical hard drives (HDD legacy HDD), Hybrid drives (HHD a new hard drive based on traditional mechanical hard drives).

Additional notes:

virtual Memory is a computer system memory management technology, it is from the hard disk partition a piece of space to act as virtual memory use. It allows the application to assume that it has contiguous available memory (a contiguous, complete address space), and in fact, it is usually separated into multiple physical memory fragments, and some are temporarily stored on external disk storage and exchanged for data when needed. Currently, most operating systems use virtual memory, such as "virtual memory" for the Windows family, "swap space" for Linux, and so on.

The computer storage system is as follows:

Probably understand:

The computer hard disk to store a file a, if the user to execute this file a, first memory will be transferred from the hard drive file a related information into memory, memory to access the hard drive this process needs to wait (sports car speed), when the CPU to access memory also need to wait (aircraft speed), relative CPU <--- > Memory <---> hard disk This process is very fast. If a file is put into CACHE,CPU directly to the cache (rocket speed), it is equivalent to the CPU does not have to access the memory, directly access the cache, the process cpu<---> cache, but the cache capacity is very small, basic is to store some frequently accessed files, has also been limited.

Ii. Brief introduction of Linux system memory Management mechanism

question: often encounter some new Linux novice will ask the Linux system also does not run too many programs, but memory consumption how much? Using the top command or both of the free commands can be seen as follows:

Answer:

In Linux often found that the free memory is very small, it seems that all the memory is occupied by the system, the surface of the memory is not enough to use, it is not. This is an excellent feature of Linux memory management, which differs from Windows memory management in this regard. The main feature is that no matter how large the physical memory is, Linux is fully utilized to read some program-called hard disk data into memory, using the high-speed features of memory read and write to improve the data access performance of Linux system. Instead, Windows allocates memory for the application only when it needs memory, and does not take full advantage of the large capacity of the memory space. In other words, with each additional physical memory, Linux will be able to take full advantage of the benefits of hardware investment, and Windows will only do it as a device, even if the increase of 8GB or even larger.

This feature of Linux, mainly uses the free physical memory, divides the part space, as the cache, buffers, improves the data access performance.

The page cache (cache) is a primary disk cache implemented by the Linux kernel. It is primarily used to reduce I/O operations on disks. Specifically, by caching the data in the disk into physical memory, access to the disk becomes access to physical memory.

The value of the disk cache is two: first, accessing the disk is much slower than accessing the memory, so accessing the data from memory is faster than accessing it from disk. Second, once the data is accessed, it is likely that it will be accessed again in a short period of time

Annotations: The CPU cache (CPU cache) and the memory cache of the Linux system do not refer to the same cache in the system, and the cache that the Linux system sees through the free command refers to the memory cache. is the cache mechanism in which the Linux system introduces the memory mechanism.

Here's a look at the Linux memory management mechanism:

1 , physical memory, and virtual memory

We know that reading and writing data directly from physical memory is much faster than reading and writing data from a hard disk, so we want all of the data read and written to be done in memory, and memory is limited, which leads to the concept of physical and virtual memory.

Physical memory is the amount of memory provided by the system hardware, is the real memory, relative to the physical memory, under Linux there is a virtual memory concept, virtual memory is to meet the lack of physical memory of the proposed strategy, it is the use of disk space virtual out of a piece of logical memory, The disk space used as virtual memory is called swap space.

As an extension of physical memory, Linux will use the virtual memory of the swap partition when physical memory is low, in more detail, the kernel will write the memory block information that is temporarily unused to the swap space, so that the physical memory has been released, this memory can be used for other purposes, when the original content needs to be used, This information is re-read into the physical memory from the swap space.

Linux memory management is a paging access mechanism, in order to ensure that the physical memory can be fully utilized, the kernel will be in the physical memory of infrequently used data blocks automatically swapped into virtual memory, and the information often used to retain the physical memory.

To learn more about Linux memory operating mechanisms, you need to know the following:

(1), The Linux system will occasionally make a paging operation to keep as much free physical memory as possible, even if there is nothing to do with memory, and Linux will swap out the memory pages that are temporarily unused. This avoids the time required to wait for the interchange.

(2), Linux for the page exchange is conditional, not all the pages are swapped to virtual memory when not in use, the Linux kernel based on the "most Frequently used" algorithm, only some infrequently used paging files to virtual memory, sometimes we will see a phenomenon: Linux physical memory is still many, But the swap space is also used a lot. In fact, this is not surprising, for example, a very large memory of the process to run, the need to spend a lot of memory resources, there will be some infrequently used paging file is swapped into virtual memory, but later this memory-intensive process ended and freed a lot of memory, the page just swapped out The file is not automatically swapped into physical memory, unless it is necessary, then the system physical memory will be much more idle, while the swap space is also being used, there is a phenomenon just said. Don't worry about that, just know what's going on.

(3), The Swap space page in use will be swapped to physical memory, if there is not enough physical memory to accommodate these pages, they will be immediately swapped out, so, virtual memory may not have enough space to store these exchange pages, Linux will eventually cause false crashes, service anomalies and other issues, although Linux can be self-recovery over a period of time, but the restored system is basically unusable.

Therefore, it is very important to plan and design the use of Linux memory rationally.

2. Linux Monitoring of Memory

As a Linux system administrator, monitoring memory usage status is very important, through monitoring to help understand the use of memory state, such as memory consumption is normal, memory shortage, and so on, the most commonly used to monitor memory commands have free, top, etc., the following is the output of a system free:

Meaning of each option:

First line mem:

Total: Indicates the amount of physical memory.

Used: Represents the amount that the total is allocated to the cache (including buffers and cache), but may not be actually used in some of the caches.

Free: Memory that has not been allocated.

GKFX: Shared memory, general system will not be used, not discussed here.

Buffers: The number of buffers that are allocated but not used by the system.

Cached: The number of caches that the system allocates but has not been used.

Second line-/+ Buffers/cache:

Used: The total amount of buffers and caches actually used, as well as the amount of memory actually used.

Free: The sum of unused buffers with the cache and unallocated memory, which is the actual memory available to the system at the moment.

Third line swap:

Total: The amount of swap space.

Used: Swap space already in use.

Free: Swap space that is not being used.

System actual Memory Calculation: (second row-/+ Buffers/cache)

(actual memory used by Linux system) used = (first row) used minus (first row) buffers minus (first row) cache

(actual memory not used by Linux system) Free = (first line) free plus (first line) buffers Plus (first row) cache

Free the memory state of the command output can be viewed in two angles: one from the kernel point of view, from the perspective of the application layer:

View the state of memory from the perspective of the kernel

Is that the kernel can now be directly assigned to, do not need additional operations, that is, the above free command output of the second line of mem value, it can be seen that the total amount of physical memory of this system is 32113M, the memory has been used only 3 53M, we do a calculation such as:

32113–31759 = 353

In fact, the total physical memory minus the physical memory that has been used to get the free physical memory size, note that the available memory value of 353M does not contain the memory size in the buffers and cached states.

If you think that the system is too small, you are wrong, in fact, the kernel completely control the use of memory, Linux will need to memory, or when the system is running progressively, the buffers and cached state memory into the Free State of memory for the system to use.

Use state of system memory from the perspective of the application layer

That is, the amount of memory that an application running on Linux can use, that is, the output of the third line of the free command-/+ buffers/cached, and you can see that the memory used by this system is 7536M, and the idle memory reaches 24576M, and continues to do such a calculation:

353+ (7+24215) =24576

This equation shows that the physical memory value available to an application is the sum of the free value of the MEM item plus the buffers and cached values, that is, the value of this value is buffers and cached item size, and for applications, buffers/ Cached occupied memory is available because buffers/cached is designed to improve the performance of file reads, and when applications need to use memory, buffers/cached is quickly recycled for application use.

buffers and Cached Similarities and differences

Cache (Cached) is to save the read data, re-read if hit (find the required data) do not read the hard disk, if not hit the hard drive. The data will be organized according to the frequency of reading, the most frequently read content in the most easily found location, the content of the no longer read to the back row, until removed from

buffer (buffers) is based on the disk read-write design, the decentralized write operations centralized, reduce disk fragmentation and hard disk repeatedly seek, thereby improving system performance.

In the Linux operating system, when the application needs to read the data in the file, the operating system allocates some memory, reads the data from the disk into the memory, and then distributes the data to the application, and when the data needs to be written to the file, the operating system allocates the memory to receive the user data first. The data is then written from memory to disk. However, if there is a large amount of data that needs to be read from disk to memory or written to disk by memory, the system's read and write performance becomes very low, since it is a time-consuming and resource-intensive process to read data from disk or write data to disk, in which case Linux introduces buffers and Cached mechanism.

Cached is between the CPU and memory, buffer is between memory and disk, both cache and buffer are the capacity of the Linux kernel to use physical memory, all to solve the problem of speed is not equal. Buffers and cached are memory operations that are used to save files and file attribute information that have been opened by the system, so that when the operating system needs to read some files, it will first look in the buffers and cached memory areas, and if found, read them directly to the application. If you do not find the data needed to read from disk, this is the operating system caching mechanism, through the cache, greatly improve the performance of the operating system. But the content of buffers and cached buffer is different.

Buffers is used to buffer the block device, it only records the file system metadata (metadata) and tracking in-flight pages, and cached is used to buffer the file. More commonly said: buffers is mainly used to store content in the directory, file attributes and permissions and so on. and cached is used directly to memorize the files and programs we have opened.

Linux Memory management mechanism