Linux memory management

 

Hi, MM!

Hedgehog @ http://blog.csdn.net/littlehedgehog

 

 

 

 

 

 

I can understand a piece of kernel code, but I am not quite sure what the code will do in Linux. Many Linux kernel beginners may feel deeply about it. This is actually a stumbling block to look at the kernel. Many beginners are not very familiar with Linux, or simply "traveling" in the Linux environment, no programming in the Linux environment (or calling the Linux API) has been performed. Many people have vaguely understood a large part of kernel code, but do not know why, this is the reading obstacle caused by lack of familiarity with Linux. As you can imagine, if he had a Linux programming experience at first, he could do his best to know about it when reading the kernel. Therefore, it is recommended that readers take a look at the application programming examples on the Internet while studying the kernel code. Here I strongly recommend that you study some hacker code, which is short and concise, it is particularly suitable for System Call reference code.

 

Next, we will analyze the Linux kernel memory management from the application perspective. This is very simple for Kernel programmers. Okay. Let's get into the question.

 

When talking about memory management, the two concepts that pop up in our minds are virtual memory and physical memory. These two concepts are mainly supported by the Linux kernel. Linux provides two types of memory management: Linear zone, similar to 00c73000-00c88000, which corresponds to virtual memory,It actually does not occupy the actual physical memoryThe first level is the specific physical page, which corresponds to the physical memory on our machine.
A very important concept is mentioned here: Memory delay allocation. When a user applies for memory, the Linux kernel only allocates a linear zone (that is, virtual memory) to the user and does not allocate the actual physical memory. Only when the user uses the memory, the kernel allocates a specific physical page to the user, which occupies valuable physical memory. The kernel releases the physical page by releasing the linear zone, finding the corresponding physical page, and releasing it all.

 

1. Char * p = malloc (2048); // only 2048 of the virtual memory is allocated, which does not occupy the actual memory.
3. Strcpy (p, "123"); // The physical page is allocated. Although only three bytes are used, the memory is still allocated with 2048 bytes of physical memory.
5. Free (p); // find the corresponding physical page through the Virtual Address, release the physical page, and release the linear area.

We know that the user's process and kernel run at different levels, and the communication between the Process and the kernel is completed through system calls. Processes are applying for and releasing memory, mainly through brk, sbrk, mmap, and unmmap system calls. The passed parameters are mainly the corresponding virtual memory.
Note that a process can only access virtual memoryIn fact, it does not see the use of the kernel physical memory, which is completely transparent to the process..
Glibc memory manager
So every time we call malloc to allocate a piece of memory, what about system calls?
The answer is no. Here I want to introduce a new concept, Memory Manager of glibc.
We know that functions such as malloc and free are included in the library functions in the glibc library. Let's think about how inefficient the program will be if every memory operation calls a system call. In fact, glibc uses a wholesale and retail method to manage memory. Glibc requests a large block of memory (Virtual Memory) each time it calls the system. When the process requests the memory, glibc extracts a block from its own memory to the process.

Difficulties faced by the Memory Manager

When we write a program, the size of the memory block requested each time is irregular, and frequent application and release exist. This will inevitably lead to memory fragmentation. However, memory fragmentation directly results in a failure to meet the application of large memory blocks, thus occupying more system resources. If the chunks are organized, the cpu load will increase, many of them are conflicting indicators. I will not elaborate on them here.
When writing a program and involving memory, we have two concepts: heap and stack.Traditionally, the memory address of the stack increases downward, and the heap memory address increases upwards.

The malloc and free functions are mainly used for heap operations, and the programmer controls the access to memory. Here, the heap memory address increases upwards. This sentence is not completely correct. Glibc uses MMAP to apply for memory larger than kb for heap memory applications. This does not ensure that the memory address increases upwards. BRK is used for Memory Applications smaller than kb, it is correct. The threshold value of KB can be set through the library function of glibc.

Here I will first talk about the application of large memory, that is, corresponding to the MMAP system call.
For large memory applications, glibc uses the MMAP system call to divide it into another virtual address for separate use by the process. When the memory is released, use the unmmap system call to release the memory. This process will not cause issues such as memory fragmentation.
For small memory applications, after the program starts, the process will get an address at the bottom of the heap. Every time the process applies for memory, glibc will increase the heap to expand the memory space, that is to say, the heap address increases upwards. When performing operations on these small blocks of memory, the memory fragmentation problem occurs. In fact, the BRK and sbrk system calls are to adjust the heap top address pointer.
When will the heap memory be released?
When glibc finds that the heap top has consecutive free space, it will adjust the heap top position through BRK or sbrk system calls and return the occupied memory to the system. In this case, the kernel will release the occupied physical memory by deleting the corresponding linear zone.
The following is a question about memory holes:
In one scenario, there is a memory in use at the top of the heap, and a large continuous memory has been released below. Can this memory be released? Can the corresponding physical memory be released?
Sorry, no.
That is to say, as long as the applied memory at the top of the heap is still in use, I will release more memory below and will not be returned to the system, but still occupy the physical memory. Why?
This is mainly because the kernel is too simple to process the heap. It can only adjust the linear zone occupied by the program by adjusting the heap top pointer; however, you can only release the memory by adjusting the linear zone. So as long as the heap top is not reduced, the occupied memory will not be released.

Memory occupied by code
The data part occupies the memory. Does the program we write also occupy the memory?
In linux, program loading involves two tools: linker and loader. Linker mainly involves the use of dynamic link libraries, and loader mainly involves software loading.
1. execute a program in exec.
2. elf is a very popular executable file format. It divides the program running into two segments. One segment is an executable segment, which is read-only and executable; the other segment is the data segment, which can be read/written and cannot be executed.
3. The loader will start and map the code end and data segment to the memory by calling the mmap system, which is actually allocated with virtual memory,Note that the physical memory is not occupied at this time. The kernel allocates physical memory to the program only when the program is executed.
4. loader searches for the link library that the program depends on. First, it checks whether the Link Library is mapped into the memory. If mmap is not used, the code segment and data segment are mapped to the memory, otherwise, it is only added to the address space of the process. In this way, the memory address space of libraries such as glibc is exactly the same.
Therefore, the execution of a 2 M program does not mean that it is allocated with 2 m physical memory, which is related to the amount of code running and the dynamic link library on which it depends.