Fragmentation and paging in Linux

Think this article to write sections and paging mechanism is quite clear, reproduced here.

A while ago read the "deep understanding of the Linux kernel" on the memory management of the part of the time spent a lot, but there are many problems are not very clear, recently spent some time to review, here to record their understanding and memory management in Linux some ideas and understanding.

I prefer to understand the development of a technology itself, in short, how this technology has evolved, what technologies exist before this technology, what are the characteristics of these technologies, why they are replaced by current technology, and how the current technology solves the problems of previous technologies. Figuring out this, we can clearly grasp a certain technology. Some of the information in the introduction of a concept directly on the meaning of the concept, the principle of its development process and the principle behind the slightest mention, as if this technology fell from the sky. In this, it is the memory management of the development process to tell today's theme.

First, I have to explain the topic of this article is the segmentation and paging techniques in Linux memory management.

Let's review the history, in the early days of the computer, the program ran directly on the physical memory. In other words, the program accesses the physical address during the run. If this system only runs a program, then as long as this program requires less memory than the machine's physical memory will not be the problem, we do not need to consider the memory management of the trouble, anyway, you a program, so little memory, eat not to eat full that is your thing. However, now the system is to support multi-tasking, multi-process, so the CPU and other hardware utilization will be higher, this time we have to consider how limited physical memory within the system is allocated to multiple programs in a timely and efficient way, this thing itself we call memory management.

Here is an example of an early computer system, memory allocation management, to make it easier for everyone to understand.

Join us There are three programs, the program is three-in-one. Program 1 requires 10M of memory in the process of running, Program 2 requires 100M of memory during operation, while program 3 requires 20M of memory during operation. If the system needs to run programs A and b at the same time, the early memory management process is probably the case, allocating the first 10M of physical memory to a, and the next 10m-110m to B. This method of memory management is relatively straightforward, well, suppose we want to let program C also run at this time, while assuming that our system memory only 128M, obviously follow this method program C because of memory is not able to run. You know that you can use virtual memory technology, the memory space is not enough when the program does not need to use the data exchange to disk space, has reached the purpose of expanding memory space. Let's look at some of the more obvious issues with this memory management approach. As the article first mentioned, it is better to grasp a technology in depth to understand its development process.

1. Process address space cannot be isolated

Since the program accesses physical memory directly, the memory space used by the program at this time is not isolated. For example, as mentioned above, the address space of a is 0-10m, but if there is a piece of code in a that is manipulating the data in the 10m-128m address space, then program B and program C are likely to crash (each program can be the entire address space of the system). So many malicious programs or Trojans can easily break the fast other programs, the security of the system will not be protected, which is not tolerated for users.

2. Low efficiency in memory usage

As mentioned above, if we want to run programs A, B, C at the same time, then the only way is to use virtual memory technology to write some programs temporarily unused data to disk, when needed to read back from disk memory. Here program C to run, will switch A to disk is obviously not possible, because the program needs a continuous address space, program C needs 20M of memory, and a only 10M of space, so you need to swap program B to disk, and B is full of 100M, You can see that in order to run program C we need to write 100M of data from memory to disk, and then when program B needs to run and then read from disk to memory, we know that IO operation is more time-consuming, so this process will be very inefficient.

3. The address of the program operation cannot be determined

Each time the program needs to run, you need to be in the memory of a large enough idle area, and the problem is that the idle location is not deterministic, which will bring some relocation problems, the relocation of the problem is determined to be referenced in the program variables and functions of the address, if you do not understand the child shoes can check the compilation principle of data.

Memory management is nothing more than a way to solve the above three problems, how to isolate the address space of the process, how to improve the efficiency of memory usage, how to solve the problem of relocation of the program?

Here is a quote from the computer industry: "Any problem in a computer system can be solved by introducing an intermediate layer." ”

The current approach to memory management is to introduce the concept of virtual memory between program and physical memory. Virtual memory resides between the program and the physical memory, and the program sees only virtual memory and no longer has direct access to physical memory. Each program has its own independent process address space, which enables process isolation. The process address space here refers to the virtual address. As the name implies since the virtual address, that is, virtual, not the actual existence of the address space.

Since we have added a virtual address between the program and the physical address space, we need to solve how to map from the virtual address to the physical address, because the program will eventually run in physical memory, mainly segmented and paging two techniques.

Segmentation (Segmentation): This approach is one of the first methods that people use, and the basic idea is to map the virtual space of the memory address space required by the program to a
Physical address space.

Segment mapping mechanism

Each program has its own virtual independent process address space, and you can see that the virtual address space for both program A and B begins with 0x00000000. We map two of the same size virtual address space and the actual physical address space one by one, that is, each byte in the virtual address space corresponds to each byte in the actual address space, the mapping process is set by the software mapping mechanism, the actual conversion is done by the hardware.

This segmented mechanism solves the problem of process address space isolation and program address relocation in the 3 issues mentioned at the beginning of the article. Program A and program B have their own independent virtual address space, and the virtual address space is mapped to a non-overlapping physical address space, if program a access to the virtual address space address is not in the scope of 0x00000000-0x00a00000, then the kernel will reject the request, So it solves the problem of isolating the address space. Our application A only need to care about its virtual address space 0x00000000-0x00a00000, and it is mapped to which physical address we do not care, so the program always follow this virtual address space to place variables, code, do not need to relocate.

However, the segmentation mechanism solves the above two problems, which is a great improvement, but it is still powerless for the memory efficiency problem. Because this memory-mapping mechanism is still in the program, you still need to swap the entire program to disk when memory is low, so that memory usage is still inefficient. So, how to calculate the efficient memory use it. In fact, according to the program's local operation Principle, a program in the process of operation, in a certain period of time, only a small amount of data will be used frequently. So we need a little more granular memory partitioning and mapping method, at this time will think of Linux buddy algorithm and slab memory allocation mechanism, haha. Another way to transform virtual addresses into physical addresses is to have a paging mechanism.

Paging mechanism:

Paging mechanism is to divide the memory address space into a number of small fixed-size pages, each page size is determined by memory, as the Linux ext file system divides the disk into several blocks, this is to improve the memory and disk utilization. Imagine the following, if the disk space is divided into n equal parts, the size of each part (a block) is 1M, if I want to store the file on disk is 1K bytes, then the rest of the 999K bytes is not wasted. So the need for more granular disk segmentation, we can set the block smaller, which is of course based on the size of the file to be considered, it seems a bit off-topic, I just want to say that the paging mechanism in memory is very similar to the disk partitioning mechanism in the Ext file system.

The size of the General page in Linux is 4KB, we divide the address space of the process by page, the common data and code page loaded into memory, the infrequently used code and data are saved on disk, we still use an example to illustrate, such as:

Page mappings between process virtual address space, physical address space, and disk

We can see that the virtual address space of process 1 and process 2 is mapped to a discontinuous physical address space (this is very significant, if one day our continuous physical address space is not enough, but there are many disjoint address space, if there is no such technology, our program will not be able to run), They even share a portion of the physical address space, which is shared memory.

The virtual pages of process 1 VP2 and VP3 are swapped to disk, and the Linux kernel generates a page fault exception when the program needs the two pages, and the exception manager reads it into memory.

This is the principle of paging mechanism, of course, the implementation of the paging mechanism in Linux is relatively complex, through the page Global directory, page parent directory, page intermediate directory, page table and other levels of paging mechanism to achieve, but the basic principle of work will not change.

The implementation of paging mechanism requires hardware implementation, this hardware name is called MMU (Memory Management Unit), he is specifically responsible for from the virtual address to the physical address translation, that is, from the virtual page to find the physical page.

Fragmentation and paging in Linux