Linux Virtual Memory

What is virtual memory?

Take a direct excerpt from Wikipedia on the introduction.

Virtual memory is a technology of computer system memory management. It allows the application to assume that it has contiguous available memory (a contiguous, complete address space), and in fact, it is usually separated into multiple physical memory fragments, and some are temporarily stored on external disk storage and exchanged for data when needed.

For C language variables, we can use the & operator to obtain its address, since it is a virtual address, that is, the address is virtual.

Virtual address mechanism is not necessary, in a simple microcontroller, the code compiled need to specify the physical RAM space distribution, there is no concept of virtual address, the address refers to the physical address in RAM.

Virtual memory, which is known as virtual memory, is relative to the logical and physical memory in the system, and logical memory is the memory that is seen from the process perspective, and therefore is something that the programmer cares about. Physical memory is the memory that is seen at the processor angle and is managed by the operating system. Virtual memory can be said to be a technical means of mapping the memory of these two different perspectives. technology is a method of on-demand dynamic memory allocation, which is taken over by the operating system, which allows the program to unknowingly use storage space larger than the actual physical space (in fact, the storage space required by the program is stored in the form of pages in physical memory and disk), so that virtual memory completely frees the programmer, Since then the programmer does not care about the size and loading of the program, you can freely write programs, tedious things to the operating system to do it.

Why do I need virtual memory?

We know that the program code and data must reside in memory to be able to run, but the amount of system memory is very limited, often can not accommodate a complete program of all the code and data, not to mention in the multi-tasking system, may need to open sub-processing programs, paint programs, browsers and many other tasks, It is obviously unlikely that you want memory to reside in all of these programs. So the first thing you can think of is to split the program into small pieces, only let the current system run it all the parts needed to stay in memory, the other parts are left on the hard disk. When the system has finished processing the current task fragment, then the next task fragment to be run is transferred from external memory . It is true that old-fashioned systems deal with large tasks in this way, and the work is done by programmers themselves. But as the programming language becomes more advanced, programmers are less dependent on the system system, and few programmers are able to control the system very clearly, so letting the programmer take responsibility for the fragmentation of the program and the need to turn it on on demand reduces efficiency, and then causes the machine to crash; Another reason is that as programs become richer, The program's behavior is almost impossible to predict, and it's hard for programmers to judge which program to load next. It is therefore difficult to rely on predictability to statically allocate fixed-size memory and then mechanically rotate the tablet into memory execution. The system must take a new technology that can be allocated on demand without the need for programmer intervention.

X86 's virtual memory technology Gdt/ldt

Both GDT and LDT were introduced into the x86 system at 80286, and the LDT and GDT had similar structures. The LDT appears to serve multiple processes using separate address spaces, typically one LDT per process, while shared memory and kernel memory use GDT. Each program determines the base address based on the segment descriptor, and each Entry has a limit field, which limits the program's access to space. However, after the introduction of more excellent paging technology in 80386, the LDT is basically no longer used.

Page out

paging as the current virtual memory technology implementation , there must be a better place than the LDT, but their implementation of ideas are similar. The operating system maintains a handle for each process, and this handle is associated with the data block from the virtual address to the physical address translation of the process . In the LDT handle is the LDT pointer and length, the data block is the LDT itself. The data block in paging mode is called paging structure, and handler is a pointer to it.

The paging structure has 4096 bytes, which contains a separate entries, each entry in different modes of different sizes. Each entry contains a physical address that can point to a page frame, or to another paging Structur, which is the way it is cascaded. The pointer to the first page structure is in the CR3 register, and then the process from the linear address to the physical address is an iterative process. A part of the linear address is used to indicate the corresponding entry, the entry if it is pointing to another page structure continues until a page frame is pointed to the address translation complete, using the last entry as the base address, and the remaining portion of the linear addresses represents the offset.

This is now devoted to paging in 32bit mode. 32bit each entry 4bytes, each paging structure contains 1024 entries, 10bit is required to differentiate each entry. In fact, the 32bit mode uses a Level 2 pageing structure. The first level is called page Directory, which is distinguished by bits 31-22 with a 32bit linear address, the second level is called a page Table, is distinguished by bits 21-12 with a 32bit linear address, and the remaining Bits 11-0 is exactly what is used to calculate the offset in the 4K page.

In all linear address to Physical address translation, CR3,PDPTE,PDT, Pte and so on are the next base addresses, and the linear address is offset.

Implementing Virtual Memory

Virtual memory is the combination of the system's hard disk space and the system's actual memory for use by the process , providing a much larger virtual space for the process than the memory. As soon as the program runs, a small portion of the virtual address space is stored in memory and the rest is stored on the hard disk (that is, the program virtual space equals the actual physical memory plus some hard disk space). When the virtual address being accessed is not in memory, it means that the address is not stored in the memory, but is storage in the hard disk, so the required virtual storage address is called into memory, and when the system memory is tight, you can also swap out the current unused virtual storage space to the hard disk, to free up physical memory space. The system is running in such a cycle-swapping in, swapping out, and the user is barely able to find out, thanks to the virtual memory mechanism.

Linux the swap partition is the space reserved by the hard disk specifically for virtual storage space. The experience size should be about twice times the amount of memory. If you are interested, you can use swapon-s to view the swap partition size.

The big idea is well understood, nothing more than using memory and hard disk space to synthesize into virtual memory space. However, the iterative translation of addresses (virtual address translation to physical address) and virtual address swapping in this process is worth careful scrutiny. How does the system translate the virtual address to the physical address? How can memory be constantly swapped in and out of the virtual address between the hard disk?

Can the use of paragraph mechanism answer the above questions? Logical address through the paragraph mechanism into a 32-bit address, enough to cover 4G of memory space, when the program needs the virtual address is not in memory, only rely on the segment mechanism is difficult to swap out virtual space, because it is not convenient to the entire size of the virtual space between the memory and the hard disk transfer (old system, will be clumsy to swap out the entire memory and even the whole process, think of this will have the consequences of it! ）。 Therefore, it is necessary to find a smaller and more flexible storage representation unit, so that the virtual address between the hard disk and memory call up. This smaller storage management unit is the page (4K size). The mechanism for managing page swapping in and out is called the page mechanism . (Refer to the section and paging mechanism summary for the page mechanism).

we know that the linear address (which can be interpreted as a logical virtual address) is the size of the space4GB, then how to divide the size of the page is reasonable? IA32The standard page size is4KB, in other words, the linear address space is divided into1Ma4KBthe size of the page. Can you make a big point? Yes, but the corresponding number of pages in memory is reduced, the result is a page is too large, then for a small data, assigning him a block is not a bit wasteful? Otherwise, the data is scattered to different blocks in the memory runtime, it is very laborious to read, so it is clever to set the page size to4KB.

Because of the reason for using the page mechanism, the address obtained through the segment mechanism is simply an intermediate address-the linear address , which does not represent the actual physical address, but rather represents the virtual space address of the entire process. On the basis of the linear address, the page mechanism then processes the linear address mapping: when the required linear address (virtual space address) is not in memory, it is paged into the required virtual memory from disk, and when the memory is not enough, the virtual space in memory is swapped out to disk. Visible, the use of pages to manage memory and disk (virtual memory) greatly facilitates the work of memory management. Undoubtedly, the page mechanism and virtual memory management are simply "perfect".

Linux Virtual Memory