Memory Management for Windows

1. Basic concept: Physical memory, virtual memory; Physical address, virtual address, logical address; page directory, page table
2.Windows Memory Management
3.CPU-Segment Memory management
4.CPU-page Memory management

First, the basic concept
1. Two Memory concepts
Physical memory: Everyone knows that the memory is plugged into the motherboard. He is fixed, how large the memory bar is, and how large the physical memory is (except for integrated graphics systems). However, if the program runs a lot or the program itself is large, it can lead to a lot of physical memory consumption, and even the exhaustion of physical memory.
Virtual memory: In brief, virtual memory is the partition of a page file on a hard disk, which acts as memory. When the program is running, some resources have not been used or open several programs at the same time to operate only one of the programs, the system does not have to plug all the resources of the program in the physical memory, so the system will be temporarily unused resources in the virtual memory, wait for the need to be transferred out.
2. Three Address concepts
Physical Address: Physical for memory chip-level cell addressing, which corresponds to the address bus for processor and CPU connections.
This concept should be one of the best understanding of these concepts, but it is worth mentioning that although the physical address can be understood directly into the machine on the memory itself, the memory as a large array of bytes from 0 bytes to the maximum empty byte, and then the array is called the physical address, but in fact, This is just a hardware-to-software image, and the memory is not addressed in this way. So, it is "relative to address bus", is more appropriate, but aside from the physical memory addressing mode of consideration, the physical address directly to the physical memory one by one corresponds, is also acceptable. Perhaps the wrong understanding is more favourable to the metaphysical image.
Logical address: Refers to the part of the offset address that is generated by the program that is related to the segment logical. For example, you can read the pointer variable itself (& operation) in the C language pointer programming, which is actually the logical address, which is relative to the address of the data segment of your current process and is not coherent with the absolute physical address. Only in Intel Real mode will the logical address be equal to the physical address (because there is no fragmentation or paging mechanism for the real mode, the CPU does not perform automatic address translation), and the logic is the offset address of the code snippet in Intel protected mode (assuming that the code snippet, the data segment is exactly the same). The application programmer only has to deal with logical addresses, and the segmentation and paging mechanisms are completely transparent to you, and are only covered by system programmers. While the application programmer can manipulate the memory directly, it will only operate on the memory segments assigned to you by the operating system.
For compatibility, Intel retains the ancient memory management of the segment. A logical address is a machine language instruction used to specify an operand or an address of an instruction. In the example above, we say that the 0x08111111 of the connector is the logical address of the assigned address.
--but I'm sorry to say, it seems to go against the Intel Middle-management, the logical address requirements, "a logical address, is a segment identifier plus a specified segment of the relative address offset, expressed as [segment identifier: Intra-paragraph offset], that is, the 0x08111111 in the above example, should be represented as [a code snippet identifier: 0x08111111], so that it is complete "
Linear address (linear addresses) or also called virtual address (Vsan)
Similar to the logical address, it is also an unreal address, if the logical address is the corresponding hardware platform segment management conversion before the address, then the linear address corresponds to the hardware page-like memory of the pre-conversion address.
-------------------------------------------------------------
Each process has a virtual address space of 4GB
This 4GB is part of 3.
(1) Partial mapping of physical memory
(2) A portion of the swap file on the mapped hard disk
(3) Part of what does not do
The program is using a virtual address of 4GB, access to physical memory requires the use of physical address, physical address is placed on the addressing bus address, in bytes (8-bit) units.
-------------------------------------------------------------
The CPU converts an address in a virtual memory space into a physical address, which takes two steps: First, given a logical address (in fact, the offset within the paragraph, this must be understood!!!). ), the CPU uses its segment memory management unit to convert a logical address into a thread address, and then use its page-based memory management unit to convert to the final physical address.
Doing this two conversions is really cumbersome and unnecessary because you can directly draw a linear address to the process. The reason for this redundancy is that Intel is completely compatible.
3. Page table, page directory concept
Using the paging mechanism, the 4G address space is divided into fixed-size pages, each of which is either mapped to physical memory, mapped to a swap file on the hard disk, or not mapped to anything. For the general program, the 4G address space, only a small part of the mapping of physical memory, a large portion of the piece is not mapped anything. Physical memory is also paged to map the address space. For 32bit Win2K, the size of the page is 4K bytes. The information that the CPU uses to convert the virtual address into a physical address is stored in a structure called a page directory and a page table.
Physical memory paging, a physical page with a size of 4K bytes, and a No. 0 physical page starting at the physical address 0x00000000. Since the page size is 4KB, it is 0x1000 bytes, so the 1th page starts at the physical address 0x00001000. The 2nd page starts at the physical address 0x00002000. You can see that the size of the page is 4KB, so you only need to address the physical page in 32bit of high 20bit.
Page Directory: a page directory size of 4K bytes, placed in a physical page. Consists of 1024 4-byte page catalog entries. The size of the page catalog entry is 4 bytes (32bit), so there are 1024 page catalog entries in a page catalog. The contents of each item in the page directory (4 bytes per item) Height 20bit is used to put a page table (page table in a physical page) of the physical address, low 12bit put some flags.
Page table: The size of a page table is 4K bytes, placed in a physical page. Consists of 1024 4-byte page table entries. The size of the page table entry is 4 bytes (32bit), so there are 1024 page table entries in a page table. The contents of each item in the page table (4 bytes per item, 32bit) high 20bit is used to place the physical address of a physical page, and low 12bit puts some flags.
For x86 systems, the physical address of the page directory is placed in the CR3 register of the CPU.
4. Convert virtual address to physical address
A virtual address size of 4 bytes that contains information to locate the physical address
Virtual address divided into 3 parts
(1) 31-22-bit (10-bit) is an index in the page directory
(2) 21-12-bit (10-bit) is an index in a page table
(2) 11-0-bit (12-bit) is in-page offset

Conversion process:
First find the page directory by CR3 the physical page-"according to 31-22 of the virtual address to find the page directory entry-" through the page directory entry to find the virtual address corresponding to the page table address-"Based on virtual address 21-12 to find the physical page physical address-" A 11-0-bit more virtual address as an offset plus the address of the physical page finds the physical address corresponding to the virtual address
The CPU translates the virtual address into a physical address: a virtual address, a size of 4 bytes (32bit), contains information to find the physical address, divided into 3 parts: the 22nd to 31st bits of this 10-bit (highest 10-bit) is the index in the page directory, the 12th-to-21st-bit 10 bits are the indexes in the page table. No. 0 to 11th this 12-bit (low 12-bit) is the in-page offset. For a virtual address to be converted to a physical address, the CPU first finds the physical page where the page directory resides, based on the value in CR3. Then, based on the 10-bit (highest 10bit) value of the virtual address from 22nd to 31st, locate the corresponding page directory entry (PDE, page directory entry), and the page catalog entry has the physical address of the page table that corresponds to the virtual address. With the physical address of the page table, based on the 10-bit value of the 12th to 21st bit of the virtual address as the index, the corresponding page table entry (pte,page table entry) in the page table is found, and the page table entry has the physical address of the physical page corresponding to the virtual address. Finally, the minimum 12 bits of the virtual address, that is, the page offset, plus the physical address of the physical page, the virtual address of the corresponding physical address.
-------------------------------------------------------------
There are 1024 items in a page directory, and the 10bit with the highest virtual address can just index 1024 items (2 of 10 equals 1024). A page table also has 1024 items, the virtual address of the middle part of 10bit, just index 1024. The lowest 12bit of the virtual address (2 of 12 equals 4096), as the in-page offset, can index 4KB, that is, each byte in a physical page.
-------------------------------------------------------------
The computational process of translating a virtual address into a physical address is that the processor finds the physical page of the current page directory by CR3, takes the high 10bit of the virtual address, and then shifts the 10bit right to 2bit (because each page directory entry is 4 bytes long and the right shift 2bit equals 4) to get the address in that page. Take the PDE (4 bytes) out of the address and find the physical page of the virtual address corresponding to the page table, taking the virtual address 12th to 21st place 10 bits, and then put this 10bit to the right 2bit (because each page table entry 4 bytes long, the right to move 2bit equivalent by 4) to get the address in the page, Taking out the PTE (4 bytes) at that address, the address of the physical page corresponding to the virtual address is found, and the physical address is obtained by adding 12bit of the in-page offset.
-------------------------------------------------------------
A pointer to 32bit that can address the range 0X00000000-0XFFFFFFFF,4GB size. This means that a 32bit pointer can address every byte of the entire 4GB addressing space. A page table entry is responsible for 4K of address space and physical memory mapping, a page table 1024, which is responsible for the 1024*4k=4m address space mapping. A page catalog item that corresponds to a page table. A page directory has 1024 items, which corresponds to 1024 page tables, and each page table is responsible for mapping the 4M address space. 1024 page tables are responsible for 1024*4M=4G address space mappings. A process has a single page directory. So page catalogs and page tables can guarantee the mapping of each page and physical memory in the 4G address space.
-------------------------------------------------------------
Each process has its own 4G address space, from 0X00000000-0XFFFFFFFF. Implemented by each process's own set of page catalogs and page tables. Because each process has its own page directory and page table, the physical memory mapped for each process's address space is not the same. The values for the same virtual address of two processes (if all have physical memory mappings) are generally different because they tend to correspond to different physical pages.

The low 2g,0x00000000-0x7fffffff in the 4G address space is the user address space, and the 4G address space is high 2G, that is, 0X80000000-0XFFFFFFFF is the system address space. Access to the system address space requires the program to have RING0 permissions.

Two. Windows Memory principle

The main contents are as follows:
1. Overview
2. Virtual memory
3. Physical memory
4. Mapping

1. Overview:
Windows in our general programming contact is linear address that is what we call the virtual address, but unfortunately in my growing up in the process of discovering that the original linear address is the operating system itself, It's not exactly where our data really exists. In other words, we wrote the string "UESTC" in the place of the 0x80000000 (virtual address), But our string doesn't really exist in the physical address of the 0x80000000 here. Besides, the real physical address is the use of an array of n long to locate (the sum of the words do not understand this is OK, one will see the physical address then you understand). But why does Windows and Linux need to be addressed in this way? The reason is simple for security. Have you heard of the protection mode? As the name implies, this mode incorporates measures to protect the security of the system, as well as the use of linear addresses as one of the so-called security measures.
    We assume that if we don't use a linear address, we can access the physical address directly, but then the operating system cannot check if the memory is writable when we write to the memory. In other words, the operating system cannot implement page access control. This is a horrible thing, like Win9x. In the application state to the kernel address to write things, there is no Justice ~ ~
    due to the security requirements of the operating system, The application of virtual address is spawned. There is something called the MMU (which should be the memory Manage unit Ram Management Unit) in the CPU, specifically responsible for the conversion between the linear address and the physical address. Every time we read and write memory, from the structure of the CPU does not have to go through the ALU, The ALU takes the virtual address and throws it to the MMU to convert it to a physical address before reading the data into the register. That's the way it is.

2. Virtual memory
    We are programming with virtual addresses, with 4G of virtual memory for each process (small additions: 4G virtual memory, high 2G memory is part of the kernel, is common to all processes, low 2G of memory data is process-specific, each process low 2G memory is not the same, but note that the virtual address is the operating system itself, make an analogy that the idea has not been put into practice, so does not constitute any loss of resources. For example, we're going to write a "0x80000000" UESTC ", the operating system will map this virtual address to a piece of physical address A, you write this virtual address is equivalent to write physical address a. But join us only to apply for a 1KB of virtual memory space, not read and write, the system will not allocate any physical memory, The system allocates the appropriate physical space only when virtual memory is to be used.
    below to say some of the details, in fact, the management of virtual memory is a bunch of data structures to achieve, the following gives the structure:
(too lazy to hit so much only play an important part ~ ~)
In Eprocess, there is a data structure as follows:
typedef struct _MADDRESS_SPACE
{
    pmemory_area memoryarearoot;// This pointer to a binary sorting tree, must have learned the data structure of friends know it ~ ~ Hey ~ ~ is mainly in this case, the use of binary sorting tree can speed up the search speed  
    ...
    ...
    ...
}maddress_space, *pmaddress_space;

However, the node structure of this binary sort tree is this:
typedef struct _MEMORY_AREA
{
    PVOID startingaddress; Start address of virtual memory segment
    PVOID endingaddress;  //End address of virtual memory segment
    struct _ Memory_area *parent; The node's dad
    struct _memory_area *leftchild;//The left son of the node
    struct _memory_area * Rrightchild; The left son of the node
    ...
    ...
    ...

}memory_area, *pmemory_area;
    This node mainly records the allocated virtual memory space, if you want to request virtual memory space to run to create a node here, if you want to delete the same node delete the corresponding nodes. But it's easy to say that there are many operations involved For example, to balance this tree or something.
    Then when we allocate the virtual memory space, the system will run to find the tree, and then through a certain algorithm to traverse the tree, find the matching memory space (unallocated memory), create a node to hang on the tree, return to the starting address is complete.

3. Physical memory
    the next to the physical memory of the east, in fact, the management of physical memory is based on an array to manage, I heard the excessive page mechanism, Here's the next page. Under Windows paging is 4KB page then assuming that we have 4GB of physical memory then Windows will page this 4GB space into 4gb/4kb = 1M page     then each page (that is, 4KB) The physical space is managed by a data structure called physical_page, this data structure is not written .... I write the hand acid and look at the people also tired ~ ~ talk about the idea is good.
    then assume that the 4GB memory operating system will produce a physical_page array, how many elements does this array have? There are 1M It just covers the physical address of the 4GB. This is the prototype of the so-called paging mechanism. Plainly, the memory is managed by dividing it by 4k. So the physical address is good location, so-called physical memory address can be directly in the array subscript to represent, in fact, here the physical memory address refers to the physical memory address of the page number ... The specific address is also to be based on the virtual memory address of the low 12-bit and physical memory of the page number together to determine the  
     Repeat the management of the physical memory, there are 3 queues in the kernel The elements in these queues are the physical_page structure   above;
They are:
A. Allocated Memory queue: Store the memory being used
B. Memory queue to be cleaned: Store the memory that has been freed, but the memory is not cleaned (0)
C. Idle queue: Storage of available free memory

The system management process is as follows:
1). Every once in a while, the system automatically extracts the queue elements from the B queue for cleanup and then puts them into the idle queue.
2). Whenever physical memory is released, the system will extract the freed memory from the a queue and put it into the B queue.
3). When requesting memory, the system will allocate memory from the C queue and put it into a queue

4. Mapping
When it comes to mapping, start with the 32-bit address of virtual memory. There is a CR3 register in the CPU that holds the page directory address of each process.

We can split the conversion process into a few steps.
1. According to the value of CR3 (note: The value in CR3 is the physical address) we can navigate to the current process's page directory base address, and then offset by the high 10 bits of the virtual address to obtain the specified PDE (Page directory Entry), the PDE content has 4 bytes, The high 20-bit section is used to make the page table base, and the rest of the bits are used to implement permissions control and so on. The system can realize the memory permission control just by detecting the corresponding bit.
2. By offset from the base address provided by the PDE plus the 10 bits (21-12) of the virtual memory, the page Table Pte (Entry) address is found, and the high 20 bits of the PTE are the base addresses of the physical memory (in fact, the lower label of the physical_page array. ...), the rest of the bits are also used for access control and the like.
3. It is possible to determine the unique physical memory by making a base address from the low 12 bits of virtual memory plus the high 20 bits in Pte.

Memory Management for Windows