Linux Memory addressing resolution __linux

1. Memory address

1. Logical address: Each logical address has a segment and an offset component.

2. Linear address: Also known as the virtual address, is a 32-bit unsigned integer that can be used to represent up to 4GB of address, which is worth range from 0x00000000 to 0xFFFFFFFF.

3. Physical Address: For memory chip-level memory unit addressing

The Memory Control unit (MMU) converts a logical address (through a segment map) to a linear address, and then (through a page map) to a physical address. 2. Segmentation in Hardware

1. Segment Selector (segment register)

(1. Including: Index, TI, RPL

(2) The index number (segment number) of the storage segment descriptor in the Segment table, the TI flag is used to indicate whether it is GDT or Ldt table, RPL flag permission.

2. Paragraph descriptor: an entry in the Paragraph descriptor table that represents a piece of information:

(1. Base address): The starting addresses in the middle of a linear address space.

(2. Paragraph boundaries (Limit): That is, segment size, in the virtual address space, the maximum offset that can be used within a segment.)

(3. Paragraph protection attribute: An attribute that represents a segment.) For example, whether the segment can be read out or written, or whether the segment is executed as a program, the privilege level of the segment, and so on.

3. The segment descriptor is placed in the global descriptor GDT, or in the local descriptor Ldt. The maximum number of segment descriptors that can be saved in GDT is 8,191.

4. Segment addressing process (get linear address)

(1. Determine the type of instruction and decide which segment register to use

(2. Read the contents of the register and find the data structure that holds the description information of the segment.) (GDTR, LDTR)

① through the TI logo, judge the operation of the segment is to the Global Segment Description table to find, or to the local section description table to find

② read the address stored in the GDTR or LDTR register to find the first address of the description table

③ from the first address of the description table according to the index number of the record in the segment register, find the nth descriptor, which is the segment information to be found.

(3. Access to the base site

(4. The instruction address is offset to determine whether the length crosses over.

(5. According to the nature of the instruction and the access rights in the paragraph descriptor to determine whether or not ultra vires

(6. Add the base address to the offset address in the instruction to get the actual memory address and complete the addresses mapping.)

5. Performance of protection

(1. Boundary protection

Cross-border judgment: Length parameter

Privileged directives: New GDTR or LDTR registers do not exist with the old instruction compatibility problem, access to their instructions set as privileged instructions, segment register access properties are still as no privileges, both maintain compatibility, but also ensure that the program can not deliberately modify the paragraph description to cross the line.

(2. Rights protection

Determine whether you have access rights based on the protection properties of the segment. For example, writing is not allowed in read-only segments.

System State, user state separation. The latter two-bit RPL in the segment register represents the privilege level required by the requester operation. The DPL field in GDTR or LDTR sets the access rights for the segment. The RPL mark required by the instruction segment should not be less than the level specified by DPL. Privileged directives can only be executed in the system state. segments in 3.Linux

Four segments in 1.linux

All Linux processes running in user state use a pair of identical fields for instruction and data addressing, that is, user code snippets and user data segments; All Linux processes running in the kernel state also use a pair of identical fields for instruction and data addressing, that is, kernel code snippets and kernel data segments.

Paragraph	Base	G	Limit	S	Type	Dpl	D/b	P
User Code Snippets	0x00000000	1	0xFFFFF	1	10	3	1	1
User Data segment	0x00000000	1	0xFFFFF	1	2	3	1	1
Kernel Code Snippets	0x00000000	1	0xFFFFF	1	10	0	1	1
Kernel data segment	0x00000000	1	0xFFFFF	1	2	0	1	1

Note: There are only 4 memory segments in Linux

2.linux segment Mechanism

They all start with 0, meaning that all processes in the user state and kernel state use the same logical address. Because the segment mechanism is to translate the logical address into a linear address, that is, the segmentation mechanism in Linux does not play a practical role, just walk the process, and Linux logical address is equal to the linear address.

3.GDT and Ldt

Each CPU corresponds to a GDT, all the GDT are stored in the cpu_gdt_table array, and all GDT addresses and their size are stored in the CPU_GDT_DESCR array. Each GDT contains 18 segment descriptors, of which 14 are empty.

LDT is not used in Linux basically. 4. Paging in hardware

For efficiency purposes, linear addresses are divided into groups of fixed-length units called pages.

The paging unit divides all ram into fixed-length page frames (also called physical pages), and each page box contains a page, which means that the length of a page box is the same as the length of a page. The page box is part of main memory and is also a storage area. A page is just a block of data that can be stored in any page box or disk. The paging unit converts a linear address into a physical address.

A data structure that maps a linear address to a physical address is called a page table. The page table is placed in main memory and must be properly initialized by the kernel before the paging unit is enabled.

1. Regular paging:

A paging unit handles 4KB of pages. The 32-bit linear address is divided into 3 domains: a directory (up to 10 digits), a page table (median 10 digits), and an offset (minimum 12 digits).

The process for each activity must have a page directory assigned to it. But there's no need to immediately allocate RAM for all the page tables in the process. The physical address of the page directory being used exists in register CR3. The directory field in the linear address determines the entry of the table of contents in the page directory, and the directory entry points to the appropriate page table. The Page table field for the address determines the table entries in the page tables, and the table entry contains the physical address of the page box on which the page is located. The offset field determines the relative position within the page box. Each page contains 4,096 bytes.

Page directories and page tables can be as large as 1024 items, so a page directory can address up to 1024*1024*4096=2^32. This is the same as the 32-bit address expects.

Or the following figure:

2. Extended Paging:

It allows a page box size of 4MB. The catalog field is 10 bits and the offset field is 22 bits.

3. Physical Address Extension (PAE) paging mechanism:

32-bit physical address, the kernel cannot directly address more than 1GB of RAM due to the needs of the user process's linear addressing space. Many servers require more than 4GB of RAM to run thousands of processes. So Intel has increased the number of corrections to 36, and the processor's addressing capability is now 2^36=64GB.

4. Hardware cache:

The cache unit is inserted between the paging unit and memory.

In Linux settings, caching is enabled for all page boxes, and a writeback policy is always used for write operations.

5. Convert Backup buffer tlb:

In addition to hardware caching, a TLB cache is used to speed up the conversion of linear addresses. page pagination in 5.Linux

The role of the Linux paging mechanism: The paging mechanism is done after the segment mechanism, which further converts the linear address to the physical address. Let's take a look at the hardware structure first:

80386 uses a 4K byte size page, and the starting address of each page is divisible by 4 K. Therefore, the early 80386 4GB-byte linear address space was divided into 1M pages, using a two-level table structure.

The first level table in level Two is called the page directory, stored in a 4K-byte page, there are 1K table entries in the page Catalog table, each table entry is 4 bytes, and the 10 bits (22-31) with the highest linear address are used to produce the first-level table index, where the contents of the table entries from the index locate the address of a table in the two-level table. The memory block number where the subordinate page table is located.

The second level table, called a page table, is stored in a 4K byte page that contains a 1K byte table entry, each containing the physical address of a page. The two-level page table is indexed by the median 10-bit (12-21) bit of the linear address, locating the page table entry and obtaining the physical address of the page. The high 20 bits of the page Physical address form the final physical address with the low 12 bits of the linear address. Convert address map using level two page table:

The paging mechanism for 80x86 is enabled by the PG bit in CR0. such as pg=1, enable the paging mechanism, and use the mechanism described in this verse to convert the linear address to physical address. such as pg=0, disable paging mechanism, directly to the previous mechanism generated by the linear address as a physical address to use.

80386 use a 4K byte size page. Each page has 4 K bytes long and is aligned on a 4K byte boundary, i.e. the starting address of each page can be divisible by 4 K (the physical address is the lowest 12 digits 0). Therefore, 80386 4G byte of linear address space, divided into 1G pages, each page has 4 K byte size.

Paging mechanism by putting pages in a linear address space, relocate to the physical address space for administration because the entire 4K byte of each page is mapped as a unit, and each page is aligned to the 4K byte boundary, so the low 12 bits of the linear address are used directly as the lower 12 bits of the physical address.

The conversion of a linear/physical address can extend its meaning to allow a linear address to be marked as invalid rather than actually producing a physical address. There are two situations in which a page may be marked as invalid: one is that the linear address is an address that the operating system does not support, and the second is that in a virtual memory system, the page corresponding to the linear address is stored on disk rather than stored in RAM memory. In the former case, the program must be terminated because it produces an invalid address.

In the latter case, the invalid address is actually a virtual storage management system that requests the operating system, transferring pages stored on disk to physical storage so that the page can be accessed by the program. Because an invalid page is usually associated with a virtual storage system, such an invalid page is often called an inactive page and is identified with a property bit called a presence bit in the page table attribute bit. An unowned page is a page that the program can access, but it is not in primary storage. Access to such a page, the form is abnormal, in fact, through the exception of the processing of pages.

5.1 Page Global Catalog

A page Global catalog table that can contain up to 1024 page catalog entries, with 4 bytes per page catalog entry, which is exactly one page, as shown in the figure:

• The No. 0 digit is the presence of a bit, present flag: If it is set to 1, the page (or page table) is in main memory, and if the flag is 0, the page is not in main memory, and the remaining bits of the table item can be used by the operating system for its own purposes. If the present flag in the page table entry or page catalog entry that is required to perform an address translation is 0, then the paging unit stores the linear address in the control register CR2 and produces an exception number 14th: a page fault. (We'll focus on how Linux uses this field in a later series of blogs).

• 1th Digit is read/write bit, 2nd is user/admin bit, Read/write flag: access rights (Read/write or Read) containing page or page table; User/supervisor flag: The privilege level required to access a page or page table. These two bits provide hardware protection for page catalog entries. When a process with a privilege level of 3 wants to access a page, a page protection check is required, and a process with a privilege level of 0 can bypass page protection, as shown in the figure:

• The 3rd bit is the PWT (Page write-through) bit, which indicates whether the write-through method is written in memory (RAM) and also write cache, which is 1 to express the use of the write-through method

• The 4th bit is the PCD (Page cache Disable) bit, which indicates whether caching is enabled, and this bit is 1 to enable caching.

• The 5th bit is the access bit, accessed flag: When the page directory entry is accessed, a bit = 1. This flag is set whenever the paging unit addresses the corresponding page frame. When a selected page is swapped out, this flag can be used by the operating system. The paging unit never resets the flag, but it must be done by the operating system.

• The 6th-bit dirty flag, which is always 1 for the page global catalog entry.
• The 7th bit is the page size flag, which is only available for pages directory entries. If you set it to 1, the page directory entry refers to a 4MB page, please see the extended paging below.
• The 8th bit is the global flag: Applies only to page table entries. This flag was introduced in Pentium Pro to prevent common pages from refreshing from the TLB cache. This flag only works if the CR4 register has a page global enable (page globalenable, PGE) flag position.

• 9th to 11th-Place is dedicated to the operating system and Linux has no special purpose

5.2 Page Table

Each page directory entry for 80386 points to a page table that contains up to 1024 page entries, each 4 bytes, containing the starting address of the page and information about the page. The starting address of the page is also an integer multiple of 4 K, so the lower 12 bits of the page are also reserved for it, as shown in the figure.

31st to 12th is a 20-bit physical page address, in addition to the 6th position No. 0 to 5th and 9~11 bit of use and page directory entries, 6th bit is unique to the page, when the page involved in the write operation, D-bit was placed 1.

4GB Storage has only one page directory, it has a maximum of 1024 page directory entries, each page directory entry contains 1024 page items, so the memory can be divided into 1024x1024=1m pages. Because each page is 4K bytes, the size of the storage is exactly 4GB.

5.3 Conversion of linear addresses to physical addresses

When accessing an action unit, how to convert a 32-bit linear address determined by a segmented structure into a 32-bit physical address through paging operations. The procedure is shown in the figure.

The first step, CR3 contains the page directory starting address, with 32-bit linear address of the highest 10-bit A31~A22 as the page table of contents of the page directory entry index, multiply it by 4, and the page directory in CR3 add the starting address, to form the corresponding page table address.

The second step is to remove the 32-bit page directory entry from the specified address with a low 12-bit 0, which is the starting address for the page table. Using the A21~A12 bit in the 32-bit linear address as the index of the page in the page table, multiply it by 4, add the starting address of the page table, and form a 32-bit page address.

The third step is to add the a11~a0 as an offset from the page address and add the 32-bit page address to form a 32-bit physical address.

Next, let's take a look at how regular pagination works. We assume that the kernel has given a running process a linear address space range of 0x20000000 to 0X2003FFFF (3GB linear address space is an upper bound, and the user state process simply refers to one subset of them). This space is made up of 64 pages. In fact, we don't have to care about the physical address of the page box that contains the pages. In fact, some of these pages may not even be in main memory. We only focus on the remaining fields in the page table entry.
The top 10 bits of a process's linear address begin. These two addresses are preceded by a 2 followed by 0, so high 10 bits have the same value, i.e. 0x080 or decimal 128. Therefore, the page directory (directory field) for these two addresses points to item 129th of the Process page catalog. The corresponding directory entry must contain the physical address of the page table assigned to the process. If no other linear address is assigned to the process, the remaining 1023 entries in the page directory are filled with 0.

The value of the bit (that is, the value of the table field) ranges from 0 to 0x03f, or decimal from 0 to 63. As a result, only the first 64 table entries in the page table make sense, and the remaining 960 table entries are filled with 0.
You need to read the bytes in the linear address 0x20021406. This address is handled by the paging unit in the following way:
1. The 0x80 of the directory field is used to select the first 0x80 directory entry for the page catalog, which points to the page table related to the page of the process.
2. Table field 0x21 is used to select the first 0x21 table entry for the page table, which points to the page box that contains the page you want.
3. Finally, the Offet field 0x406 is used to read bytes in the target page box that have an offset of 0x406.

If the present flag for the page Table 0x21 table entry is 0, the page is not in main memory; In this case, the paging unit generates a page fault exception at the same time as the linear address translation. Whenever a process attempts to access a linear address that is limited to 0x20000000 to the 0X2003FFFF range, a page-fault exception is generated because the pages table entries are populated with 0, especially their present flags are cleared 0.

Today, Linux employs a common paging model that applies to both 32-bit and 64-bit systems. As we've seen earlier, the Level two page table is sufficient for 32-bit systems, but 64-bit systems require a greater number of paging levels. Until version 2.6.10, Linux uses a three-page paging model. Starting with version 2.6.11, a four-page paging model was used:

As shown in the figure above:

The 4 types of page tables shown in the figure are called:
? Page Global Catalog (page globally directory)
? Page Parent directory (page Upper directory)
? Page Middle directory (page Middle directory)
? Page tables (pages table)

The page global catalog contains the addresses of several pages of parent directories, which in turn contain the addresses of several pages of intermediate directories, and the page's middle directory contains the addresses of several page tables. Each page table entry points to a page box. The linear address is therefore divided into five sections. The number of digits is not shown in the figure because the size of each part is related to the specific computer architecture.
For a 32-bit system with no physical address extensions enabled, the Level two page table is sufficient. In essence, Linux completely cancels the page ancestor directory and the page intermediate catalog fields by making the page ancestor directory bit and the page middle directory all 0. However, the position of the page ancestor directory and the page middle directory in the pointer sequence is retained so that the same code can be used under 32-bit systems and 64-bit systems. The kernel retains a location for the page ancestor directory and the page intermediate directory by setting their page catalog entries to 1 and mapping the entries to a suitable directory entry for the page global catalog.
A 32-bit system with physical Address extensions enabled uses a Level three page table. The Linux page global catalog corresponds to the 80x86 page Catalog pointer table (PDPT), cancels the page superior catalog, the page middle directory corresponds to the 80x86 page catalog, the Linux page table corresponds to the 80x86 page table.
Ultimately, the 64-bit system uses level three or four paging depending on the hardware's partitioning of the bits of the linear address. So why is Linux so keen on paging technology that it's so cold on the staging mechanism because the process processing of Linux relies heavily on paging. In fact, the automatic conversion of linear addresses to physical addresses makes the following design goals feasible:
? Assigning a different physical address space to each process ensures that addressing errors are effectively prevented.
? Differences between pages (that is, a set of data) and page frames (that is, physical addresses in main memory). This allows a page in a page box to be stored on a disk and then mounted in a different page box when reloading the same page later. This is the basic element of the virtual memory mechanism.

Each process has its own page global catalog and its own page table set. When a process switch occurs, Linux saves the contents of the CR3 control register in the descriptor of the previous execution process, and then loads the next descriptor of the process to execute the CR3 register. Therefore, when the new process restarts on the CPU, the paging unit points to a correct set of page tables.

Mapping a linear address to a physical address is a bit complicated, but it's now a mechanical task. The following sections of this chapter enumerate some of the more tedious functions and macros that retrieve the information the kernel needs to find addresses and manage leaf tables; Most of these functions have only one or two rows. You may want to skip this part now, but it's very useful to know how these functions and macros work, because you'll see them often in later chapters of the discussion.

5.4 Linear Address field processing

The following Macros simplify page table processing:

Page_shift

Specifies the number of digits of the offset field, and when used for the 80x86 processor, it returns a value of 12. Because all addresses in the page must be placed in the Offset field, the page size of the 80x86 system is 212 = 4096 bytes. The Page_mask macro produces a value of 0xfffff000 to mask all bits of the offset field.

Pmd_shift

Specifies the total number of digits of the offset and table fields for a linear address, in other words, the logarithm of the area size that the page's middle directory item can map. The Pmd_size macro is used to calculate the size of the area mapped by a separate table entry for the middle of the page, which is the size of a page table. Pmd_mask macros are used to mask all bits of the offset field and the table field. When the PAE is disabled, the pmd_shift produces a value of 22 (12 bits from offset plus 10 bits from the table), Pmd_size produces a value of 222 or 4 MB, and Pmd_mask produces a value of 0xffc00000. Conversely, when the PAE is activated, the pmd_shift produces a value of 21 (12 bits from offset plus 9 bits from the table), Pmd_size produces a value of 221 or 2 MB, and Pmd_mask produces a value of 0xffe00000. Large pages do not use the last page table, so the Large_page_size macro that produces a large page size is equal to Pmd_size (2pmd_shift), while the large_page_ that masks all bits of the Offset field and table field in a large page address MASK macro is equal to Pmd_mask.

Pud_shift
Determines the logarithm of the size of the area that the page's parent catalog item can map. The Pud_size macro is used to calculate the area size that a single table item in the page global catalog can map. Pud_mask macros are used to mask the Offset field, table field, middle Air field, and all bits of the upper air field. On the 80x86 processor, Pud_shift is always equivalent to Pmd_shift, and pud_size is equal to 4MB or 2MB.

Pgdir_shift

Determines the logarithm of the area size that the page Global page catalog item can map. The Pgdir_size macro is used to calculate the size of the area that can be mapped by a single table item in the page global catalog. Pgdir_mask macros are used to shield all bits of offset, table,middle air, and upper air. When PAE is blocked, Pgdir_shift produces a value of 22 (the same value as Pmd_shift and Pud_shift), Pgdir_size produces a value of 222 or 4 MB, and Pgdir_mask produces a value of 0xffc00000. Conversely, when the PAE is activated, Pgdir_shift produces a value of 30 (12-bit offset plus 9-bit table plus 9-bit middle Air), Pgdir_size produces a value of 230 or 1 GB, and Pgdir_mask produces a value of 0xc0000 000.

Ptrs_per_pte, PTRS_PER_PMD, Ptrs_per_pud and PTRS_PER_PGD

Used to calculate the number of table entries in the page table, the middle of the page, the page's parent catalog, and the page Global catalog table. When the PAE is blocked, they produce values of 1024,1,1 and 1024, respectively. When <