Information Security System design basics 13th Week Study summary-20135227 Huang

Nineth Chapter Virtual Memory

The main memory is considered as a cache of the address space stored on disk, only the active area is saved in main memory, and the data is transferred back and forth between the disk and main memory as needed, in this way it uses main memory efficiently. In fact, this chapter also explains virtual memory, which is related to memory management in the kernel.

• It provides a consistent address space for each process, simplifying memory management

• It protects the address space of each process from being destroyed by other processes.

• Plays an important role in the design of hardware exceptions, assemblers, linker, loaders, shared objects, files, and processes.
• Memory gives applications the power to create and destroy memory slices, map memory slices to portions of disk files, and share storage with other processes.

I. Physical and virtual addressing

Now the virtual address is started, and when the virtual address is used, the CPU accesses the main memory by generating a virtual address, which is converted to the appropriate physical address before being sent to the memory. The task of translating a virtual address into a physical address is called address translation. Like exception handling, address translation requires close collaboration between the CPU hardware and the operating system. The special hardware called the Memory management unit on the CPU chip uses the query table stored in main memory to dynamically translate the virtual address, and the contents of the table are managed by the operating system.

　　　　　　　　　　

Second, address space

Address space when an ordered set of non-negative integer addresses, if the integer in the address space is contiguous, then we say it is a linear address space. The CPU generates a virtual address from a n=2n address space, which is called a virtual address space. The size of an address space is described by the number of bits required to represent the maximum address. A virtual address space with a n=2n address is called an n-bit address space. A system also has a physical address space that corresponds to the number of bytes of physical memory in the system. Each byte in main memory has a virtual address selected from the virtual address space and a physical location selected from the physical address space.

　　　　　　　　　　　　

Third, virtual memory as a tool for caching

The virtual memory is organized into an array of n contiguous byte-sized cells stored on disk. Each byte has a unique virtual address, and this unique virtual address is used as an index to the array. The contents of the data on disk are cached in main memory. As with other caches in the memory hierarchy, the data on the disk is partitioned into blocks that act as transmission units between the disk and main memory.

Four, virtual memory as a tool for memory management

The operating system provides a separate page table for each process and therefore a separate virtual address space. In, the page table VP1 of process I maps to the PP2,VP2 map to PP7. Similarly, the page table of process J maps VP1 to PP7,VP2 mapping to PP0. Multiple virtual pages can be mapped to the same shared physical page.

　　　　　　　　　　　　

The notation for mapping a contiguous set of virtual pages to any location in any file is called a memory map.

Five, virtual memory as a tool for memory protection

Six, address translation

　　　　　　　　　　　　

Implementation of the translation process, how to use the page table to achieve this mapping. A control register in the CPU, and the page table base Register (PTBR) points to the current page table. The virtual address of the N-bit contains two parts: a virtual page offset for a P-bit and a virtual page number for a (n-p) bit. MMU uses a VPN to select the appropriate PTE (page table entry, a page table is actually an array of multiple page table entries). Page hit, CPU hardware steps to perform:

• The processor generates a virtual address and transmits it to the MMU
• MMU generates a PTE address and obtains it from cache/main memory request
• Cache/Main Memory returns PTE to MMU
• The MMU constructs the physical address and transmits it to the cache/main memory
• Cache/Main memory returns the requested data Word to the processor

Page hits are entirely handled by the hardware, and, unlike this, the processing of pages requires the hardware and operating system cores to work together:

• The first three steps are the same.
• The valid bit in the PTE is 0, so the MMU triggers an exception that passes the control of the CPU to the fault handler in the operating system kernel.
• The page fault handler determines the sacrifice page in the physical memory, and if it is modified, swap it out to disk.
• Page is paged into the new page and updates the PTE in memory
• The fault-pages handler returns to the original process and executes the instruction that caused the missing pages again. The CPU sends the virtual address that caused the missing pages back to the MMU

　　　　　　　　　　　　

The above is the first three steps of the process, the page hit when the processing. If the page does not hit.

　　　　　　　　　　

The overall process will be shown. Finally, the processor sends the virtual address to the MMU again, but this time it does not cause a missing pages exception.

PS: It is known that the Virtual page offset (VPO) and the physical page offset (PPO) size are equal because the virtual page offset is the size of the virtual page, and the size of each amount can be constructed for a given number of virtual address bits and physical address bits. For example, the 32-bit virtual address space and 24 are physical addresses for pages with a size of 1KB. Then the size of VPO and PPO is 10-bit, 22-bit virtual page number, physical page number is 12 bits.

But in today's systems, all are used in conjunction with caching and virtual storage, and most systems choose physical addresses to access the cache.

　　　　　　　　　　

Each time the CPU generates a virtual address, the MMU must consult a PTE to translate the virtual address into a physical address. In the worst case, this would require the data to be taken from the memory at the cost of dozens of to hundreds of cycles at a time. If Pte happens to be in L1, then the overhead drops to 1 or 2 cycles, trying to get to this overhead, they include a small cache of PTEs in the MMU, called the translation fallback buffer (TLB), which is a small one. A cache of virtual addresses in which each row holds a block of a single Pte.

　　　　　　　　

Here's a look at what the process is in the case of TLB hits and misses:
　　　　　　

CPU generates a virtual address

MMU removes the corresponding PTE from the TLB

The MMU translates this virtual address into a house address and sends it to the cache/main memory

Cache/Main memory returns the requested data Word to the CPU.

If the TLB is not hit, the MMU must remove the corresponding PTE from the L1 cache, and the newly fetched PTE is stored in the TLB and may overwrite an existing entry.

Multi-level page table

If we have a 32-bit address space, a 4KB page and a 4-byte Pte (according to this argument, if the 4KB page, then the virtual page offset shift requires 12 bits), then even if the application refers to only a small part of the virtual address space, Page always need a 4MB page table resides in memory (why do you need 4MB of space, because the 32-bit address, 4KB page, may be used 1MB page, PTEs store is the page address, then is the 1MB address, each address needs 4 bytes, Then 4MB of space is required to store an address space).

A common approach is to compress the page table, using a hierarchical page table, assuming that the 32-bit virtual address space is divided into 4KB pages, and each page table entry is 4 bytes. Also assume that at this moment, the virtual address space has the following form: Memory of the first 2K pages assigned to the code and data, the next 6K pages are not allocated, then the next 1023 page page is not allocated, the next 1 pages assigned to the user stack.

　　　　　　　　

Each PTE of the first-level page table is responsible for mapping a 4MB slice in the virtual address space, where each piece is made up of 1024 contiguous pages. Assuming that the address space is 4gb,1024 a PTE is sufficient to cover the entire space.

If each page in the partial I is not assigned, then the first level of Ptei is empty, and the slice 2~7 is unassigned. If at least one page in slice I is assigned, the first level of Ptei points to the base address of a two-level page table. All or parts of the slices 0, 1, and 8 are assigned, so their one-level PTE points to the two-level page table.

Each PTE in a Level two page table is responsible for mapping a 4KB virtual memory page as if we were looking at a page table. Note that using 4-byte PTEs, each level and level two page table is 4KB bytes, which is exactly the same size as a page (why?). 4-Byte Pte, a page table has 1024 page table entries, so a page table is 4KB).

If a PTE in a page table is empty, then the corresponding Level two page table will not exist at all, which represents a huge potential savings; only a single-level page table needs to always exist in main memory; the virtual memory system can create, page, or bring up a Level two page table when needed, which reduces the stress of main memory Only the most frequently used level two page table needs to be cached in main memory.

　　　　　　　　

Address translation using a K-level page table hierarchy is shown. The virtual address is classified as a K VPN and a VPO, and each VPNI is an index to the Level I page table, where 1<=i<=k. Each Pte in the K-level page table contains the address of a physical page PPN or a disk block. Before the PPN can be determined, the MMU must access the K Pte. For a single-level page table structure, PPO and VPO are the same.

Access to K Pte, the first look expensive unrealistic, this TLB can work, is by the different levels of the page table of PTEs cache. In fact, an address translation with a multilevel page table is not much slower than a single-level page table.

Seven, Memory mapping

Storage mapping problems for Linux virtual Storage systems:

Linux maintains a separate virtual address space for each process.

　　　　　　　　　　

Kernel virtual memory contains code and data structures in the kernel. Some areas of the kernel virtual storage are mapped to physical pages shared by all processes. For example, each process shares the kernel's code and global data structures. A Linux page maps a contiguous set of virtual pages to the corresponding set of contiguous physical pages. This provides a traversal method for the kernel to access any particular location in the physical memory, for example, when it needs to access the page table, or perform memory-mapped I/O operations on some devices that are mapped to a specific physical storage location.

Other areas of the kernel virtual storage contain data that is not the same for each process. For example, a page table, a stack used by the kernel to execute code in the context of a process, and a variety of data structures that record the current organization of the virtual address space.

The kernel data structure that records the virtual memory area in a process is emphasized. The kernel maintains a separate task structure for each process in the system (task_struct in the source code). The elements in the task structure contain or point to all the information that the kernel needs to run the process (for example, a PID, a pointer to the user stack, the name of the executable target file, and the program counter)

　　　　　　　　

A bar code in task_struct points to Mm_struct, which describes the current state of the virtual memory. Where PGD points to the base address of the first-level page table (the page global catalog). and mmap points to a list of vm_area_structs, where each vm_area_structs describes an area of the current virtual address space. When the kernel runs this process, it stores the PGDF in the CR3 control register.

• Vm_start: Point at the beginning of this area
• Vm_end: Point at the end of this area
• Vm_port: Describes read-write and permissions for all pages contained within this area
• Vm_flags: Describes whether pages in this area are shared with other processes, or whether the process is private
• Vm_next: Points to the next area structure in the list.

PS: Virtual memory is an abstraction of main memory. Processors that support virtual memory refer to main memory by using an indirect form called virtual addressing. The processor generates a virtual address that is translated into a physical address before it is sent to main memory. Translating from virtual address space to physical address requires hardware and software to work closely together. Specialized hardware uses page tables to translate virtual addresses, and the contents of page tables are provided by the operating system.

Eight Memory Mapping

1. Linux initializes the contents of this virtual memory area by associating a virtual memory area with an object on a disk, a process known as memory mapping. The virtual memory area can be mapped to two types of objects:

1) Common Files in UNIX file systems: a region can be mapped to a contiguous portion of a regular disk file.

2) Anonymous file: An area can also be mapped to an anonymous file, the anonymous file is created by the kernel, and contains all binary zeros.

Once a virtual page is initialized, it is swapped between a dedicated swap file maintained by the kernel (swap files).

2. Shared objects

An object can be mapped to an area of the virtual storage, either as a shared object, or as a private object

　　　　　　　　

3. Private object, copy on write (Copy-on-write)

Private objects are mapped to virtual memory using a technique called write-time copying. As shown in: Whenever a process tries to write a page in a private zone, the write action triggers a protection policy. It will create a new copy of the page in the physical memory, update the page entry to point to the new copy, and then restore the writable permissions on the page.

　　　　　　　　

4. UNIX processes can use the MMAP function to create new virtual memory regions and map objects to these zones.

Nine Dynamic Memory allocation

1. When additional virtual memory is required, a dynamic memory allocator (allocator) is used. A dynamic memory allocator maintains a virtual memory area of a process, called a heap. In most Unix systems, the heap is a region that requests binary 0, and for each process the kernel maintains a variable brk that points to the top of the heap.

　　　　　　　　

2. The allocator maintains the heap as a set of blocks of different sizes. Each block is a contiguous set of virtual memory blocks (chunk), either assigned or unassigned.

1) Explicit allocator (explicit allocator): Allocates and frees a block by new,delete through Malloc,free or C + +.

2) Implicit splitter (implicit allocator): Also known as the garbage collector (garbage collector). The process of automatically releasing unused allocated blocks is called garbage collection (garbage collection).

3. malloc does not initialize the memory it returns, Calloc is a malloc-based wrapper (wrapper) function that initializes the allocated memory to 0. To change the size of a previously allocated block, you can use the ReAlloc function.

4. The allocator must align blocks so that they can hold any type of data object. In most systems, the 8-byte boundary is aligned.

Do not modify an allocated block: The allocator can only manipulate or change the free block. Once assigned, it is not allowed to modify or move it.

5. Fragments (fragmentation)

There are internal fragments (internal) and external fragments (external).

External fragment: Occurs when an allocated block is more than the payload. (such as alignment requirements, assigning minimum limits, etc.)

External fragmentation: Occurs when the free memory is aggregated enough to satisfy an allocation request, but there is not a single free block large enough to handle the request.

6. Implicit spatial chain list

　　　　　　　　　　

　　　　　　　　　　　　

The strategies for placing allocated blocks are: First fit, next adaptation (next fit), and best fit.

If the free block has been combined to the maximum extent and still cannot generate a large enough block to satisfy the requirement, the allocator will request additional heap storage to the kernel either by calling Nmap or by calling the SBRK function, which will convert the extra (added) memory into a large free block. Insert the block into the free list, and then place the requested block in the new free block.

7, a popular method to reduce the time allotted, called the separation of storage (segregated storage), maintain a number of idle linked lists, where the blocks in each list have roughly equal size.

10 Garbage Collection

1. The garbage collector treats the memory as an accessible graph (reachability graph).

　　　　　　　　

2. The Mark%sweep garbage collector consists of the mark stage and the Purge (sweep) stage. The mark phase marks out all the accessible and assigned successors of the root node, while the subsequent purge phase releases each tagged allocated block. Typically, one of the idle lows in the block head indicates whether the block is marked.

　　　　　　　　　　

3. The arithmetic operations of pointers are made in units of the size of the objects they point to.

11 Common Errors

1. Read Uninitialized memory

2. Allow stack buffer overflow

　　

3. Assume that the pointer and the object they point to are of the same size

　　　　

4. Referencing bad pointers

5. Error caused by dislocation

6. Refer to the pointer instead of the object it points to

Information Security System design basics 13th Week Study summary-20135227 Huang