Win32 Virtual Memory Management

Manage Virtual Memory in Win32

Randy Kath Microsoft developer network technical team was created in: July January 20, 1993

Click here to open or copy files in the processwalker sample program. The Win32 sample program requires the Microsoft Windows NT environment.

Summary

In the Microsoft Windows NT operating system, if you do not know enough about the functions of each group of functions and their functions, in the Win32 application, it is difficult to decide which function to use or which group of functions to manage the memory. To simplify this problem, this article mainly focuses on Win32 virtual memory management functions: which functions are available and how to use them, and the impact of using them on the operating system. This article will discuss the following topics:

• Reserve, submit, and release virtual memory

• Change Protection on the Virtual Memory Page

• Lock Virtual Memory Page

• Queries the virtual memory of a process.

In Microsoft Developer Network CD, there is a sample program called processwalker, which will appear in this technical article. This example program is useful for exploring the memory address space of a process. It also uses virtual memory functions to implement a list structure that is linked to each other.

　

Overview

This article is one of three related technical articles. These three articles are "manage Virtual Memory in Win32" and "manage memory in Win32-Map File ", and the "heap memory management in Win32" section explain how to manage memory in Win32 programming interface applications. In the overview section of each article, specify the basic memory components in the Win32 programming model, and specify which article you should refer to if you are interested in special fields.

The first version of Microsoft Windows introduces how to manage dynamic memory based on a single global heap and multiple proprietary local heaps, all applications and systems coexist with the global heap, and each application has its own local heaps ). It also provides global and local memory management functions, providing extended features for this new memory management system. Recently, Microsoft C's Runtime (CRT) library was modified to include the following features, that is, using pure CRT functions such as malloc and free to manage windows heaps. Therefore, developers should now make a choice to either learn the new application programming interface (API) provided as part of Windows 3.1 ), either stick to using portable, typical, and familiar CRT functions to manage memory in applications written for Windows 3.1.

As Win32 API content increases, selection opportunities also increase. Win32 provides three additional function groups to manage the application memory: memory- ed file functions, heap memory functions, and virtual memory functions. These new functions do not replace the existing memory management functions in Windows 3.1. On the contrary, they provide new features that allow developers to write memory management for their Win32 applications, days are much easier.

Figure 1. Win32 API provides different levels of memory management for the diversity of application programming.

In short, what 1 sees is that there are six groups of memory management functions in Win32, all of which are designed to be used independently. So which function should you use? To answer this question, we mainly rely on the following two things: What type of memory management you want, and how the functions associated with it are implemented in the operating system. In other words, whether you are creating a large database application, you want to operate on a large sub-set of memory structures, or you are planning some simple dynamic memory structures, for example, the link list or binary tree (Binary Trees ). In both cases, you need to find out which functions are most suitable for your intention and know exactly how much resources are used when using each function.

Table 1 classifies the memory management function groups in Win32 and identifies the behavior of each group described in the three technical articles in this series. In each technical article, the impact of these functions on the system is emphasized by describing the system behavior in response to these functions.

Table 1. memory management functions available in Win32

Memory settings	Affected System Resources	Related Technical Articles
Virtual Memory Functions	Virtual address space of a process System page file System memory Hard Disk Space	"Manage Virtual Memory in Win32"
Memory- ing file functions	Virtual address space of a process System page file Standard file I/O System memory Hard Disk Space	"Manage memory- ing files in Win32"
Heap memory functions	Virtual address space of a process System memory Process heap Resource Structure	"Manage heap memory in Win32"
Global heap memory functions	Heap resource structure of a process	"Manage heap memory in Win32"
Local heap memory functions	Heap resource structure of a process	"Manage heap memory in Win32"
C Runtime Reference Library	Heap resource structure of a process	"Manage heap memory in Win32"

Every technical article discusses the purpose of Win32 functions. For more information about how to manage system memory in a Windows NT operating system, see the "the virtual-Memory Manager in Windows NT" on Microsoft Developer Network CD, win32 and Windows NT articles ).

　

Windows NT Memory System Overview

Windows NT uses a page-based virtual memory system that uses a 32-bit linear address. Internally, the system manages all the memory in the 4096-byte segment called the page. Is the physical memory on each page backed up? /Font> Use pagefile for temporary memory pages, while use disk files for read-only memory pages. You can have up to 16 different page files at the same time. Code, resources, and other read-only data are directly backed up through the files they create.

Windows NT provides an independent, 2 GB user address space for each application (process) in the system. For applications, it seems that there is 2 GB of available memory, instead of considering the actual amount of available physical memory. If an application requires more memory than the available memory, Windows NT meets this requirement. It paging non-critical memory from this and/or other processes) to a page file and release these physical memory pages. As a result, the global heap does not exist in Windows NT. On the contrary, each process has its own 32-bit address space, in which all the memory of the process is allocated, including code, resources, data, DLL (Dynamic Link Library ), and dynamic memory. In fact, the system is still subject to the restrictions of available hardware resources, but it achieves management of available resources unrelated to the application programs in the system.

　

Virtual Memory in Win32

Windows NT distinguishes between memory and address space. Each process is allocated 2 GB of user address space, regardless of the actual amount of physical memory available for the process. In addition, all processes use a linear 32-bit address in the same range, ranging from 20171000016-7fffffff16, regardless of the available memory address. Windows NT is responsible for ing the Memory Page (paging) to the disk and ing back the disk page to the memory at the appropriate time, so that each process can address the memory required by it. Although two processes may attempt to access the memory on the same virtual address at the same time, the Windows NT virtual memory manager describes the memory locations at different physical locations. The two addresses are not necessarily the same as the original virtual addresses. This is the virtual memory.

Because of the existence of virtual memory, an application can manage its own address space without considering the impact on other processes in the system. The memory manager in Windows NT is responsible for checking whether all applications have enough physical memory for effective operations at any given time. Unlike Windows 3.1 or earlier versions, applications in Windows NT do not have to consider sharing system memory with other applications. In addition, even in the application's own address space, they can still share memory with other applications.

One advantage of distinguishing between memory and address space is that it provides applications with the ability to load very large files into memory. A large file is not necessarily read into the memory. Windows NT provides support for applications to retain the address range required for the file. Then, when necessary, the part of the file can be browsed (physically read into the memory ). With the support of virtual memory, the allocation of large segments of dynamic memory can also be achieved.

In earlier versions of Windows, before an application can operate on addresses in memory, the application must first allocate memory. In Windows NT, the address space of each process has been allocated. Whether the memory is associated with the address in this segment of the address space is another problem. Win32 virtual memory management functions provide low-level support for managing the address and memory of processes separately.

All Win32 virtual memory functions are:

• Virtualalloc and virtualfree

• Virtuallock and virtualunlock

• Virtualquery or virtualqueryex

• Virtualprotect or virtualprotectex

For each function, if they correspond to each other, they form a group. Use virtualalloc to allocate memory. Once allocated, use virtualfree to release the memory. Similarly, when a page is locked using virtuallock, you must use virtualunlock to unlock it when it is no longer needed. Virtualquery and virtualprotect have no corresponding functions, but both have complete functions (indicated by the ex extension on the function name ). In this way, they are allowed to be used in other processes except the calling process. However, the calling process requires proper privileges. These functions will be explained in the appropriate context below.

　

Free, retained, and committed virtual memory

At any given time, each address in a process can be considered free, retained, or submitted. At the beginning of a process, all addresses are free, meaning they are free and can be committed to the memory, or reserved for future use. Before any free address can be used, it must first be allocated as reserved or submitted. An access violation exception occurs when you attempt to access a reserved or submitted address ).

All 2 GB addresses in a process are either free to use, reserved for future use, or committed to a specific memory (in use ). Figure 2 describes a hypothetical process that contains free, reserved, and submitted addresses.

Figure 2. A 2 GB virtual memory address space of a process is allocated as a free, reserved, and committed memory location.

　

Reserved address

When the address is retained in a process, no physical memory page is submitted, and, more importantly, no space is reserved in the page file to back up the memory. Additionally, keeping an address range will not guarantee that there will be available physical memory for submission to these addresses in the future. In fact, it only saves a specified free address until it needs to be used, and blocks other requests allocated to this segment of address. If this type of protection is not available, routine operations (routine operations), such as loading a DLL or resource, may occupy the specified address and endanger future use of it.

A retained address is a quick operation. It has nothing to do with the size of the reserved address range. This function is fast regardless of the 1 GB address range or 4 K address range. This is not surprising because no resources are allocated during this operation. This function only enters the Virtual Address Descriptor (VAD) Tree of the process. For more information about vad, see the developer network CD's "The Virtual-Memory Manager in Windows NT" (Technical article, Win32 and Windows NT article)

To retain an address range, use the following code to call the virtualalloc function:

/* Retain the address space of 10 MB */

Lpbase = virtualalloc (null,

10485760,

Mem_reserve,

Page_noaccess );

As shown here, the first parameter, lpaddress, uses a null value to guide the function to retain the address range at a certain convenient location. In addition, a specified address may have been passed, indicating an accurate initial address for the range to be retained. No matter which of the two methods, the return value of the function indicates the starting position of the reserved address range, unless the function cannot complete the request. In this case, the returned value of the virtualalloc function is an error status value.

The second parameter indicates the address range to be allocated by the function. The value size can be any value from one page to 2 GB, but virtualalloc is actually limited to a small range. The minimum value that can be retained is 64 K, and the maximum value that can be retained is the maximum continuous free address space in the process. The reserved address of the request. The result is a 64 K address range. On the contrary, the request range of 2 GB will fail, because there is so much available address space at any given time, it is impossible. (Remember that loading an application also uses part of the initial 2 GB address space .)

Note that Windows NT generates a protection facility (Safeguard) in the address space of each process ). Each process's top 65,536 bytes and low-end 65,536 bytes are permanently retained by the system. These address spaces are reserved as trap stray pointers that attempt to address the memory within the range of %%16-%ffff16 or 7fff%16-7fffffff16. It is not a coincidence that in this range, you only need to ignore the four low bits (two rightmost bytes) of these addresses to easily detect the pointer. Basically, if the four-digit length is 000016 or 7fff16, the pointer is invalid. All other values indicate valid addresses.

In the last two parameters of the virtualalloc function, dwallocationtype and dwprotect are used to determine how to allocate addresses and the protection associated with them. The address can be assigned to the mem_commit or mem_reserve type. Page_readonly, page_readwrite, and page_noaccess are three types of protection that can be applied to virtual memory. No matter what value is passed to this function, the reserved address is always page_noaccess, which is the default value forced by the system. Submitted pages can be read-only, read-write, or inaccessible.

　

Submitted memory

To use a reserved address, the memory must first be submitted to the address. The committed memory to address is similar to the reserved memory. The commit calls virtualalloc and sets the dwallocation parameter to mem_commit during the call. At this time point, the resource is submitted to the address. Each time, the memory can be submitted by page size. The maximum memory size that can be committed depends only on the maximum range of consecutive free or reserved addresses (but the two cannot be combined). You do not need to consider the size of the available physical memory of the system.

When the memory is submitted, the physical pages of the memory are allocated, and the space of this section is kept in a page file. That is to say, submitted memory pages always exist in the form of physical memory pages or page files on the paged disk. When you submit a large block of memory, it is also possible that some or all of its memory does not reside in the physical memory in the initial stage. Some memory pages reside in page files at the beginning until they are accessed. In the system, once the memory page has been submitted, the Virtual Memory Manager treats it like all other memory pages.

In Windows NT virtual memory system, page tables are used to access physical memory pages. Each page table is also a memory page, just like a submitted page. Occasionally, when the memory is submitted, additional pages must be allocated to the page table. Therefore, to submit a page of memory request, you may need to allocate one page to the page table and one page to the requested page, in addition, two pages of space are required in the page file to back up each of these pages. Therefore, the time required for virtualalloc to complete a memory submission request changes greatly, depending on the system status and the size of the Request space.

The following example shows how to submit a specified page of the reserved address in the previous example to a memory page.

/* Submit the memory for the 3rd page address. */

Lppage3 = virtualalloc (lpbase + (2x4096 ),

4096,

Mem_commit,

Page_readwrite );

Note that the lpaddress is not specified as null, but a specific address is specified to accurately indicate which page of the reserved address will be submitted to the Memory Page. In addition, the memory page is initially protected by page_readwrite, rather than page_noaccess in the previous example. The return address of this function is the virtual address of the submitted address on the first page.

　

Release virtual memory

Once an address is assigned as a reserved address or submitted address, virtualfree is the only way to release the addresses, that is, to return them to a free address. Virtualfree can also be used to unsubmit submitted pages and return these addresses to the reserved status. When the address is released, all physical memory and page file spaces related to the address are released. The following example shows how to unsubmit a submitted memory page in the previous example.

/* Release the submitted memory for the 3rd page address. */

Virtualfree (lpbase + (2*4096 ),

4096,

Mem_decommit,

Page_noaccess );

Only submitted addresses can be unsubmitted. It is important to remember this when you need to unsubmit a large range of addresses. For example, if you have a fixed range of addresses, the subset of multiple addresses has been submitted, while the other parts are retained. To retain the addresses in all ranges, the only method is to unsubmit each subaddress subset one by one. If you try to unsubmit the entire range of addresses, it will fail because the reserved addresses cannot be unsubmitted.

In contrast, all addresses in the same range in an operation can be released. Because when an address is released, the status of an address is not important. The following example shows how to release the address in the range of 10 MB that is retained in the first example.

/* Release the address in the range of 10 MB. */

Virtualfree (lpbase,

10485760,

Mem_release,

Page_noaccess );

　

Change the protection of the Virtual Memory Page

Win32 provides the virtualprotect function as a method to protect submitted pages with memory changes. For example, an application can submit the address of a page by page_readwrite and enter the data to the page immediately. Then, the page protection will be changed to page_readonly, which can effectively protect data from being overwritten by any thread in the process. The following example uses the virtualprotect function to make an inaccessible page available.

/* Change page protection to readable/writable. */

Virtualprotect (lpstack + 4096,

4096,

Page_readwrite,

Lpdwoldprot );

The following environment can be considered as the environment where the function is used. An application used to buffer data receives a set of data streams of varying sizes. Based on specific hardware configurations and other software applications, the data stream may be at times) exceeds the capability of the process. To prevent this, the application designs a memory system and submits some memory pages for a buffer at the beginning. Then, the application uses page_noaccess protection to protect the top pages of the memory, so that any request to access the memory will generate an exception. The application also uses an exception handler in the outer code of the Code to handle access conflicts.

When an access conflict occurs, the application can determine whether the buffer has reached its limit. The application responds by changing page protection to page_readwrite, allowing the buffer zone to receive any additional data and continue uninterrupted execution. At the same time, the application loads another thread to slow down the data stream until the buffer is restored to an ideal operation range. When the condition returns to normal, the top page returns to page_noaccess, and the additional thread ends. This situation describes how to use page protection and exception handling programs in Win32 to provide unique memory management opportunities.

　

Lock Virtual Memory Page

A process in Windows NT is called a working group (Working Set) Is to ensure that the process can run smoothly and must be provided in the memory during runtime. At startup, Windows NT allocates a default number of pages to a process, and gradually adjusts the number to optimize the performance of all activated processes in the system. When a process is running (in fact, the thread of a process is running ), windows NT "Work hard" To ensure that the Working Group pages of the process always reside in the physical memory.

Processes in Windows NT are authorized to use the virtuallock and virtualunlock functions to cleverly influence the behavior of the system. Basically, a process can create a specific page to lock it into the working group. However, this does not give the Working Group of the process a free range. It cannot affect the number of pages in the Working Group that comprise it (the system adjusts the Working Group for each process according to specifications), and it cannot control when the Working Group is in memory and when it is out of memory. The number of pages locked by the working group of each process cannot exceed 32. If an application locks the submitted memory page in the Working Group, this may be advantageous because it may force other key pages in the process to be replaced. In that case, the page may be mapped to the disk, and page faults will occur at any time when accessed. Therefore, the process will spend a lot of CPU time, but only map key pages to memory and map out memory.

Please note that locking one page of memory in Win32 does not mean that the page of memory cannot be mapped to the disk by pages. Instead, it means that when the process is running, the locked memory page will be in a piece of physical memory. It is not only possible, but very likely that when a process is in an ideal situation, the entire page of the process's working group will be mapped to the disk. When the work of the Process gets worse, the workgroup page will be immediately mapped back to the memory, including the virtuallocked page.

The following example locks an address to the memory during the process.

/* Lock the Key address (critical addresses) to the memory. */

Virtuallock (maid, 1024 );

Note that the address range locked to the memory in this example is less than one page. It is not necessary that the entire range be in a single page of memory. The final result is that the entire page of memory containing the data in the address is locked to the memory, not just the data of the pointed address is locked to the memory. If the data crosses the page boundary, both pages will be locked.

　

Queries the virtual memory of a process.

Given that the address space of a process is 2 GB, it is difficult to manage all the ranges of addresses without the ability to query address information. Because the addresses themselves represent independent memory, these memory may be submitted or not submitted for query, it's just one thing to read the data structure that saves their statuses. In Windows NT, this structure is a previously mentioned Virtual Address Descriptor tree. Win32 provides the ability to traverse the VAD structure in virtualquery and virtualqueryex functions. Similarly, the ex suffix indicates which function can be called from one process to query another process worker. If the called process has sufficient security privileges to execute this function. The following example is separated from the processwalker sample program:

/* Query the next memory area in the sub-process. */

Virtualqueryex (hchildprocess,

Lpmem,

Lplist,

Sizeof (memory_basic_information ));

The main function of the processwalker application is to traverse the address space of a process, identify each unique address zone of the process, and display specific status information about each zone. It is done by enumerating each area from the bottom of the process to the top. Lpmem is used to indicate the location of each region. At the beginning, it is set to 0. After it is returned from the query of each new region, it increases progressively by the size of the queried region. This process repeats until the lpmem reaches the highest reserved area of the system.

Lplist is a pointer to the memory_basic_information structure, which is filled by the virtualqueryex function. When this function returns, this structure indicates information about the queried region. This structure contains the following members:

Typedef struct _ memory_basic_information {/* MBI */

Pvoid baseaddress;/* Basic address of the Region */

Pvoid allocationbase;/* allocate the basic address */

DWORD allocationprotect;/* initial access protection */

DWORD regionsize;/* The Byte size of the region */

DWORD state;/* submitted, reserved, and free */

DWORD protect;/* Current access protection */

DWORD type;/* Page type */

} Memory_basic_information;

The virtualquery function returns this status information for any connected address area. The function determines the underlying boundary and region size of the region, and the exact status of the addresses in the region. It is used to determine whether the address of the region can be any address in the region. Therefore, if you want to determine how much stack space has been committed at any given time, follow these steps:

1. Obtain the thread environment for the thread in question.
2. Call the virtualquery function to provide the stack pointer address in the thread environment information as the lpmem parameter in the function.

This query returns the size of the committed memory, and returns the address of the base stack in the memory_basic_information structure in the form of regionsize and baseaddres.

The memory area, defined by virtualqueryregions, is a continuous address range with the same protection, type, and basic allocation. Type and protection values have been described before in this technical article. The basic allocation is the lpaddress parameter value. This parameter is used when the entire memory area is allocated for the first time through the virtualalloc function. In the memory_basic_information structure, it is represented as the allocationbase field.

When free addresses become retained or submitted, their basic allocation is determined. From any perspective, the memory area is not static. Once a single page in the reserved address area is submitted, the area is divided into one or more reserved areas and one submitted area. When the page memory status changes, this situation continues. Similarly, when one of multiple submitted pages of page_readwrite is changed to page_readonly protection, this area is also divided into several smaller areas.

　

Summary

The virtual memory management function in Win32 provides the ability to directly manage Virtual Memory in Windows NT. The 2 GB user address space of each process is divided into reserved, committed, or free virtual address memory areas. A zone is defined as a continuous address range. Its Protection, type, and basic allocation of each address are the same. Each zone contains one or more address pages, it can also carry protection and page lock flag status bits.

The virtual memory management function provides applications with the ability to convert the Page Status of a virtual address space. An application can change the memory type from submitted to retained, or change the protection mode from page_readwrite to page_readonly to prevent access to a certain address space. An application can lock one page of memory to the working group so that a process can at least page key memory pages. Virtual Memory functions are considered as low-level functions, which means they are relatively fast, but lack the functions of many advanced functions.