Windows Memory Management Mechanism and C ++ memory allocation instance (4): Memory ing File

4. Memory Management Mechanism-memory ing file (MAP)
Like the virtual memory, the memory ing file can be used to reserve a process address area. However, unlike the virtual memory, it does not submit physical memory or virtual page files, but files on the hard disk.

· Application scenarios
It has three main purposes:
When the system loads EXE and DLL files, the operating system uses it to load the process of creating EXE and DLL files and run exe. This saves page files and startup time.
To access a big data file, if the file is too large, such as 2 GB in the process user zone, fopen cannot be used to operate the file. Memory ing files are available. For big data files, you do not have to perform I/O operations on the files, and do not have to cache all files.
Process sharing mechanism. The memory ing file is an effective way for multiple processes to share data. It is also the underlying implementation method of the Process Communication Mechanism of the operating system. RPC, COM, Ole, DDE, window message, clipboard, pipeline, socket, and so on are all implemented using memory ing files.

· The system loads EXE and DLL files

EXE file format
Each EXE and DLL file consists of multiple sections. Each section has the following protection attributes: read, write, execute, and shared (which can be shared by multiple processes, disable the copy-on-write attribute of the page ).
The following are common sections and functions:
Node name Function
Code for. Text. EXE and. DLL files
. Data initialized data
. BSS uninitialized data
. Reloc relocation table (loading process address space)
. RDATA read-only data during runtime
. Crt c read-only data during runtime
. Debug debugging information
. Xdata Exception Handling table
Local Storage of. TLS threads
. Idata file input table
. Edata output file table
. Rsrc resource table
. Didata delayed input file name table

Loading Process
1. The system creates the process kernel object, page object, and page table based on the EXE file name, that is, the virtual space of the process.
2. Read the size of the EXE file. Retain the appropriate size on the default base address 0x0040 0000. You can use the/base option to change the base address when linking the program (in VC project properties/linker/advanced settings ). When submitting a file, the operating system manages the page and page tables and maps files on the hard disk to the process space. The address saved in the page table is the page offset of the EXE file.
3. Read the. idata section of the EXE file. This section lists all DLL files used by the exe. Then, map the DLL file to the process space like the EXE file. If it cannot be mapped to the base address, the system locates it again.
4. After the ing is successful, the system loads the first page of code to the memory and updates the page and table. Give the address of the first instruction to the thread instruction pointer. When the system finds that the Code is not in the memory, it loads the code in the EXE file into the memory.

The second loading (running multiple process instances)

1. process Creation and ing process space are the same as above, but when the system finds that this EXE has created a memory ing file object, it is directly mapped to the process space; only when the system allocates a physical page, the page protection attribute is assigned according to the protection attribute of the section, the read attribute is assigned to the Code Section, and the copy-on-write attribute is assigned to the global variable section.

2. Different instances share code sections and other sections. When the instance needs to change the page content, the page content will be copied to the new page to update the page content and page table.

3. For variables that need to be shared by different process instances, the EXE file has a default section to assign the shared attribute to this section.

4. You can also create your own shared section.
# Pragma data_seg ("node name ")
Long instcount;
# Pragma data_seg ()
Then, you need to tell the default attributes of the compiler section when linking the program.
/Section: node name, RWS
Alternatively, use the following expression in the program:
# Pragma comment (linker, "/section: node name, RWS ")
In this case, the compiler will create the. drective section to save the above command, and then use it to change the section attribute during the link.
Note that shared variables may pose security risks because they can read data from other processes.
C ++ program: Examples of variables shared by multiple processes
*. Cpp start:
# Pragma data_seg (". Share ")
Long enough COUNT = 0;
# Pragma data_seg ()
# Pragma comment (linker, "/section:. Share, RWS ")
Distinct count ++;
Note that processes generated by the same EXE file share the same count, and must be in the same location.

· Access big data files

Create a file Kernel Object
Use createfile (file name, access attribute, sharing mode ,...) APIS can be created.
The access attributes include:
0 cannot be read/written (it can be used to access file attributes)
Generic_read
Generic_write
Generic_read | generic_write;

Sharing Mode:
0 exclusive files, which cannot be opened by other applications
File_pai_write
File_pai_read | file_pai_write
This attribute depends on the access attribute and must not conflict with the access attribute.
If creation fails, invalid_handle_value is returned.

The C ++ program is as follows:
Try to open a 1g file:
Memorystatus memstatus;
Globalmemorystatus (& memstatus );
Handle HN = createfile (l "d: // 1g. rmvb", generic_read | generic_write,
File_pai_read | file_share_write, null, open_existing, file_attribute_normal, null );
If (HN = invalid_handle_value)
Cout <"failed to open the file! "<Endl;
File * P = fopen ("D: // 1g. rmvb", "rb ");
If (P = NULL)
Cout <"cannot open large files with fopen! "<Endl;
Memorystatus memstatus2;
Globalmemorystatus (& memstatus2 );
Cout <"space after opening the file:" <Endl;
Cout <"reduce physical memory =" <memstatus. dwAvailPhys-memStatus2.dwAvailPhys <Endl;
Cout <"reduce available page file =" <memstatus. dwAvailPageFile-memStatus2.dwAvailPageFile <Endl;
Cout <"reduce available process space =" <memstatus. dwAvailVirtual-memStatus2.dwAvailVirtual <Endl;
The result is as follows:

It can be seen that the system requires some memory to manage the kernel objects. The results of each run are different, but the difference is not great.
Fopen in C language cannot open such a large file. Theoretically, the 32-bit system supports 232 bytes, but the process space is only 2 GB, which can only represent that large space.

Create a file ing Kernel Object
The API is as follows:
Handle createfilemapping (handle file, psecurity_attributes Security Attribute, DWORD protection attribute, DWORD file size 32-bit, DWORD file size 32-bit, pctstr ing name)
"File" is the handle created above;
"Security Attribute" is required by the kernel object, and null indicates that the system uses the default security attribute. "Protection attribute" is the page attribute required when the memory is submitted to the process space: page_readonly, page_readwrite and page_writecopy. This attribute cannot conflict with the access attribute of the file object. In addition to these three attributes, there are two attributes that can be connected to them for use (| ). When the file content is updated, the cache is not provided, and the file is written directly. sec_nocache is available. When the file is an executable file, the system will assign different page attributes according to the Section and use sec_image. In addition, sec_reserve and sec_commit are used for file ing for sparse submission. For details, refer to the following.
The combination of "32-bit high file size" and "32-bit low file size" tells the system the file size supported by this ing (the operating system supports 264b file size ); when the value is larger than the actual file size, the system will expand the file to this value, because the system needs to ensure that the process space can be fully mapped. If the value is 0, the default value is the file size. If the file size is 0, creation fails.
The ing name identifies this kernel object for sharing by processes. If it is null, it cannot be shared.
If the object fails to be created, null is returned.
After the creation is successful, the system still does not reserve the process space for the file.
C ++ program:
Memorystatus memstatus2;
Globalmemorystatus (& memstatus2 );
Handle hmap = createfilemapping (HN, null, page_readwrite, 0, 0, l "yeming-map ");
If (hmap = NULL)
Cout <"An error occurred while creating the memory ing object! "<Endl;
Memorystatus memstatus3;
Globalmemorystatus (& memstatus3 );
Cout <"space after memory ing file is created:" <Endl;
Cout <"reduce physical memory =" <memstatus2.dwavailphys-memstatus3.dwavailphys <Endl;
Cout <"reduce available page files =" <memstatus2.dwavailpagefile-memstatus3.dwavailpagefile <Endl;
Cout <"reduce available process space =" <memstatus2.dwavailvirtual-memstatus3.dwavailvirtual <Endl;
The result is as follows:

The default memory ing size is 1 GB. No loss of memory and process space. What it does is to create a kernel object and collect some attributes.

File ing Kernel Object ing to process space
The API is as follows:
Pvoid mapviewoffile (handle ing object, DWORD access attribute, DWORD offset high 32-bit, DWORD offset low 32-bit, size_t bytes)
"Ing object" is the previously created object;
"Access attribute" can be the following values: file_map_write (read and write), file_map_read, file_map_all_access (read and write), and file_map_copy. When file_map_copy is used, the system allocates virtual page files. When there is a write operation, the system copies data to these pages and assigns the page_readwrite attribute.
We can see that this type of attribute needs to be set in each step to achieve multi-point control. Imagine that it would be useful if you want to operate different parts of a file with different attributes in this step.
"32-bit offset" and "32-bit offset" are combined to identify the starting byte of the ing (the address is a multiple of the allocation granularity );
The number of bytes indicates the number of bytes mapped. The default value is 0 to the end of the file.
When you need to specify the ing location, you can use:
Pvoid mapviewoffile (handle ing object, DWORD access attribute, DWORD offset high 32-bit, DWORD offset low 32-bit, size_t bytes, pvoid base address)
The "base address" is the first address mapped to the process space and must be a multiple of the allocation granularity.
C ++ program:
Memorystatus memstatus3;
Globalmemorystatus (& memstatus3 );
Lpvoid pmap = mapviewoffile (hmap, file_map_write, 0, 0 );
Cout <"space after memory ing file:" <Endl;
If (pmap = NULL)
Cout <"failed to map process space! "<Endl;
Else
Printf ("first address = % x/N", pmap );
Memorystatus memstatus4;
Globalmemorystatus (& memstatus4 );
Cout <"reduce physical memory =" <memstatus3.dwavailphys-memstatus4.dwavailphys <Endl;
Cout <"reduce available page files =" <memstatus3.dwavailpagefile-memstatus4.dwavailpagefile <Endl;
Cout <"reduce available process space =" <memstatus3.dwavailvirtual-memstatus4.dwavailvirtual <Endl;
The result is as follows:

The process space is reduced by 1 GB, and the system will also open up some memory for file caching.

Use Files
1. You can use multiple ing methods to access large files. A bit like awe technology.
2. windows only ensures data consistency of multiple mappings between the same file and the kernel object. For example, when two mappings between the same object and two process spaces change the data of one process space, the data in another process space also changes. The consistency of multiple mappings between different mapped kernel objects is not guaranteed. Therefore, when using file ing, it is best to set the shared model to 0 exclusive during createfile. Of course, this is not necessary for read-only files.
C ++ program: Use 1g files
Memorystatus memstatus4;
Globalmemorystatus (& memstatus4 );
Cout <"before reading 1g files:" <Endl;
Cout <"available physical memory =" <memstatus4.dwavailphys <Endl;
Cout <"Available page file =" <memstatus4.dwavailpagefile <Endl;
Cout <"available process space =" <memstatus4.dwavailvirtual <Endl;
Int * pint = (int *) pmap;
Cout <"Before change =" <pint [1000001536/4-1] <Endl; // The Last integer of the File
For (INT I = 0; I <1000001536/4-1; I ++)
Pint [I] ++;
Pint [1000001536/4-1] = 10;
Pint [100] = 90;
Pint [101] = 100;
Cout <"after reading 1g file:" <Endl;
Memorystatus memstatus5;
Globalmemorystatus (& memstatus5 );
Cout <"available physical memory =" <memstatus5.dwavailphys <Endl;
Cout <"Available page file =" <memstatus5.dwavailpagefile <Endl;
Cout <"available process space =" <memstatus5.dwavailvirtual <Endl;
The result is as follows:

The program adds 1 integer data of 1g files. It can be seen that the memory loss is more than 600 mb, but sometimes the loss is less than 10 MB, which may be related to the current state of the system.
In any case, you will not be able to see I/O operations at all, just as convenient as accessing a common data structure.

Save file changes
To increase the speed, the file may only be changed to the system cache. In this case, you need to forcibly Save the changes to the hard disk, especially before un ing.
Bool flushviewoffile (pvoid process space address, size_t bytes)
"Process space address" refers to the first byte address to be changed, and the system will change to the page address;
"Number of bytes", the system will become a multiple of the page size.
After writing data to a disk, the function returns. If you want to write data to a network hard disk, you must pass the file_flag_write_through parameter to createfile.
When file_map_copy is used to create a ing, the changes to the data are only modifications to the virtual page file rather than the hard disk file. When the ing is revoked, the modifications will be lost. What should I do if I want to save it?
You can use file_map_write to create another ing, which maps to another space of the process; scan the page_readwrite page of the first ing (because the attribute is changed). If the page changes, use movememory or other copy functions to copy the page content to the second ing space, and then call flushviewoffile. Of course, you need to record which page is changed.

Undo ing
You can use the following API to cancel the ing:
Bool unmapviewoffile (pvoid pvbaseaddress)
This address must be the same as the return value of mapviewoffile.

Disable kernel objects

Release the kernel object as soon as possible to prevent memory leakage. Since they are kernel objects, you can call closehandle (handle.
Close the file handle immediately after createfilemapping;
Immediately disable the memory ing handle after mapviewoffile;
Finally, undo the ing.

· Process Sharing Mechanism

Memory ing Based on Hard Disk Files
If the process needs to share files, as long as the memory ing object is established according to the previous method and then shared by name, the process can map the object to its own process space.
The C ++ program is as follows:
Handle mapyeming = openfilemapping (file_map_write, true, l "yeming-map ");
If (mapyeming = NULL)
Cout <"memory ing object not found: yeming-map! "<Endl;
Memorystatus memstatus3;
Globalmemorystatus (& memstatus3 );
Lpvoid pmap = mapviewoffile (mapyeming, file_map_write, 100000000 );
Cout <"space after memory ing file is created:" <Endl;
If (pmap = NULL)
Cout <"failed to map process space! "<Endl;
Else
Printf ("first address = % x/N", pmap );
Memorystatus memstatus4;
Globalmemorystatus (& memstatus4 );
Cout <"reduce physical memory =" <memstatus3.dwavailphys-memstatus4.dwavailphys <Endl;
Cout <"reduce available page files =" <memstatus3.dwavailpagefile-memstatus4.dwavailpagefile <Endl;
Cout <"reduce available process space =" <memstatus3.dwavailvirtual-memstatus4.dwavailvirtual <Endl;
Int * pint = (int *) pmap;
Cout <pint [100] <Endl;
The result is as follows:

The value of the memory file also changes. Windows ensures that the data mapped from the same memory ing object is consistent. As you can see, 1.exewill change the value from 90to 91,2.exe, because they have a common cache page.

Page-based memory ing
If you only want to share memory data, there is no need to create a hard disk file and then create a ing. You can directly create a ing object:
You only need to pass a file handle invalid_handle_value to createfilemapping. Therefore, when createfile is used, check the return value. Otherwise, a page-based memory ing object will be created. The next step is to map to the process space. At this time, the system will allocate a page file to it.
The C ++ program is as follows:
Handle hpagemap = createfilemapping (invalid_handle_value, null, page_readwrite, 0,00000000, l "yeming-Map-page ");
If (hpagemap = NULL)
Cout <"An error occurred while creating the page-based memory ing object! "<Endl;
Memorystatus memstatus6;
Globalmemorystatus (& memstatus6 );
Cout <"space after page-based memory ing file is created:" <Endl;
Cout <"reduce physical memory =" <memstatus5.dwavailphys-memstatus6.dwavailphys <Endl;
Cout <"reduce available page files =" <memstatus5.dwavailpagefile-memstatus6.dwavailpagefile <Endl;
Cout <"reduce available process space =" <memstatus5.dwavailvirtual-memstatus6.dwavailvirtual <Endl;
Lpvoid ppagemap = mapviewoffile (hpagemap, file_map_write, 0, 0 );
The result is as follows:

It can be seen that, unlike the memory ing based on data files, the MB memory is allocated when the kernel object is created. The advantage is that other processes can share this memory through this kernel object as long as it also maps.

Sparse memory ing File
The workbook program is mentioned in the virtual memory section. The Virtual Memory solves the memory waste problem that indicates that few cells have data but all memory must be allocated. But what if you want to share the workbook structure among multiple processes?
If you use page file-based memory ing, You need to allocate page files first, or waste space, without the advantages of virtual memory.
Windows provides a memory ing mechanism for sparse submission.
When createfilemapping is used to protect attributes from using sec_reserve, it does not submit physical storage. When sec_commit is used, it immediately submits the physical storage. Note: These two parameters can be used only when the file handle is invalid_handle_value.
During normal method ing, the system only retains the process address space and does not submit the physical storage.
The physical memory is submitted only when it is submitted. You can use the common virtualalloc function to submit the physical memory.
When the memory is released, you cannot call the virtualfree function. You can only call unmapviewoffile to cancel the ing and release the memory.
The C ++ program is as follows:
Handle hvirtualmap = createfilemapping (invalid_handle_value, null, page_readwrite | sec_reserve, bytes 0000, l "yeming-Map-virtual ");
If (hpagemap = NULL)
Cout <"failed to create a sparse memory ing object based on page files! "<Endl;
Memorystatus memstatus8;
Globalmemorystatus (& memstatus8 );
Cout <"space after page-based sparse memory ing files are created:" <Endl;
Cout <"reduce physical memory =" <memstatus7.dwavailphys-memstatus8.dwavailphys <Endl;
Cout <"reduce available page files =" <memstatus7.dwavailpagefile-memstatus8.dwavailpagefile <Endl;
Cout <"reduce available process space =" <memstatus7.dwavailvirtual-memstatus8.dwavailvirtual <Endl;
Lpvoid pvirtualmap = mapviewoffile (hvirtualmap, file_map_write, 0, 0 );
Cout <"space after memory ing process:" <Endl;
If (pvirtualmap = NULL)
Cout <"failed to map process space! "<Endl;
Else
Printf ("first address = % x/N", pvirtualmap );
Memorystatus memstatus9;
Globalmemorystatus (& memstatus9 );
Cout <"reduce physical memory =" <memstatus8.dwavailphys-memstatus9.dwavailphys <Endl;
Cout <"reduce available page files =" <memstatus8.dwavailpagefile-memstatus9.dwavailpagefile <Endl;
Cout <"reduce available process space =" <memstatus8.dwavailvirtual-memstatus9.dwavailvirtual <Endl;
The result is as follows:

After sec_reserve is used, only a memory ing object is created, which is the same as normal. The difference is that after ing, a virtual process space is obtained. Now, this space is not allocated with any physical memory. You can use virtualalloc to submit the storage to it. For more information, see the previous article <virtual memory (VM)>.
Note: you cannot use virtualfree to release it. You can only use unmapviewoffile.
The C ++ program is as follows:
Lpvoid pp = virtualalloc (pvirtualmap, 100*1000*1000, mem_commit, page_readwrite );
Memorystatus memstatus10;
Globalmemorystatus (& memstatus10 );
Cout <"reduce physical memory =" <memstatus9.dwavailphys-memstatus10.dwavailphys <Endl;
Cout <"reduce available page files =" <memstatus9.dwavailpagefile-memstatus10.dwavailpagefile <Endl;
Cout <"reduce available process space =" <memstatus9.dwavailvirtual-memstatus10.dwavailvirtual <Endl;
Bool result = virtualfree (PP, 100000000, mem_decommit );
If (! Result)
Cout <"Release failed! "<Endl;
Result = virtualfree (pp, 100000000, mem_release );
If (! Result)
Cout <"Release failed! "<Endl;
Closehandle (hvirtualmap );
Memorystatus memstatus11;
Globalmemorystatus (& memstatus11 );
Cout <"added physical memory =" <memstatus11.dwavailphys-memstatus10.dwavailphys <Endl;
Cout <"add Available page file =" <memstatus11.dwavailpagefile-memstatus10.dwavailpagefile <Endl;
Cout <"add available process space =" <memstatus11.dwavailvirtual-memstatus10.dwavailvirtual <Endl;
Result = unmapviewoffile (pvirtualmap );
If (! Result)
Cout <"An error occurred while canceling the ing! "<Endl;
Memorystatus memstatus12;
Globalmemorystatus (& memstatus12 );
Cout <"added physical memory =" <memstatus12.dwavailphys-memstatus11.dwavailphys <Endl;
Cout <"add Available page file =" <memstatus12.dwavailpagefile-memstatus11.dwavailpagefile <Endl;
Cout <"add available process space =" <memstatus12.dwavailvirtual-memstatus11.dwavailvirtual <Endl;
The result is as follows:

As you can see, virtualfree cannot release this sparse ing. At last, unmapviewoffile can be used to release process space and physical memory.