Use memory ing files in VC ++ to process large files

Post firstCode:

 # Include  "  Stdafx. h  "  
# Include  <  Windows. h  >  
# Include  <  Iostream  >  
  Using     Namespace STD;


  String  Getvalue (  Const     Char  *  Lpbmapaddress,  Const     Char  *  Sname );

  Int  Main ()
{
Handle hfile  = Createfile (
Text (  "  E: \ test.txt  "  ),
Generic_read  |  Generic_write,
  0  ,
  0  ,
Open_existing,
File_attribute_normal,
Null );

  If  (Hfile  = Invalid_handle_value)
{
Cout  <     "  Failed to create file object, error code:  "  <  Getlasterror ()  <  Endl;
  Return     0  ;
}

  ///  /Obtain the file size 
    //  DWORD dwfilesizehigh;
  //  _ Int64 qwfilesize = getfilesize (hfile, & dwfilesizehigh );
  //  Qwfilesize | = (_ int64) dwfilesizehigh) <32 );  
  
  ///  /  
    // DWORD sizeread;
  //  _ Tstring strr;
  //  Char P [max_path] = {0 };
  //  It must use an array of char instead of tchar, because if it is Unicode, tchar is a two-byte wide character,
  //  But readfile can only read one byte and one byte. This will read two single-byte characters into one wide byte,
  //  Then, an error occurs when the two characters are interpreted as a wide byte character!
  //  Bool brestule = readfile (hfile, P, qwfilesize, & sizeread, null );


 //  Create a file ing object  
  Handle hfilemap  =  Createfilemapping (hfile, null, page_readwrite,  0  ,  0  , Null );
  If  (Hfilemap  =  Null)
{
Cout  <  " Failed to Create File ing object. Error code:  "  <  Getlasterror ()  <  Endl;
  Return     0  ;
}

  //  Obtain the system allocation granularity  
  System_info sysinfo;
Getsysteminfo (  & Sysinfo );
DWORD dwgran  =  Sysinfo. dwallocationgranularity;

  //  Get File Size  
  DWORD dwfilesizehigh;
_ Int64 qwfilesize  =  Getfilesize (hfile,  &  Dwfilesizehigh );
Qwfilesize  | =  (_ Int64) dwfilesizehigh)  <    32  );

  //  Close object  
  Closehandle (hfile );

  //  Offset address  
  _ Int64 qwfileoffset  =     0  ;

  //  Block Size 
  DWORD dwblockbytes  =     0  ;
  For  (  Int  I  =  1  ;;  ++  I)
{
  If  (Qwfilesize <=  I  *  Dwgran)
{
Dwblockbytes  =  I  *  Dwgran;
  Break  ;
}
}

  If  (Qwfileoffset  > =     0  )
{
 //  Ing View
  //  If dwnumberofbytestomap is larger than the file size when mapviewoffile () is called, the call will fail,
  //  And the error code is Access denied. Therefore, the last parameter of mapviewoffile cannot use dwblocksize but qwfilesize.
  //  The returned value of mapviewoffile cannot be tchar *. Because Unicode is interpreted as a wide character, an error occurs. Only char * can be used *  
    Char     *  Lpbmapaddress =  (  Char     *  ) Mapviewoffile (hfilemap, file_map_read,  0  ,  0  , Qwfilesize );
  If  (Lpbmapaddress  =  Null)
{
Cout  <  " Failed to map the file. Error code:  "  <  Getlasterror ()  <  Endl;
  Return     0  ;
}

  String  Str  =  Getvalue (lpbmapaddress ,(  "  HKx "  ));
Cout  <  Str. c_str ()  <  Endl;

Getchar ();

  //  Undo file Image  
  Unmapviewoffile (lpbmapaddress );
}

  //  Close file ing object handle  
  Closehandle (hfilemap );

  Return    0  ;
}


  String  Getvalue (  Const     Char  *  Lpbmapaddress,  Const     Char  *  Sname)
{
  String  STR;
 Const     Char  *  P1  =  Strstr (lpbmapaddress, sname );
  If  (Null  ! =  P1)
{
  Const     Char  *  P2  = Strstr (P1, sname );
  If  (P2  ! =  Null)
{
  Const     Char  *  P3  =  Strstr (P2,  "  =  "  );
  If (P3  ! =  Null)
{
  Const     Char  *  P4  =  P3  +     1  ;
Str  =  P4;
}
}
}
 Return  STR;
} 

Introduction

File Operations are applicationsProgramOne of the most basic functions is Win32.
Both APIs and MFC provide functions and classes that support file processing. Commonly Used functions include Win32.
API createfile (), writefile (), readfile (), and cfile classes provided by MFC. In general, these functions can meet the requirements of most scenarios, but for some special application fields, the massive storage needs may be dozens of GB, hundreds of GB, or even several TB, it is obviously not feasible to use the common file processing method. Currently, the operations on such large files are generally handled in the memory ing file mode. This article will discuss this core windows programming technology.

Memory ing file Overview

Memory file ing is also a memory management method for windows. It provides a unified memory management feature that allows applications to access files on disks through memory pointers, the process is like accessing the memory of the loaded file. By using file ing, You can map all or part of the disk file to a region in the virtual address space of the process and directly access the mapped file, you do not need to perform file I/O operations or buffer the file content. Memory file ing is very suitable for managing large-size files.

When memory ing files are used for I/O processing, the system transfers data by page. All internal memory pages are managed by the Virtual Memory Manager, which determines when the Memory Page is paged to the disk, which pages should be released to provide free space for other processes, and how many page spaces each process can have beyond the actually allocated physical memory. Because the Virtual Memory Manager processes all disk I/O in a unified way (reads and writes memory data on pages ), therefore, this optimization enables it to process memory operations at a sufficient speed.

Any actual I/O INTERACTION performed when a memory ing file is used is performed in the memory and accessed in the form of a standard memory address. The periodic paging of disks is also implemented by the operating system in the background, which is completely transparent to applications. This feature of memory ing files provides high benefits when performing disk transaction operations on large files.

It should be noted that during normal paging operations of the system, the memory ing file is not static and will be updated on a regular basis. If the page to be used is currently occupied by a memory ing file, the system releases the page. If the page data is not saved, the system automatically writes data on the page to the disk before releasing the page.

For Windows operating systems that use web virtual storage management, the memory ing file is an extension of its existing memory management components. Applications composed of executable code pages and data pages can be swapped in or out of memory by the operating system as needed. If a page in the memory is no longer needed, the operating system revokes the control of the page's original user and releases the page for other processes to use. The memory will be re-read from the executable files on the disk only when the page becomes a required page again. Similarly, when a process is initialized and started, the pages in the memory are used to store static and dynamic data of the application. Once operations on the application are submitted, these pages will also be backed up to system page files, which is similar to the process in which executable files are used to back up and execute code pages. Figure 1 shows the backup process of the code page and data page on disk storage:

Figure 1
Backup of Process Code pages and data pages on disk storage

Obviously, if you can process code and data pages in the same way, it will undoubtedly improve the execution efficiency of the program, and the use of memory ing files can meet this requirement.

Management of large files

The memory ing file object does not need to undo all views of the memory ing file before closing the object. Before an object is released, all dirty pages are automatically written to the disk. Pass
Closehandle () disables the memory ing file object, but only releases the object. If the memory ing file represents a disk file, you also need to call the standard file I/O function to disable it. When processing large files, the memory ing file represents an excellent advantage. It only consumes a very small amount of physical resources and has little impact on the system. The following describes the general programming process of the memory ing file:

Figure 2
General process of using memory ing files

In some special industries, it is often necessary to deal with giant files with dozens of or even dozens of GB capacity, while a 32-bit process only has 232 of the virtual address space =
4 GB. Obviously, you cannot map all file images at a time. In this case, you can only map all parts of a large file to a small address space in the process in sequence. This requires appropriate changes to the preceding general process:

1) The image starting with the ing file.

2) access the image.

3) cancel this image

4) map a new image starting from a deeper shift in the file.

5) Repeat Step 2 until you access the full file data.