Memory mapping for large file usage

Usually rarely use large file memory mapping, happened to meet such requirements, so the process is recorded, when giving you a primer, because the application is not complicated, there may be no consideration of the place, welcome to exchange.

For some small files, with ordinary file stream can be a good solution, but for large files, such as 2G or more, the file stream is not, so to use the API's memory mapping method, even the memory map, can not map the size of all files at once, so you must take a block map, Handle a small portion at a time.

Let's look at a few functions

CreateFile: Open File

GetFileSize: Get File size

CreateFileMapping: Creating Mappings

MapViewOfFile: Mapping File

See MapViewOfFile's help, his last two parameters all need to be the page granularity of the integer times, the general machine page granularity of 64k (65536 bytes), and we actually operate, is generally not such rules, arbitrary position, arbitrary length is possible, so we have to do some processing.

The task of this example is from a length list (finfolist), read the length value sequentially, and then go to another large file (Fsourcefilename) to read the specified length of the data, if it is a small file, this is good, once read the file stream, and then read it sequentially, Large numbers for large files, you need to constantly change the location of the map to get the data we want.

This example shows that the page granularity is obtained by GetSystemInfo, and then 10 times times the page granularity as a mapped block, in the for loop, the length of the read (Totallen) plus the length to be read, whether within this mapping range (10 times times the page granularity), If you continue to read, if it is exceeded, write down the rest of the data, then remap the next piece of memory, and then merge the remaining data from the record into the newly read data, a little bit around (maybe my idea is too wrapped up), the code listed below.

[Delphi]View Plaincopy

2. Procedure Tgetdatathread.  Dogetdata;
4. Var
6. Ffile_handle:thandle;
8. Ffile_map:thandle;
10. List:tstringlist;
12. P:pchar;
14. I,interval:integer;
16. Begin
18. Try
20. Totallen: = 0;
22. Offset: = 0;
24. Tstream: = Tmemorystream.  Create;
26. Stream: = Tmemorystream.  Create;
28. List: = Tstringlist.  Create;
30. //Get System Information
32. GetSystemInfo (SysInfo);
34. //page allocation granularity size
36. BlockSize: = SysInfo. dwallocationgranularity;
38. //Open File
40. Ffile_handle: = CreateFile (PChar (fsourcefilename), Generic_read,file_share_read,Nil,open_existing,file_  Attribute_normal,0);
42. if Ffile_handle = Invalid_handle_value then Exit;
44. //Get file size
46. FileSize: = GetFileSize (Ffile_handle,nil);
48. //Create Mappings
50. Ffile_map: = CreateFileMapping (Ffile_handle,nil,page_readonly,0,0,Nil);
52. if Ffile_map = 0 then Exit;
54. //Here we have 10 times times blocksize is mapped to a data block, and if the file size is less than 10 times times blocksize, the entire file length is mapped directly
56. if filesize div blocksize > ten Then
58. Readlen: = 10*blocksize
60. Else
62. Readlen: = FileSize;
64. For i: = 0 to finfolist. Count- 1 do
66. begin
68. List.  Delimiter: = ': ';
70. List. Delimitedtext: = Finfolist.  Strings[i];
72. //Get length, I've done the parsing here because I've stored the information for A:B:C this type, so separated by: number
74. Len: = Strtoint (List.  strings[1]);
76. Interval: = Strtoint (List.  strings[2]);
78. if (i = 0) or (Totallen+len >=readlen) Then
80. begin
82. //If the length of the read plus the length that is about to be read is greater than 10 times times blocksize, then we want to keep the contents of the previous mapping to merge with the newly mapped content
84. if i > 0 Then
86. begin
88. Offset: = offset + Readlen;
90. //write temporary stream
92. Tstream.  Write (P^,readlen-totallen);
94. Tstream.  Position: = 0;
96. end;
98. //If the length of the unread data is not enough for one allocation granularity, then the remaining length is mapped directly
100. if Filesize-offset < blocksize Then
102. Readlen: = Filesize-offset;
104. //Map, p is a pointer to the map area
106. //Note here The third parameter, always set to 0, this value is set according to the actual situation
108. P: = PChar (MapViewOfFile (Ffile_map,file_map_read,0,offset,readlen));
110. end;
112. //If there is data in the staging stream, you need to merge
114. if Tstream. Size > 0 Then
116. begin
118. //Copy the temporary stream data.
120. Stream. CopyFrom (Tstream,tstream.  Size);
122. ///Then write new data at the end, merge complete
124. Stream. Write (P^,len-tstream.  Size);
126. Totallen: = Len-tstream.  Size;
128. //Move the position of the pointer, pointing to the beginning of the next data
130. INC (P,len-tstream.  Size);
132. Tstream.  Clear;
134. End
136. Else
138. begin
140. Stream.  Write (P^,len);
142. Totallen: = Totallen + len;
144. INC (P,len);
146. end;
148. Stream.  Position: = 0;
150. //Save the stream as a file
152. Stream.  SaveToFile (IntToStr (i) +'. txt ');
154. Stream.  Clear;
156. end;
158. finally
160. Stream.  Free;
162. Tstream.  Free;
164. CloseHandle (Ffile_handle);
166. CloseHandle (FFILE_MAP);
168. end;
170. End

http://blog.csdn.net/bdmh/article/details/6369250

Memory mapping for large file usage