Memory-Mapped large files

For some small files, with ordinary file stream can be a good solution, but for large files, such as 2G or more, the file stream is not, so to use the API's memory mapping method, even the memory map, can not map the size of all files at once, so you must take a block map, Handle a small portion at a time.

Let's look at a few functions

CreateFile: Open File

GetFileSize: Get File size

CreateFileMapping: Creating Mappings

MapViewOfFile: Mapping File

See MapViewOfFile's help, his last two parameters all need to be the page granularity of the integer times, the general machine page granularity of 64k (65536 bytes), and we actually operate, is generally not such rules, arbitrary position, arbitrary length is possible, so we have to do some processing.

The task of this example is from a length list (finfolist), read the length value sequentially, and then go to another large file (Fsourcefilename) to read the specified length of the data, if it is a small file, this is good, once read the file stream, and then read it sequentially, Large numbers for large files, you need to constantly change the location of the map to get the data we want.

This example shows that the page granularity is obtained by GetSystemInfo, and then 10 times times the page granularity as a mapped block, in the for loop, the length of the read (Totallen) plus the length to be read, whether within this mapping range (10 times times the page granularity), If you continue to read, if it is exceeded, write down the rest of the data, then remap the next piece of memory, and then merge the remaining data from the record into the newly read data, a little bit around (maybe my idea is too wrapped up), the code listed below.

Procedure Tgetdatathread.dogetdata;

Var

Ffile_handle:thandle;

Ffile_map:thandle;

List:tstringlist;

P:pchar;

I,interval:integer;

Begin

Try

Totallen: = 0;

Offset: = 0;

Tstream: = tmemorystream.create;

Stream: = Tmemorystream.create;

List: = Tstringlist.create;

Get System Information

GetSystemInfo (SysInfo);

Size of page allocation granularity

BlockSize: = sysinfo.dwallocationgranularity;

Open File

Ffile_handle: = CreateFile (PChar (fsourcefilename), Generic_read,file_share_read,nil,open_existing,file_attribute_ normal,0);

If Ffile_handle = Invalid_handle_value then Exit;

Get File size

FileSize: = GetFileSize (Ffile_handle,nil);

Creating mappings

Ffile_map: = CreateFileMapping (Ffile_handle,nil,page_readonly,0,0,nil);

If Ffile_map = 0 then Exit;

Here we have 10 times times blocksize for a data block to map, if the file size is less than 10 times times blocksize, then directly map the entire file length

If FileSize div blocksize > Ten Then

Readlen: = 10*blocksize

Else

Readlen: = FileSize;

For I: = 0 to Finfolist.count-1 do

Begin

List. Delimiter: = ': ';

List. Delimitedtext: = Finfolist.strings[i];

To get the length, I did the parsing here because I stored the information for A:B:C this type, so separated by: number

Len: = Strtoint (list. STRINGS[1]);

Interval: = Strtoint (list. STRINGS[2]);

if (i = 0) or (Totallen+len >=readlen) Then

Begin

If the length of the read plus the length that is about to be read is greater than 10 times times blocksize, then we want to keep the contents of the previous mapping to merge with the newly mapped content

If i > 0 Then

Begin

Offset: = offset + Readlen;

Write temporary stream

Tstream. Write (P^,readlen-totallen);

Tstream. Position: = 0;

End

If the length of data that has not been read is not enough for one allocation granularity, then the remaining length is mapped directly

If Filesize-offset < blocksize Then

Readlen: = Filesize-offset;

Map, p is a pointer to the map area

Note that the third parameter here is always set to 0, which is set according to the actual situation

P: = PChar (MapViewOfFile (Ffile_map,file_map_read,0,offset,readlen));

End

If there is data in the staging stream, you need to merge

If Tstream. Size > 0 Then

Begin

Copy the temporary stream data.

Stream. CopyFrom (Tstream,tstream. Size);

Then write the new data at the end, merge complete

Stream. Write (P^,len-tstream. Size);

Totallen: = Len-tstream. Size;

Moves the position of the pointer, pointing to the beginning of the next data

INC (P,len-tstream. Size);

Tstream. Clear;

End

Else

Begin

Stream. Write (P^,len);

Totallen: = Totallen + len;

INC (P,len);

End

Stream. Position: = 0;

Save a stream as a file

Stream. SaveToFile (IntToStr (i) + '. txt ');

Stream. Clear;

End

Finally

Stream. Free;

Tstream. Free;

CloseHandle (Ffile_handle);

CloseHandle (FFILE_MAP);

End

End

Http://www.cnblogs.com/hnxxcxg/archive/2012/11/04/2753265.html

Memory-Mapped large files