Memory ing large files

For some small files, the use of common file streams can be a good solution, but for large files, such as 2 GB or more, the file stream will not work, therefore, you need to use the memory ing method of the API. Even the memory ing cannot map the size of all files at a time. Therefore, you must adopt block ing to process a small part each time.

Let's look at several functions.

Createfile: open the file

Getfilesize: get the file size

Createfilemapping: Creates a ing.

Mapviewoffile: ing File

Looking at the help of mapviewoffile, the last two parameters of mapviewoffile need to be an integer multiple of the page granularity. Generally, the page granularity of the machine is 64 KB (65536 bytes). In actual operations, this is generally not the case. It is possible to have any length at any position, so some processing is required.

In this example, the task reads the length value from a length list (finfolist) sequentially, and then reads the data of the specified length in sequence in another large file (fsourcefilename, if it is a small file, this is easy to do. Read the file stream once and then read it in sequence. For large files, you need to constantly change the ing position, to obtain the data we want.

In this example, getsysteminfo is used to obtain the page granularity, and a ing data block with a page granularity of 10 times. In the for loop, the read length (totallen) is determined) add the length to be read, and determine whether the data is within the ing range (10 times the page granularity). If the data is read, record the remaining data, re-map the next memory and merge the remaining data in the record into the newly read data. This is a bit confusing (maybe my idea is too messy), which is listed below.Code.

Procedure tgetdatathread. dogetdata;

VaR

Ffile_handle: thandle;

Ffile_map: thandle;

List: tstringlist;

P: pchar;

I, interval: integer;

Begin

Try

Totallen: = 0;

Offset: = 0;

Tstream: = tmemorystream. Create;

Stream: = tmemorystream. Create;

List: = tstringlist. Create;

// Obtain system information

Getsysteminfo (sysinfo );

// Page allocation granularity

Blocksize: = sysinfo. dwallocationgranularity;

// Open the file

Ffile_handle: = createfile (pchar (fsourcefilename), generic_read, file_share_read, nil, open_existing, file_attribute_normal, 0 );

If ffile_handle = invalid_handle_value then exit;

// Obtain the file size

Filesize: = getfilesize (ffile_handle, nil );

// Create a ing

Ffile_map: = createfilemapping (ffile_handle, nil, page_readonly, 0, 0, nil );

If ffile_map = 0 Then exit;

// We have 10 times blocksize as a data block for ing. If the file size is smaller than 10 times blocksize, the entire file length will be mapped directly.

If filesize Div blocksize> 10 then

Readlen: = 10 * blocksize

Else

Readlen: = filesize;

For I: = 0 to finfolist. Count-1 do

Begin

List. delimiter: = ':';

List. delimitedtext: = finfolist. Strings [I];

// Get the length. I have resolved it here, because the information I store is A: B: C, so it is separated:

Len: = strtoint (list. Strings [1]);

Interval: = strtoint (list. Strings [2]);

If (I = 0) or (totallen + Len> = readlen) then

Begin

// If the read Length plus the block size to be read is greater than 10 times, we need to retain the content at the end of the previous ing to merge with the new ing content.

If I> 0 then

Begin

Offset: = offset + readlen;

// Write a temporary stream

Tstream. Write (P ^, readlen-totallen );

Tstream. Position: = 0;

End;

// If the length of the unread data is not enough allocation granularity, the remaining length will be mapped directly.

If filesize-offset <blocksize then

Readlen: = filesize-offset;

// Ing. P is a pointer to the ing area.

// Note that the third parameter is always set to 0. This value must be set according to the actual situation.

P: = pchar (mapviewoffile (ffile_map, file_map_read, 0, offset, readlen ));

End;

// If data exists in the temporary stream, merge the data.

If tstream. size> 0 then

Begin

// Copy the temporary Stream Data

Stream. copyfrom (tstream, tstream. size );

// Write new data at the end and merge the data.

Stream. Write (P ^, len-tstream.Size );

Totallen: = len-tstream.Size;

// Move the pointer to the start of the next data

INC (p, len-tstream.Size );

Tstream. Clear;

End

Else

Begin

Stream. Write (P ^, Len );

Totallen: = totallen + Len;

INC (p, Len );

End;

Stream. Position: = 0;

// Save the stream as a file

Stream.savetofile(inttostr(imo-{'.txt ');

Stream. Clear;

End;

Finally

Stream. Free;

Tstream. Free;

Closehandle (ffile_handle );

Closehandle (ffile_map );

End;

End;