A filestream with extremely fast reading speed!

Recently I have been making my own album software (http://www.tonixsoft.com/ultraalbum/index.php? Lang = CHS) it is slow and depressing to open a large file. I used to use tfilestream to open a file, find the data segment in it, and copy the content to a tmemorystream, the reason for copying it to an independent tmemorystream is that the subsequent processing of a file-type database component must accept the entire tstream as its storage medium, and the entire process is simply unbearable.

There are two reasons for slow speed:
1. When you use tfilestream to open a file, the operating system will generate a memory image for the file after opening the file. If the file is large, the open space and memory copy operations will become extremely slow.
2. Copy part of tfilestream to tmemorystream. This replication process will enable new memory and then replicate it. Of course, if the memory is too large, the replication time will also become longer.

I am determined to write another file reading class for my current problems. Currently, tfastfilestream must be inherited from tstream so that it can be easily combined with other components.

First, we need to solve the problem of slow opening of large files. For this, we can use mapviewoffile () to directly access the file as a memory image. For more information about mapviewoffile (), and file memory image, can refer to this article: http://www.vccode.com/file_show.php? Id = 2409

The method for creating a file image in Delphi is as follows:
Constructor tfastfilestream. Create (const afilename: string );
VaR
Filesizehigh: longword;
Begin
Ffilehandle: = createfile (pchar (afilename), generic_read, file_share_read, nil, open_existing, file_attribute_normal, 0 );
If ffilehandle = invalid_handle_value then begin
Raise fastfilestreamexception. Create ('error when open file ');
End;
Fsize: = getfilesize (ffilehandle, @ filesizehigh );
If fsize = invalid_file_size then begin
Raise fastfilestreamexception. Create ('error when get file size ');
End;
Fmappinghandle: = createfilemapping (ffilehandle, nil, page_readonly, 0, 0, nil );
If fmappinghandle = 0 then begin
Raise fastfilestreamexception. Create ('error when mapping file ');
End;
Fmemory: = mapviewoffile (fmappinghandle, file_map_read, 0, 0 );
If fmemory = nil then begin
Raise fastfilestreamexception. Create ('error when map view of file ');
End;
End;

Finally, the image data is stored in fmemory, and then overwrite the read () method of tstream. When the external needs to obtain the data, let it go to fmemory:
Function tfastfilestream. Read (VAR buffer; count: longint): longint;
Begin
If (fposition> = 0) and (count> = 0) then
Begin
Result: = fsize-fposition;
If result> 0 then
Begin
If result> count then result: = count;
// Move (pointer (longint (fmemory) + fposition) ^, buffer, result );
Copymemory (pointer (@ buffer), pointer (longint (fmemory) + fposition), result );
INC (fposition, result );
Exit;
End;
End;
Result: = 0;
End;
This function is mainly written by imitating the Same Name method in tcustommemorystream, But it is strange that when I use Delphi's memory copy function move (), the program will always access an Invalid Address, so I had to use the API function copymemory.

In addition, the functions to be implemented are as follows:
Function tfastfilestream. getsize (): int64;
Begin
Result: = fsize;
End;

Function tfastfilestream. Seek (const offset: int64; origin: tseekorigin): int64;
Begin
Case ord (origin)
Sofrombeginning: fposition: = offset;
Sofromcurrent: Inc (fposition, offset );
Sofromend: fposition: = fsize + offset;
End;
Result: = fposition;
End;
In this way, a complete file reading mechanism is available.

Due to the complexity, I have not implemented the file storage mechanism. If you are interested, please implement it by yourself.

Next, we need to solve the problem of how to optimize the replication operations of the two streams currently used. The first way we thought was to create a new stream class, when copying data from another stream, it does not open the memory, but points the internal memory pointer to a segment of the data block in the source stream. However, this stream class is available only during the lifetime of the source stream, and the relationship becomes a little messy.

Later, I suddenly thought of another method. In fact, for external classes (that is, the file-type Database Component I used), it only uses methods such as read () and seek () to access data, so I only need to use some spoofing methods to make the internal class return to the external only a certain segment of its internal data.
For my program, after finding the location of the data I want, set a virtual data range for it. during future external access, all return data within the virtual data range. In this way, you only need to perform some transformation on the basis of the original tfastfilestream.

Procedure tfastfilestream. setuseablememory (const startpos: int64; const size: int64 );
Begin
Fuseablestartpos: = startpos;
Fsize: = size;
End;

Function tfastfilestream. Read (VAR buffer; count: longint): longint;
Begin
...
Copymemory (pointer (@ buffer), pointer (longint (fmemory) + fuseablestartpos + fposition), result );
...
End;

Well, the transformation is over so far. Finally, I switched to the newly written filestream class. The speed was amazing. I opened a file of nearly 30 mb, it takes about 40 seconds to use two stream classes. After being changed to the new tfastfilestream, you only need one class. The time is within 5 seconds. Haha, it's really nice!

If you need the complete code for this class, you can write a letter to contact me:
Tonyki [at] citiz.net