File read/write speed test

I. File Read speed

In Linux, reading files is implemented by calling sys_read (FD, Buf, count) by the system. Therefore, to increase the speed, it is the simplest way to call the sys_read encapsulation, for example, read () or fread () is used directly (). Below are some tests in Linux.

First create a 130m data file dd If =/dev/Zero of = data BS = 1024 k count = 130

[Dd If =/dev/Zero of = data BS = 1024 k count = 130 convert and copy files, copy from the input if source to the target address of, each block1024k, A total of 130 copies.]

Compare the time consumed by reading all size files with fread, read, and fgets, respectively.
Size = 130*1024*1024
Char * Buf = new char [size];

The following is the test result (machine Intel (r) Pentium (r) 4 CPU 3.20 GHz, Mem 1G ):
1. fread (BUF, size, 1, FP) read once
Real 0m0. 187 s
User 0m0. 000 s
Sys 0m0. 180 s

2. Read (fdin, (void *) BUF, size) One read
Real 0m0. 187 s
User 0m0. 000 s
Sys 0m0. 184 s

3. Multiple times of fgets (BUF, size, FP), each time 1 K
Real 0m0. 356 s
User 0m0. 136 s
Sys 0m0. 220 s

4. One fgets (BUF, size, FP) read
Real 0m0. 305 S
User 0m0. 072 s
Sys 0m0. 232 s

The simpler the functions (read () and fread () shown above, the faster the speed. Other input encapsulated functions are just for convenience to meet special needs, it is not necessarily faster to read. For 3 and 4, because sys_read () has a judgment on the read size and a while loop, it is not necessary to call fgets () multiple times like 3 to read large files () to read files.

In addition, the time for reading files using MMAP () memory ing is as follows:
Real 0m0. 231 s
User 0m0. 068 s
Sys 0m0. 164 s

It is not faster than direct read (). The explanation on the internet is that the MMAP efficiency can be significantly improved only when frequent read/write operations are required. The following describes how to simulate frequent read/write operations.

Ii. Test the frequent file read/write speed

1. this test simulates frequent file read/write operations, with a data file size of MB. in, each time you read 1 K bytes from it, add 1 simple computation to each byte, and then write it to another file. out.
// Mmapx1.c
# Define size 1024*1024*500
# Deprecision Len 1024
# Include <stdio. h>
Int main ()
{
File * FP1, * fp2;
Char * Buf = new char [Len];
Int I, J;
FP1 = fopen ("data. In", "rb ");
Fp2 = fopen ("data. Out", "WB ");
For (j = 0; j <1024*500; j ++)
{
Fread (BUF, 1024,1, FP1 );
For (I = 0; I <Len; I ++)
Buf [I] ++;
Fwrite (BUF, Len, 1, fp2 );
}
Printf ("OK! /N ");
Fclose (FP1 );
Fclose (fp2 );
}
Time Command test time, each result is different, the machine load is related. The output result is as follows:
Real 19.592 s 18.517 s 18.003 s 20.470 s 20.004 s
USR 2.924 s 2.964 s 3.000 s 2.812 s 2.972 s
Sys 2.472 s 2.360 s 2.344 s 2.652 s 0.396 s

2. If memory ing is used to map the file data. In to the storage area, the frequent read/write operations on the file are avoided, and all the operations are converted to I/O storage read/write operations. The following program uses MMAP ing to add 1 to data. Out in the file data. In.
// Mmapx2.c
# Include <stdio. h>
# Include <fcntl. h>
# Include <string. h>
# Include <sys/Mman. h>
# Include <unistd. h>
# Define size 1024*1024*500
# Deprecision Len 1024
Int main ()
{
Int fdin, fdout;
Struct stat statbuf;
Void * SRC, * DST;
Char * P;
Int I, J;
Fdin = open ("data. In", o_rdonly );
Fdout = open ("data. Out", o_rdwr | o_creat | o_trunc );

If (src = MMAP (0, size, prot_read, map_shared, fdin, 0) = map_failed)
{
Printf ("src map error/N ");
Return-1;
}
Lseek (fdout, size-1, seek_set );
Write (fdout, "/0", 1 );
// Because data. Out is a null file, you must create a hole to make it size to perform the following MMAP

If (DST = MMAP (0, size, prot_read | prot_write, map_shared, fdout, 0) = map_failed)
{
Printf ("DEST map error/N ");
Return-1;
}
Memcpy (DST, SRC, size );
P = (char *) DST;
For (I = 0; I <size/Len; I ++)
{
For (j = 0; j <Len; j ++)
P [J] ++;
P + = Len;
}
Printf ("OK/N ");
Close (fdout );
Return 0;
}
The five running times of the time test are as follows:
Real 9.603 s 8.977 s 9.416 s 9.587 s 9.322 s
USR 2.764 s 2.748 s 2.784 s 2.840 s 2.787 s
Sys 1.516 s 1.384 s 1.384 s 1.224 s 1.276 s

Conclusion: for frequent Io reads and writes, MMAP storage ing can effectively improve the efficiency (20 s to 9 S)

3. another advantage of MMAP memory ing is that you can directly read and write a certain byte in a file, just like operating the storage area. For example, the following program implements data. add 1 for each byte of data in.

// Mmapx3.c
# Include <stdio. h>
# Include <fcntl. h>
# Include <string. h>
# Include <sys/Mman. h>
# Include <unistd. h>
# Define size 1024*1024*500
# Deprecision Len 1024
Int main ()
{
Int FD;
Void * SRC;
Char * P;
Int I, J;
FD = open ("data. In", o_rdwr );
If (src = MMAP (0, size, prot_read | prot_write, map_shared, FD, 0) = map_failed)
{
Printf ("src map error/N ");
Return-1;
}
P = (char *) SRC;
For (I = 0; I <size/Len; I ++)
{
For (j = 0; j <Len; j ++)
P [J] ++;
P + = Len;
}
Printf ("OK/N ");
Close (FD );
}
The five running times of the time test are as follows:
Real 2.820 s 2.828 s 2.856 s 2.818 s 2.889 s
USR 2.624 s 2.624 s 2.676 s 2.648 s 2.584 s
Sys 0.196 s 0.196 s 0.176 s 0.172 s 0.288 s

All the bytes in a large file are changed to 1 and saved in the original file. The MMAP storage ing method can greatly improve the efficiency. This is an effective option for frequent random rewriting of some bytes in a file.