Find the fastest way to read files in C ++

Reprinted from: http://www.byvoid.com/blog/fast-readfile/

 

 

 

During the competition, when encountering big data, reading files often becomes the bottleneck of program running speed and requires faster reading. I believe that almost all c ++ learners are stuck at the slow speed of the CIN machine, so I vowed not to use the CIN to read data. It is also said that the speed of Pascal's read statement is not comparable to that of scanf in C/C ++, and C ++ players can only be anxious. Is C ++ really inferior to Pascal? The answer is self-evident. An advanced method is to read the data and then convert the string. This method is a good method, but I have never tried it, so today, we simply tested all the methods we could think of to read data, and the results were amazing.

In the competition, the most recent data reading experience could not be reached. I wrote a program to generate 10 million random numbers to data.txt, a total of 55 MB. Then I wrote a program trunk to calculate the running time. The Code is as follows:

? View code CPP

1234567

#include <ctime>int main(){int start = clock();//DO SOMETHINGprintf("%.3lf/n",double(clock()-start)/CLOCKS_PER_SEC);}

The simplest method is to write a circular scanf. The Code is as follows:

? View code CPP

12345678910

const int MAXN = 10000000; int numbers[MAXN]; void scanf_read(){freopen("data.txt","r",stdin);for (int i=0;i<MAXN;i++)scanf("%d",&numbers[i]);}

But how efficient? The test result on my computer Linux platform is 2.01 seconds. The following is the CIN code:

? View code CPP

12345678910

const int MAXN = 10000000; int numbers[MAXN]; void cin_read(){freopen("data.txt","r",stdin);for (int i=0;i<MAXN;i++)std::cin >> numbers[i];}

Unexpectedly, CIN took only 6.38 seconds, faster than I thought. There is a reason for the slow cin. By default, CIN and stdin are always synchronized. That is to say, these two methods can be mixed without worrying about file pointer confusion, the same is true for cout and stdout. This compatibility feature causes many additional overhead for CIN. How can I disable this feature? You only need one statement STD: IOS: sync_with_stdio (false); To cancel synchronization of CIN to stdin. The procedure is as follows:

? View code CPP

1234567891011

const int MAXN = 10000000; int numbers[MAXN]; void cin_read_nosync(){freopen("data.txt","r",stdin);std::ios::sync_with_stdio(false);for (int i=0;i<MAXN;i++)std::cin >> numbers[i];}

What is the efficiency after canceling synchronization? The test run time dropped to 2.05 seconds,The efficiency is almost the same as that of scanf.! With this, you can safely use CIN and cout.

Next, let's test the method of reading the entire file and then processing it. First, we need to write a function that converts a string to an array. The Code is as follows:

? View code CPP

12345678910111213

const int MAXS = 60*1024*1024;char buf[MAXS]; void analyse(char *buf,int len = MAXS){int i;numbers[i=0]=0;for (char *p=buf;*p && p-buf<len;p++)if (*p == ' ')numbers[++i]=0;elsenumbers[i] = numbers[i] * 10 + *p - '0';}

The most common method to read the entire file into a string is to use fread. The Code is as follows:

? View code CPP

12345678910111213

const int MAXN = 10000000;const int MAXS = 60*1024*1024; int numbers[MAXN];char buf[MAXS]; void fread_analyse(){freopen("data.txt","rb",stdin);int len = fread(buf,1,MAXS,stdin);buf[len] = '/0';analyse(buf,len);}

The above code has an amazing efficiency. It took only 10000000 seconds to read the 0.29 data records, which has improved the efficiency by almost 10 times! It is absolutely invincible to master these methods. However, I remember that fread is encapsulated with read. Is it faster to directly use read? The Code is as follows:

? View code CPP

12345678910111213

const int MAXN = 10000000;const int MAXS = 60*1024*1024; int numbers[MAXN];char buf[MAXS]; void read_analyse(){int fd = open("data.txt",O_RDONLY);int len = read(fd,buf,MAXS);buf[len] = '/0';analyse(buf,len);}

The test shows that the running time is still 0.29 seconds. Therefore, read does not have any special advantages. Is it over now? No, I can call the underlying function MMAP in Linux. The function maps files to the memory, which is the basic method to encapsulate all file reading methods, what if I directly use MMAP? The Code is as follows:

? View code CPP

123456789101112

const int MAXN = 10000000;const int MAXS = 60*1024*1024; int numbers[MAXN];char buf[MAXS];void mmap_analyse(){int fd = open("data.txt",O_RDONLY);int len = lseek(fd,0,SEEK_END);char *mbuf = (char *) mmap(NULL,len,PROT_READ,MAP_PRIVATE,fd,0);analyse(mbuf,len);}

After testing, the running time was shortened to 0.25 seconds, and the efficiency was improved by 14%. So far, I have no better way to continue improving the speed of reading files. How is Pascal's speed? ResultAstonishingIt runs for as many as 2.16 seconds. The procedure is as follows:

? View code Pascal

1234567891011

constMAXN = 10000000;varnumbers :array[0..MAXN] of longint;i :longint;beginassign(input,'data.txt');reset(input);for i:=0 to MAXN doread(numbers[i]);end.

To ensure accuracy, I switched to the Windows platform for a test. The results are as follows:

Method/platform/time (seconds)	Linux gcc	Windows mingw	Windows vc2008
Scanf	2.010	3.704	3.425
CIN	6.380	64.003	19.208
Cin cancel synchronization	2.050	6.004	19.616
Fread	0.290	0.241	0.304
Read	0.290	0.398	Not Supported
MMAP	0.250	Not Supported	Not Supported
Pascal read	2.160	4.668

Several problems can be seen from the above

1. Programs running on Linux platforms are generally faster than those running on Windows.
2. In Windows, VC-compiled programs generally run faster than mingw (minimal GCC for Windows.
3. Whether VC cancels synchronization of CINInsensitive, With the same efficiency. In turn, mingwVery sensitive, The efficiency is 8 times different.
4. Read is a Linux system function, and mingw may adopt some analog method. Read is slower than fread.
5. The speed at which Pascal programs run is not flattering.

I hope this article will inspire you and welcome to continue with the discussion.