C + + Large file data read (for example, read floating-point numbers)

Reference article: http://www.cnblogs.com/yicoder/

Because there are several recent experiments that need to deal with large data, because it is necessary to read 10 million double type data, although it is not required to quickly read the file data, but it is unbearable that a few 10 seconds of the pause. So I searched the internet for the processing of big data.

Although scanf () can be used to increase the speed of reading, there are still a few 10 seconds of pauses. So here you choose to use Fread () to read out all the characters.

In order to facilitate the experiment, we first wrote the file data of the double type data n which generated 0< n <10. This did not optimize, longer time.

Source:

void Product_data ()
{
uniform_int_distribution<unsigned> u (0, 9999);
Default_random_engine e (Time (0));
Freopen ("Data.txt", "w", stdout);
for (int i = 0; i < 10000000; i++) {
cout << (Double) U (e)/1000 << Endl;
}
Fclose (stdout);
}

Comparison of the results of various methods

It is visible that using fread to read the entire file is faster than the other two methods.

The reason why CIN is slower than Scand is that the flow data processing is slower than standardized processing. And it's faster with fread. All data is read as a string and read into the entire file at once, the main time overhead of this method is to convert the character to the required value.

Although read with Fread is much faster, but this method has a great flaw, the buffer array capacity is larger than all the characters contained in the file, otherwise you can not read out all the data, perhaps the use of block processing to solve the problem. If you have a better way of prawn, please advise twos.

Source:

#include <iostream>#include<fstream>#include<ctime>#include<random>#include<cstdio>#defineN 10000000#defineM 70000000using namespacestd;voidCin_read () {time_t start_time=clock (); Double*nums =New Double[N]; Freopen ("Data.txt","R", stdin);  for(inti =0; i < N; i++) {cin>>Nums[i]; } time_t end_time=clock ();    Fclose (stdin); cout<<"cin_read Time:"<< End_time-start_time <<"Ms"<<Endl;}voidScanf_read () {time_t start_time=clock (); Double*nums =New Double[N]; Freopen ("Data.txt","R", stdin);  for(inti =0; i < N; i++) {scanf ("%LF", &Nums[n]); } time_t end_time=clock ();    Fclose (stdin); cout<<"scanf_read Time:"<< End_time-start_time <<"Ms"<<Endl;}Double* Transform_num (Char* Buffer,intlenght) {    BOOLIs_dec =false; Double*nums =New Double[M]; intK =0; intNT =Ten;  for(inti =0; I < lenght; i++){        if(buffer [i] = ='\ n') {k++; NT=Ten; Continue; }        Else if(buffer [i] = ='.') Is_dec =true; if(!IS_DEC) Nums[k] = Buffer[i]-'0'; Else{Nums[k]+= (Double) (Buffer[i]-'0')/NT; NT= nt*NT; }    }}voidFread_read () {time_t start_time=clock (); Freopen ("Data.txt","R", stdin); Char*buffer =New Char[M];//read all characters at once to buf, including ' \ n '    intLenght = fread (buffer,1, M, stdin); Double*nums =transform_num (buffer, lenght); time_t End_time=clock ();    Fclose (stdin); cout<<"fread_read Time:"<< End_time-start_time <<"Ms"<<Endl;}voidProduct_data () {uniform_int_distribution<unsigned> U (0,9999); Default_random_engine E (0)); Freopen ("Data.txt","W", stdout);  for(inti =0; I <10000000; i++) {cout<< (Double) U (e)/ +<<Endl; } fclose (stdout);}intMain () {cout<<"*******************************"<<Endl;    Cin_read (); cout<<"*******************************"<<Endl;    Scanf_read (); cout<<"*******************************"<<Endl;    Fread_read (); cout<<"*******************************"<<Endl; return 0;}

Finish.

C + + Large file data read (for example, read floating-point numbers)