Php sorting 0.1 billion numbers

Split into thousands of parts for sorting and merging. First, create a txt file with 0.1 billion QQ numbers. PHPcode split into thousands of parts for sorting and merging.


First, create a txt file with 0.1 billion QQ numbers.

PHP code?

// Create a txt file with 0.1 billion QQ numbers (about 85 ~ 100 seconds)

Set_time_limit (0 );

$ Fn = 'qq.txt ';

$ Fp = fopen ($ fn, 'w ');

$ St = microtime (true );

$ L = range (0,10000 );

Shuffle ($ l );

Foreach ($ l as $ k => $ v)

{

$ Arr = range ($ v * 10000 + 10000,100 00 * ($ v + 1) + 9999 );

Shuffle ($ arr );

Fputs ($ fp, implode ("\ n", $ arr). "\ n ");

Unset ($ arr );

}

Echo microtime (true)-$ st;

?>

Wait a minute or two to complete the creation of 0.1 billion random QQ.

QQ number range:> 10000. The file size is about 840 MB.



The following is a classification of thousands of files.

Take the length of the QQ number as the folder, and the first three digits of the QQ number as the file name.

PHP code?

// Length number classification (about 360 ~ 400 seconds)

Set_time_limit (0 );

$ St = microtime (true );

If (! Is_dir ('qq _ no') mkdir ('qq _ no ');

$ File = fopen('qq.txt ', 'r ');

$ I = 0;

$ End_s = '';

While (! Feof ($ file ))

{

$ G = 1042*1024;

Fseek ($ file, $ g * $ I );

$ S = fread ($ file, $ g );

$ End = strrpos ($ s, "\ n ");

$ Arr_s = $ end_s.substr ($ s, 0, $ end );

$ End_s = substr ($ s, $ end );

$ Arr = explode ("\ n", $ arr_s );

Foreach ($ arr as $ k => $ v)

{

If ($ v! = '')

{

$ Tag = "$ v [0] $ v [1] $ v [2]";

$ Text_arr [strlen ($ v)] [$ tag] [] = $ v;

}

}

Foreach ($ text_arr as $ k => $ v)

{

$ N_dir = 'qq _ no/'. $ k;

If (! Is_dir ($ n_dir) mkdir ($ n_dir );

Foreach ($ v as $ tag => $ val)

{

$ N_tf = fopen ($ n_dir. '/'.20.tag.'.txt', 'A + ');

Fputs ($ n_tf, implode ("\ n", $ val). "\ n ");

}

}

Unset ($ text_arr );

++ $ I;

}

Echo microtime (true)-$ st;

?>

Finally, sort and merge data for each file.

PHP code?

// Pull after sorting (800 ~ 920 seconds)

Set_time_limit (0 );

$ St = microtime (true );

$ Qq_done = fopen('qq_done.txt ', 'A + ');

$ Root = 'qq _ no ';

$ Dir_array = scandir ($ root );

Foreach ($ dir_array as $ key => $ val)

{

If ($ val! = '.' & $ Val! = '..')

$ Dirs [$ val] = scandir ($ root. '/'. $ val );

}

Foreach ($ dirs as $ key => $ val)

{

Foreach ($ val as $ v)

{

If ($ v! = '.' & $ V! = '..')

{

$ File = $ root. '/'. $ key. '/'. $ v;

$ C = file_get_contents ($ file );

$ Arr = explode ("\ n", $ c );

Sort ($ arr );

Fputs ($ qq_done, implode ("\ n", $ arr ));

Unlink ($ file );

}

}

Rmdir ($ root. '/'. $ key );

}

Rmdir ($ root );

Echo microtime (true)-$ st;

?>