Php sorting solution for 0.1 billion QQ numbers

Php sorted 0.1 billion QQ numbers and posted them. Split into thousands of parts for sorting and merging. First, create a txt file with 0.1 billion QQ numbers. PHPcode & lt ;? Php creates a txt file with 0.1 billion QQ numbers (about 85 ~ 100)set_time_limit(0)%%fnqq.txt; $ fpfopen ($ fn, w); $ stmicroti php sorts 0.1 billion QQ numbers
I have enough to eat and drink, and I have posted a post.
Split into thousands of parts for sorting and merging.


First, create a txt file with 0.1 billion QQ numbers.

PHP code

  
  $ V) {$ arr = range ($ v * 10000 + 10000,100 00 * ($ v + 1) + 9999); shuffle ($ arr); fputs ($ fp, implode ("\ n", $ arr ). "\ n"); unset ($ arr);} echo microtime (true)-$ st;?>

Wait a minute or two to complete the creation of 0.1 billion random QQ.

QQ number range:> 10000. The file size is about 840 MB.



The following is a classification of thousands of files.

Take the length of the QQ number as the folder, and the first three digits of the QQ number as the file name.

PHP code

  
  $ V) {if ($ v! = '') {$ Tag =" $ v [0] $ v [1] $ v [2] "; $ text_arr [strlen ($ v)] [$ tag] [] = $ v ;}} foreach ($ text_arr as $ k =>$ v) {$ n_dir = 'qq _ no /'. $ k; if (! Is_dir ($ n_dir) mkdir ($ n_dir); foreach ($ v as $ tag => $ val) {$ n_tf = fopen ($ n_dir. '/'.20.tag.'.txt', 'A + '); fputs ($ n_tf, implode ("\ n", $ val ). "\ n") ;}} unset ($ text_arr); ++ $ I;} echo microtime (true)-$ st;?>

Finally, sort and merge data for each file.

PHP code

  
  $ Val) {if ($ val! = '.' & $ Val! = '.. ') $ Dirs [$ val] = scandir ($ root. '/'. $ val);} foreach ($ dirs as $ key => $ val) {foreach ($ val as $ v) {if ($ v! = '.' & $ V! = '.. ') {$ File = $ root. '/'. $ key. '/'. $ v; $ c = file_get_contents ($ file); $ arr = explode ("\ n", $ c); sort ($ arr); fputs ($ qq_done, implode ("\ n", $ arr); unlink ($ file) ;}} rmdir ($ root. '/'. $ key) ;}rmdir ($ root); echo microtime (true)-$ st;?>

It took about 20 minutes.

Although it is done, the method is very good. The experts in the altar have improved.


------ Solution --------------------
Come to a C version

C/C ++ code

# Include
  
   
# Define BITSPERWORD 32 # define SHIFT 5 # define MASK 0x1F # define N distinct int a [1 + N/BITSPERWORD]; void set (int I) {a [I> SHIFT] | = (1 <(I & MASK); // I & MASK is equivalent to 1 & (32-1 ), that is, 1% 32} void clr (int I) {a [I> SHIFT] & = ~ (1 <(I & MASK);} int test (int I) {return a [I> SHIFT] & (1 <(I & MASK ));} int main () {int I; // initialize for (I = 0; I <N; I ++) clr (I); // read the file, set while (scanf ("% d", & I )! = EOF) set (I); for (I = 0; I <N; I ++) if (test (I) printf ("% d \ n ", i); return 0 ;}
   
------ Solution --------------------
   

   
Since there are ready-made data files, there is no need to construct an insert string.
   

   

    
PHP code
    
set_time_limit(0);$sql =<<< SQLCREATE TABLE IF NOT EXISTS qq1 (  `qq` int(10) NOT NULL,  KEY `qq` (`qq`)) ENGINE=MyISAM DEFAULT CHARSET=utf8;SQL;mysql_connect('localhost', 'root', '');mysql_select_db('test');mysql_query($sql);$filename = str_replace('\\', '/', realpath('qq.txt'));$sql =<<< SQLLOAD DATA INFILE '$filename' INTO TABLE qq1SQL;check_speed(1);mysql_query($sql) or print(mysql_error());;check_speed();