PHP several dozens of g large file data statistics and sorting processing

Many big internet companies interview will have this problem, there is a 4G file, how to use only 1G memory machine to calculate the number of occurrences of the file in the most number of numbers (assuming that 1 lines is an array, such as QQ number). If this file is only 4 B or a few 10 trillion, then the simplest way is to read the file directly after the analysis statistics. But this is a 4G file, of course, it may be dozens of g or even hundreds of g of files, which is not directly read to solve.

Also for such a large file, simply do with PHP is definitely not working, my idea is no matter how large files, first to cut for multiple applications can withstand small files, and then batch or in turn analyze the statistics of small files and then gross position the results of the summary of the final results to meet the requirements. Similar to the more popular MapReduce model, the core idea is "map" and "Reduce", plus Distributed file processing, of course I can understand and use only Reduce after processing.

  $fp  = fopen  (' qq.txt ', ' w+ ' ); for  ( $i  = 0;  $i  <1000000000;  $i  ++ $ str  = mt_rand  (10000,9999999999). "    \ n ";  fwrite  ( $fp ,  $str   fclose  ( $fp ); 

The world of Makefile is longer, and running PHP files directly under Linux can save time and, of course, generate files in other ways as well. php-client The generated file is approximately 11G.

Then use the Linux split to cut the file, cutting the standard for 1 files per 1 million rows of data.

Split -l 1000000-A 3 qq.txt Qqfile

Qq.txt is divided into the name is qqfileaaa to qqfilebml 1000 files, each file 11mb size, then use any processing method will be relatively simple. I still use PHP to analyze statistics:

$results=Array();foreach(Glob('/tmp/qq/* ') as $file ){    $fp=fopen($file, ' R '); $arr=Array();  while($QQ=fgets($fp) ){        $QQ=Trim($QQ); isset($arr[$QQ]) ?$arr[$QQ]++ :$arr[$QQ]=1; }    Arsort($arr); //there are problems with the following processing methods     Do{        $i=0; foreach($arr  as $QQ=$times ){            if($i> 10 ){                isset($results[$QQ]) ?$results[$QQ]+=$times:$results[$QQ]=$times; $i++; } Else {                 Break; }        }    }  while(false); fclose($fp); }if($results ){    Arsort($results);  Do{        $i=0; foreach($results  as $QQ=$times ){            if($i> 10 ){                Echo $QQ. "\ T".$times. "\ n"; $i++; } Else {                 Break; }        }    }  while(false);}

So each sample taken the first 10, and finally put together to analyze the statistics, do not exclude the number in each sample ranked 11th, but the total number of absolute in the first 10 of the possibility, so the statistical calculation algorithm needs to be improved.

Some people may say that using the awk and sort commands in Linux can be sorted, but I tried to do it if it was a small file, but 11G of files, both memory and time, were unbearable. Here is my change of 1 Awk+sort script, perhaps the writing has a problem, ask the cattle people guidance.

Awk-f ' \\@ ' {name[$1]++} END {for (count in name) print Name[count],count} ' qq.txt | sort -n > 123.txt

awk-F ‘\\@‘ ‘{name[$1]++ } END {for (count in name) print name[count],count}‘ qq.txt |sort-n > 123.txt

Source: http://www.feiyan.info/50.html

PHP several dozens of g large file data statistics and sorting processing