How to deal with large files in PHP

Requirements: There is a log file of about 1G, about more than 5 million lines, in PHP to return the last few lines of content.

In PHP, the quickest way to read a file is to use functions such as file, file_get_contents, and just a few lines of code to do what we need. But when the file being manipulated is a large file, these functions may be out of the way, the following will start from a requirement to illustrate the common method of operation when reading large files.

1. Use the file function directly to manipulate

Because the file function reads everything into memory at once, and PHP prevents some poorly written programs from taking up too much memory and causing the server to go down, the default limit is to use only 16M of memory, which is through php.ini Memory_ Limit = 16M To set, this value if set-1, the memory makes the usage unrestricted.

The following is a piece of code that uses file to remove the last line of the document. The code executes about 2 minutes or so.

$fp = fopen ($file, "R"), $num = ten; $chunk = 4096; $fs = sprintf ("%u", FileSize ($file)); $max = (Intval ($fs) = = Php_int_max)? Php_int_max:filesize ($file); for ($len = 0; $len < $max; $len + = $chunk) {  $seekSize = ($max-$len > $chunk) ? $chunk: $max-$len;    Fseek ($fp, ($len + $seekSize) *-1, seek_end);    $readData = Fread ($fp, $seekSize). $readData;    if (Substr_count ($readData, "\ n") >= $num + 1) {        Preg_match ("!". *?\n) {". ( $num). "} $! ", $readData, $match);        $data = $match [0];        break;    }} Fclose ($FP); Echo $data;

My machine is 2 g of memory, when the press F5 run, the system directly gray, almost 20 minutes after the recovery, it can be seen so large files directly into memory, the consequences is how much serious, so not million can not, memory_limit this thing can not be adjusted too high, otherwise only call room, Let the reset machine out.

2. Directly call the Linux Tail command to display the last few lines

Under the Linux command line, you can directly use Tail-n access.log very easy to display the last few lines of the log file, you can directly use PHP to invoke the tail command, execute the PHP code as follows. The entire code execution takes time 0.0034 (s)

File = ' Access.log '; $file = Escapeshellarg ($file); Safe escape of command-line arguments $line = ' tail-n 1 $file '; echo $line;

3. Use PHP fseek directly for file operation

This method is the most common way, it does not need to read all the contents of the file into the memory, but directly through the pointer to operate, so efficiency is quite efficient. There are many different ways to use fseek to manipulate files, and the efficiency may also vary slightly, and here are two common approaches.

Method One:

First, find the last EOF of the file by fseek, then find the beginning of the last line, take this row of data, then find the starting position of the next line, then take the position of this line, and so on, until the $num line is found.

1 functionTail$fp,$n,$base=5)2 {3     assert($n>0);4     $pos=$n+1;5     $lines=Array();6      while(Count($lines) < =$n){7         Try{8             fseek($fp,-$pos,seek_end);9}Catch(Exception $e){Ten             fseek(0); One              Break; A         } -         $pos*=$base; -          while(!feof($fp)){ the             Array_unshift($lines,fgets($fp)); -         } -     } -     return Array_slice($lines, 0,$n); + } - Var_dump(Tail (fopen("Access.log", "r+"), 10);

Method Two:

Or use fseek way from the end of the file to read, but this is not a single read, but a piece of reading, each read a piece of data, the data will be read in a buf, and then by the number of newline characters (\ n) to determine whether the last $num line of data is read.

1 $fp=fopen($file, "R");2 $line= 10;3 $pos=-2;4 $t= " ";5 $data= "";6  while($line> 0) {7      while($t! = "\ n") {8         fseek($fp,$pos,seek_end);9         $t=fgetc($fp);Ten         $pos--; One     } A     $t= " "; -     $data.=fgets($fp); -     $line--; the } - fclose($fp); - Echo $data

Method Three:

Ini_set (' Memory_limit ', '-1 '); $file = ' Access.log '; $data file ($file); $line $data [Count($data)-1]; Echo $line;

How to deal with large files in PHP