How PHP processes large files

Requirement: there is a 1 GB log file, with about more than 5 million lines. use php to return the last few lines of content. In php, when reading files, the quickest way is to use functions such as file and file_get_contents. a few simple lines of code can be very float... "/> <scripttype =" text/javascript "src =" h requirement: there is a 1 GB log file, which contains about more than 5 million lines. use php to return the last few lines of content. In php, the quickest way to read files is to use functions such as file and file_get_contents, A few simple lines of code can perfectly complete the functions we need. However, when the operated file is a relatively large file, these functions may not be able to work properly. The following describes how to read large files from a requirement, common operation methods. 1. directly use the file function to operate. because the file function reads all the content into the memory at a time, php causes insufficient system memory to prevent some poorly written programs from occupying too much memory, this causes server downtime. by default, the maximum memory usage is 16 MB. set memory_limit = 16 m in ini. if this value is set to-1, the memory usage is not limited. Below is a piece of code that uses file to retrieve the last line of the file. Code execution takes about 2 minutes. 01 $ fp = fopen ($ file, "r"); 02 $ num = 10; 03 $ chunk = 4096; 04 $ fs = sprintf ("% u ", filesize ($ file); 05 $ max = (intval ($ fs) = PHP_INT_MAX )? PHP_INT_MAX: filesize ($ file); 06 for ($ len = 0; $ len <$ max; $ len + = $ chunk) {07 $ seekSize = ($ max-$ len> $ chunk )? $ Chunk: $ max-$ len; 08 fseek ($ fp, ($ len + $ seekSize) *-1, SEEK_END); 09 $ readData = fread ($ fp, $ seekSize ). $ readData; 10 11 if (substr_count ($ readData, "\ n") >=$ num + 1) {12 preg_match ("! (.*? \ N) {". ($ num)."} $! ", $ ReadData, $ match); 13 $ data = $ match [0]; 14 break; 15} 16} 17 fclose ($ fp); 18 echo $ data; my machine has 2 GB of memory. when I press F5 to run the machine, the system will be grayed out directly, and it will be restored after about 20 minutes. it can be seen that all such large files will be directly read into the memory, the consequence is serious, so no more than. memory_limit cannot be too high. Otherwise, you only need to call the data center to let the reset machine go. 2. directly call the linux tail command to display the last few lines under the linux command line, you can directly use the tail-n 10 access. log easily displays the last few lines of the log file. you can directly use php to call the tail command and run the following php code. it takes 0.0034 (s) 1 file = 'Access. log'; 2 $ file = escapeshellarg ($ file); // escape command line parameters safely. 3 $ line = 'tail-n 1 $ file'; 4 echo $ line; 3. directly using php fseek for file operations is the most common method. it does not need to read all the file content into the memory, but directly uses a pointer to perform operations, therefore, the efficiency is quite efficient. when using fseek to operate files, there are also a variety of different methods, and the efficiency may be slightly different. below are two common methods. method 1: First find the last EOF the file through fseek, and then find the last The starting position of a row, the data of this row, the starting position of the next row, the position of this row, and so on until the $ num row is found. View sourceprint? 01 function tail ($ fp, $ n, $ base = 5) 02 {03 assert ($ n> 0); 04 $ pos = $ n + 1; 05 $ lines = array (); 06 while (count ($ lines) <= $ n) {07 try {08 fseek ($ fp,-$ pos, SEEK_END ); 09} catch (Exception $ e) {10 fseek (0); 11 break; 12} 13 $ pos * = $ base; 14 while (! Feof ($ fp) {15 array_unshift ($ lines, fgets ($ fp); 16} 17} 18 return array_slice ($ lines, 0, $ n ); 19} 20 var_dump (tail (fopen ("access. log "," r + "), 10); Method 2: Use the fseek method to read the last part of the file, but this is not a one-bit read, it is a piece of READ. when reading a piece of data, it puts the read data in a buf, and then uses the line break (\ n) to determine whether the last $ num row of data has been read. 01 $ fp = fopen ($ file, "r"); 02 $ line = 10; 03 $ pos =-2; 04 $ t = ""; 05 $ data = ""; 06 while ($ line> 0) {07 while ($ t! = "\ N") {08 fseek ($ fp, $ pos, SEEK_END); 09 $ t = fgetc ($ fp); 10 $ pos --; 11} 12 $ t = ""; 13 $ data. = fgets ($ fp); 14 $ line --; 15} 16 fclose ($ fp); 17 echo $ data method 3: 1 ini_set ('memory _ limit ', '-1'); 2 $ file = 'Access. log'; 3 $ data = file ($ file); 4 $ line = $ data [count ($ data)-1]; 5 echo $ line;