How PHP quickly reads large files

In PHP, the quickest way to read a file is to use functions such as file and file_get_contents, and a few lines of code can be pretty good at accomplishing the functions we need. However, when the file being manipulated is a larger file, these functions may be out of order, the following will be from a requirement to explain the reading of large files, the common method of operation.

Demand

There is a 800M log file, about more than 5 million lines, with PHP to return the contents of the last few lines.

Implementation methods

1. Directly using the file function to operate

Because the file function is a one-time read all content into memory, and PHP in order to prevent some poorly written programs to occupy too much memory and cause the system is out of memory, so that the server is down, so by default, limit the maximum use of memory 16M, this is through the php.ini Memory_ Limit = 16M To set, if the value is set to 1, then the amount of memory is unrestricted.

Here is a section of code that uses file to remove the last line of the document:

<?php
ini_set (' memory_limit ', '-1 ');
$file = ' Access.log ';
$data = file ($file);
$line = $data [Count ($data)-1];
echo $line;
? >

Complete code Execution time consuming 116.9613 (s).

My machine is 2 G of RAM, when pressed F5 run, the system directly dimmed, almost 20 minutes later to recover, visible will be so large files directly into the memory, the consequences of how much serious, so not million, memory_limit this thing can not be too high, otherwise only call the room, Let's reset the machine.

2. Direct call to Linux Tail command to display the last few lines

The Linux command line, you can directly use Tail-n access.log very easy to display the last few lines of log files, you can directly invoke the tail command PHP, execute PHP code as follows:

<?php
$file = ' access.log ';
$file = Escapeshellarg ($file); Secure escape of command-line arguments
$line = ' tail-n 1 $file ';
echo $line;
? >

Complete code Execution time consuming 0.0034 (s)

3. Direct use of PHP fseek for file operation

This is the most common way, it does not need to read the contents of the file into the content, but directly through the pointer to operate, so efficiency is quite efficient. There are a number of different ways in which you can use fseek to manipulate a file, and the efficiency may vary slightly, and here are two common methods:

Method One

First through the fseek find the last EOF of the file, then find the starting position of the last line, take this row of data, find the first line of the starting position, and then take the position of the line, and so on until the $num line is found.

#实现代码如下

<?php
$fp = fopen ($file, "R");
$line = ten;
$pos =-2;
$t = "";
$data = "";
while ($line > 0)
{while
	($t!= "\ n")
	{
		fseek ($fp, $pos, seek_end);
		$t = fgetc ($fp);
		$pos-;
	}
	$t = "";
	$data. = Fgets ($fp);
	$line-;
}
Fclose ($FP);
echo $data
?>

Complete code Execution time consuming 0.0095 (s)

Method Two

Or in a fseek way from the end of the file to read, but this is not a read, but a piece of reading, each reading a piece of data, the read data in a buf, and then through the number of line breaks (\ n) to determine whether the last $num row of data has been read.

#实现代码如下

<?php
$fp = fopen ($file, "R");
$num = ten;
$chunk = 4096;
$fs = sprintf ("%u", FileSize ($file));
$max = (Intval ($fs) = = Php_int_max)? Php_int_max:filesize ($file);
for ($len = 0; $len < $max; $len + = $chunk)
{
	$seekSize = ($max-$len > $chunk)? $chunk: $max-$len; 
  
   fseek ($fp, ($len + $seekSize) *-1, seek_end);
	$readData = Fread ($fp, $seekSize). $readData;
	if (Substr_count ($readData, "\ n") >= $num + 1)
	{
		Preg_match ("!. *?\n) {". ($num). "}$!", $readData, $match);
		$data = $match [0];
		break;
	}
}
Fclose ($FP);
echo $data;
? >
  

Complete code Execution time consuming 0.0009 (s).

Method Three

<?php
function Tail ($fp, $n, $base = 5)
{
	assert ($n > 0);
	$pos = $n + 1;
	$lines = Array ();
	while (count ($lines) <= $n)
	{
		try
		{
			fseek ($fp,-$pos, seek_end);
		}
		catch (Exception $e)
		{
			fseek (0);
			break;
		$pos *= $base;
		while (!feof ($fp))
		{
			array_unshift ($lines, Fgets ($fp));
		}

	Return Array_slice ($lines, 0, $n);
}

Var_dump (Tail (fopen ("Access.log", "r+"));
>

Complete code Execution time consuming 0.0003 (s)

Articles that you may be interested in

• A function of PHP to read directories and tables to display files in a directory
• PHP uses Curl functions to implement multi-threaded crawl Web pages and download files
• PHP Read XML file summary
• PHP error_log () writes error messages to a file
• How to write an array variable to a file in PHP
• PHP Delete directory and all files in directory
• How to solve the problem of concurrent read and write file conflicts in PHP
• A program in which PHP gets all the files in the directory and saves the results to an array