Read and write problems with large PHP files

Tag: Log log php large file read large file write

In the usual learning and development, because we seldom have access to large amounts of data read and write, so when the need suddenly, we can

Can still follow some relatively fast methods, such as File_get_contents,fread and other methods to read the file, but so since if the file read too

Large, there will be problems, in the implementation of large files read and write the time to find some information on the Internet, but some examples are not very consistent with my needs

, so I will combine the online examples, and then write a summary of the blog bar .

So what is the problem, which is to say some of the lower level of PHP implementation number,file_get_contents and Fread , first of all, say

PHP file read functions,file_get_contents and Fread, these two functions are the same principle, are read into the system's memory,

However, if you just want to read the contents of a file into a string, use file_get_contents (), which has a much better performance than fread ().

There is no problem reading a file that is not very large, but when reading large files (e.g. 2GB logs), if your machine has only 4 g of memory,

If you read the entire file and then put it into a string, it can cause the system to explode and cause the memory to die, because there is still a portion of memory to be used for

Maintain the operation of the system and other processes, since this is the case, we need some other way to avoid reading too much content at once, through this

method to implement a large file read.

PHP file read:

The following is an example of reading large files on the Internet, to illustrate the phenomenon that memory will explode.

_________________________________ Invincible Split Line _______________________________________

The requirements are as follows: an existing log file of about 1G, about more than 5 million lines, with PHP to return the last few lines of content.

Implementation method:

1. Use the file function directly to manipulate
Note: Because the file function reads everything into memory at once, and PHP prevents some poorly written programs from taking up too much memory and causing the server to go down, the limit is limited to 16M of memory by default, which is through php.ini Memory_ Limit = 16M To set, this value if set-1, the memory makes the usage unrestricted.

The following is a piece of code that uses file to remove the last line of the document.
The entire code execution takes time 116.9613 (s).

My machine is 2 g of memory, when the press F5 run, the system directly gray, almost 20 minutes after the recovery, it can be seen so large files directly into memory, the consequences is how much serious, so not million can not, memory_limit this thing can not be adjusted too high, otherwise only call room, Let the reset machine out.

_____________________________________ Invincible Split Line _____________________________________

The above example is an example of reading the last few lines, but because the contents of the file are traversed, it is the same as reading the entire file, although

The result is to read the contents of the last few lines, or you can directly use fseek to locate and read portions of the content.

Let's explore how to read and write large files.

Large file reads:

Because the read part is required, if the file is not particularly large, it can be read by file_get_contents or fread with the split parameter (this place feels the need to add a sleep function to reduce the IO peak size, but I do not know if it is correct, There is also a way to read through the while loop, with Fgets, because Fgetss reads a row through a file pointer, and the efficiency is relatively high.

The following is an example of a large file read by fgets and an encoded conversion of the contents of a file (UTF-8-GBK), with the following code:

            $file = fopen ($old _file_path, "R");            $result = fopen ($temporary _file_path, "a");             $re _sign = 0;                while (!feof ($file)) {$content = Fgets ($file);                 $encode = mb_detect_encoding ($content, Array (' ASCII ', ' UTF-8 ', ' GB2312 ', ' GBK ', ' BIG5 '));                    if ($encode = = ' UTF-8 ') {$str = Iconv ($encode, "Gbk//ignore", $content);                     $encode = mb_detect_encoding ($content, Array (' ASCII ', ' UTF-8 ', ' GB2312 ', ' GBK ', ' BIG5 '));                     Fwrite ($result, $STR);                 $re _sign = 1;                 } else {fwrite ($result, $content);            }} fclose ($file);            Fclose ($result); if ($re _sign = = 1) {rename ($old _file_path, $old _file_path.                '. Bak ');                            Rename ($temporary _file_path, $old _file_path);               } else { Unlink ($temporary _file_path); }

Large File Write:

Large file writes are not expensive to write relative to large file reads, because file writes are written to the hard disk, and if too many files are written at once,

will only produce the phenomenon of card hard disk, if the efficiency of the one-time direct write is the longest and most efficient, so the writing of large files is recommended to write the file directly after a single read.

Read and write problems with large PHP files