How to export big file data (executed in the background, automatically generated) and export data automatically generated _ PHP Tutorial

Export big file data (executed in the background, automatically generated) and export data automatically generated. Export big file data (executed in the background, automatically generated) and export data automatically generated I. preface record the previous functions of exporting statistics in the background excel format, also, the big file data export (executed in the background, automatically generated) and data export is automatically generated.
I. Preface

Record the previous function of exporting statistical information in the background excel format. Recently, my colleague asked about the related information and forgot the specific details for a moment;

As you know, the excel data export function is almost required in the background. generally, after clicking, the file is generated and automatically downloaded,

If the data volume is small, the request can be completed immediately and downloaded to the local device;

However, when the data volume is very large, the page must be waiting until the excel file is successfully written,

In this way, the background user cannot operate other pages. Therefore, the excel export function is optimized as follows:

II. generate an excel file

There are many ways to generate excel files. At present, there are no one-to-one records, but this method is used to record the files;

Here the table html format and the corresponding excel statement are used.

(I vaguely remember that the other methods opened with office07 seem to be garbled. later I tried to use a csv file, but it was still garbled, so I used the table format)

File start:

1 $ struserdata = < 5 6789 10 121314

15 16

End of the file:

3

456 Eof;View Code

Of course, there are some tr td tags in the middle of the file.

3. run the program in the background

Scenario:
Click "generate excel" to go to the download page. The program is executed in the background. you do not have to wait until the file is generated. you can perform other operations;
On the download page, you can see the progress of file generation and whether the file is downloadable.
Ideas:
Click generate excel to display the download page ---> show_download method
Generate an excel ---> create_excel method

The create_excel method is called in the show_download method, while the show_download method uses the command line method to execute the program,

Use the php command line method to pass the parameter to the create_excel method

1  // $cmd = "/usr/bin/php  /home/xxx/xxx.php " . $strjoin . "  >/dev/null & ";2  // $a=exec($cmd, $out, $returndata);3  4  5  $command = "/usr/bin/php ".STATISTIC_EXPORT_SCRIPT_DIR."xxx.php " . "'" .$strjoin ."'". " " . $uid . " ". $action ."  & ";6  $process = proc_open($command, array(),$pipes); 7  $var = proc_get_status($process); 8  proc_close($process);9  $pid = intval($var['pid'])+1;

In the create_excel method:

Enter the following code:

1 set_time_limit (0); // The maximum timeout value for canceling the script running time is 2 3 ignore_user_abort (TRUE); // The backend running is not affected by the user's browser closure.

Call related APIs to obtain data:

1 $ statistic = call_user_func (array ('shellscript', 'get _ result'), $ url, $ params); 2 if (! Is_object ($ statistic) |! Isset ($ statistic-> data-> items) {3 usleep (400000); // stop 400 milliseconds 4 $ statistic = call_user_func (array ('shellscript ', 'Get _ result'), $ url, $ params); 5}

4. display the file generation progress

But how can I display the progress of the corresponding file generation? how can I know if the file has been generated?

Here, I use data. xsl when writing data files. each data file generates a corresponding file progress file, which is also called flag_data.xsl;

Ideas:

How to view the progress of a file:

1 public function execscript_process () {2 $ this-> load-> library ('smarty '); 3 $ file_arr_str = array (); 4 $ file_arr_process = array (); 5 $ file_arr_name = array (); 6 $ file_arr = array (); 7 $ refresh_flag = 'yes'; 8 $ uid = $ _ REQUEST ['uid']; 9 $ url_dir = STATISTIC_EXPORT_FILE_DIR. $ uid. '/'; // @ todo10 if (! Is_dir ($ url_dir) {11 @ mkdir ($ url_dir, 0777); 12} 13 $ files = scandir ($ url_dir); 14 15 if (! Empty ($ files) {16 foreach ($ files as $ key => $ value) {17 if ($ value! = '.' & $ Value! = '.. ') {18 if (substr ($ value, 0, 5) = "flag _") {19 $ file_size = filesize ($ url_dir. $ value); 20 if (! Empty ($ file_size) {21 $ fhandle = fopen ($ url_dir. $ value, 'RB + '); 22 fseek ($ fhandle,-1, SEEK_END); 23 $ fstr = ''; 24 while ($ c = fgetc ($ fhandle ))! = False) {25 if ($ c = "\ n" & $ fstr) break; 26 $ fstr = $ c. $ fstr; 27 fseek ($ fhandle,-2, SEEK_CUR); 28} 29 fclose ($ fhandle); 30 $ fstr = trim ($ fstr ); 31 $ fstr_arr_str = explode (',', $ fstr); 32 $ file_arr_process [] = 100 * number_format ($ fstr_arr_str [0]/$ fstr_arr_str [1], 2 ). '%'; 33 $ file_arr_name [] = substr ($ value, 5); 34} 35} 36} 37} 38 39 foreach ($ file_arr_process as $ key => $ value) {40 if ($ value! = '000000') {41 $ refresh_flag = 'no'; 42 break; 43} 44} 45} 46 47 $ file_arr = array (48 'process' => $ file_arr_process, 49 'name' => $ file_arr_name, 50' refresh _ flag '=> $ refresh_flag51); 52 $ file_arr_json = json_encode ($ file_arr); 53 echo $ file_arr_json; 54}View Code 5. download an object

The download method is as follows:

 1     public function execscript_download(){ 2         $filename = $_REQUEST['filename']; 3         $uid = $_REQUEST['uid']; 4         $file_dir = STATISTIC_EXPORT_FILE_DIR.$uid.'/'.$filename; 5         if (!file_exists($file_dir)){ 6             header("Content-type: text/html; charset=utf-8"); 7             echo "File not found!"; 8             exit;  9         } else {10             ini_set("memory_limit","500M"); 11             header('Content-Description: File Transfer');12             header('Content-Type: application/octet-stream');13             header('Content-Disposition: attachment; filename='.basename($file_dir));14             header('Content-Transfer-Encoding: binary');15             header('Expires: ' . gmdate('D, d M Y H:i:s') . ' GMT');16             header('Cache-Control: must-revalidate,post-check=0, pre-check=0');17             header('Pragma: public');18             header('Content-Length: ' . filesize($file_dir));19             readfile($file_dir);20         }21 22     }

6. problems after going online

The local database has been tested and can be launched, but a strange problem occurs;

Symptom description:
When you click generate file in the background to jump to the download page, because the download page displays the file progress page,
It turns out that sometimes there is a progress of the file you just clicked, and sometimes there is no, it feels like no corresponding file is generated;
Solution:
Because both data files and progress files are generated in a folder file of the program, the files in the read folder are used to judge the display progress;
Later I learned that because the background program has two servers, the corresponding folder cannot be found during reading and downloading, and the corresponding folders of the two servers can be shared.

VII. Subsequent optimization

Due to the large number of downloaded files, there are more and more files in the folder, and the originally generated files are of no value. Therefore, a regular file deletion function is added to keep the files for nearly seven days.

Of course, you can use crontab, but I am relatively lazy. when I click generate file, I have determined the expired files in the folder and deleted them.

1 public function execscript_process_show () {2 $ this-> load-> library ('smarty '); 3 $ uid =$ _ REQUEST ['uid']; 4 $ url_dir = STATISTIC_EXPORT_FILE_DIR. $ uid. '/'; // @ todo 5 if (! Is_dir ($ url_dir) {6 @ mkdir ($ url_dir, 0777); 7} 8 $ files = scandir ($ url_dir); 9 if (! Empty ($ files) {10 foreach ($ files as $ key => $ value) {11 if ($ value! = '.' & $ Value! = '..') {12 foreach ($ files as $ key => $ value) {13 if ($ value! = '.' & $ Value! = '..') {14 if (substr ($ value, 0, 5 )! = "Flag _") {15 $ filenamedate = substr ($ value, 0, 10); 16 $ today = date ('Y-m-D', time ()); 17 $ filenamedate = date ('Y-m-D', strtotime ($ filenamedate) + (STATISTIC_FILE_EXPIRE_DAY-1) * 24*3600); 18 if ($ today> $ filenamedate) {// file expired 19 @ unlink ($ url_dir. $ value); 20 @ unlink ($ url_dir. 'flag _'. $ value); 21} 22} 23} 24} 25} 26} 27} 28 29 $ this-> smarty-> assign ('uid', $ uid ); 30 $ this-> smarty-> display ('interact/statistic/execscript. tpl '); 31}

8. postscript

The export of large files is like this. you are welcome to join us;

At that time, when using the command line to execute the method, I also referred to the relevant information and recorded it;

http://blog.csdn.net/yysdsyl/article/details/4636457http://www.codesky.net/article/201202/163385.htmlhttp://www.cnblogs.com/zdz8207/p/3765567.htmlhttp://blog.163.com/mojian20040228@126/blog/static/4112219320097300922992/http://php.net/manual/en/features.commandline.phphttp://blog.csdn.net/yangjun07167/article/details/5603425http://blog.csdn.net/yunsongice/article/details/5445448http://www.cppblog.com/amazon/archive/2011/12/01/161281.aspxhttp://blog.51yip.com/tag/proc_openhttp://www.justwinit.cn/post/1418/http://limboy.me/tech/2010/12/05/php-async.html

Export (executed in the background, automatically generated), data export automatically generated 1. preface record the previous background excel format export statistics function, also...

17 Eof;View Code1 $ struserdata = <