I have experienced large file data export (background execution, automatic generation), data export automatic generation _php tutorial

I have experienced large file data export (background execution, automatic generation), data export automatic generation

First, preface

Record a previous do in the background Excel format Export statistics function, also recently colleague asked related things, in the moment unexpectedly forget the specific details, so record;

As you know, Excel exports the function of the data, the background is almost necessary to function, usually after the click, generate files and then automatically download,

If it is a small amount of data, you can request the completion of a complete, download to local;

However, if the amount of data is particularly large, the page must be waiting until it is written to excel successfully,

This affects the background user cannot manipulate the other pages, for this reason, the following functional optimizations for Excel export:

Second, generate Excel file

There are many ways to generate Excel files, not a single record, just record the method;

This uses the table's HTML format, and the corresponding Excel declaration

(Vaguely remember other methods with office07 open when it seems to be garbled, after the attempt to use CSV format file, can still be garbled, so the form of table)

The beginning of the file:

1 $struserdata = <<<Eof 2 3 xmlns:x= "urn: Schemas-microsoft-com:office:excel " 4 xmlns=" HTTP://WWW.W3.ORG/TR/REC-HTML40 "> 5 6 Public"-//w3c//dtd XHTML 1.0 transitional//en" "Http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd" > 7 8 9 Ten A - - the - - View Code

End of File:

1 $struserdata = <<<Eof2 Eof; 3 4 5 6Eof; View Code

Of course, there are some TR TD labels in the middle of the file.

Third, let the program in the background to execute

    Scene:
        After the user clicks generate Excel, jumps to the download page, the program executes in the background, the user may not have to wait for the build to complete, can perform other actions;
        Download page to see the progress of file generation and whether the status can be downloaded
    Ideas:
        Click Generate Excel to display the download page  ---> Show_download method
        Generate Excel---> Create_excel methods

The Create_excel method is called in the Show_download method, and the Show_download method uses the command line to execute the program.

Use the PHP command line to pass parameters to the Create_excel method

1  // $cmd = "/usr/bin/php  /home/xxx/xxx.php". $strjoin. "  >/dev/null &"; 2 //$a =exec ($cmd, $out, $returndata); 3  4  5  $command = "/usr/bin/php". Statistic_export_script_dir. " xxx.php "." ". $strjoin $uid $action . "  & "; 6  $process Proc_open ($commandArray(),$pipes7  $var Proc_get_status ($process8  Proc_close ($process); 9  $pid intval ($var[' pid ']) +1;

In the Create_excel method:

You need to fill in the following code:

1 Set_time_limit // maximum timeout for canceling script run time 2 3 Ignore_user_abort (TRUE// background run, not affected by user close browser

Call the relevant API to get the data:

1 $statistic Call_user_func (array(' Shellscript ', ' Get_result '),$url,$params); 2 if (! Is_object ($statistic) | | ! isset ($statistic->data->items)) {3     usleep(400000);   stop 400 milliseconds 4     $statisticcall_user_func(' Shellscript ', ' Get_result '),$url,$params); 5 }

Iv. show the progress of the file generation

But how to show the corresponding file generation progress, how to know that the file is actually generated?

Here, I use the method is to write the data file when the data.xsl, each data file generated a corresponding file progress file, temporarily known as flag_data.xsl;

Ideas:

To view a file's Progress method:

1 Public functionexecscript_process () {2 $this->load->library (' Smarty ');3 $file _arr_str=Array();4 $file _arr_process=Array();5 $file _arr_name=Array();6 $file _arr=Array();7 $refresh _flag= ' Yes ';8 $uid=$_request[' UID '];9 $url _dir= Statistic_export_file_dir.$uid.' /';//@todoTen if(!Is_dir($url _dir)){ One@mkdir($url _dir, 0777); A } - $files=Scandir($url _dir); - the if(!Empty($files)){ - foreach($files as $key=$value) { - if($value! = '. ' &&$value!='..'){ - if(substr($value, 0, 5) = = "Flag_"){ + $file _size=filesize($url _dir.$value); - if(!Empty($file _size)){ + $fhandle=fopen($url _dir.$value, ' rb+ '); A fseek($fhandle,-1,seek_end); at $fstr= ''; - while(($c=fgetc($fhandle)) !==false) { - if($c= = "\ n" &&$fstr) Break; - $fstr=$c.$fstr; - fseek($fhandle,-2,seek_cur); - } in fclose($fhandle); - $fstr=Trim($fstr); to $fstr _arr_str=Explode(',',$fstr); + $file _arr_process[] = 100 *Number_format($fstr _arr_str[0]/$fstr _arr_str[1],2). ' %'; - $file _arr_name[] =substr($value, 5); the } * } $ }Panax Notoginseng } - the foreach($file _arr_process as $key=$value) { + if($value! = ' 100% '){ A $refresh _flag= ' No '; the Break; + } - } $ } $ - $file _arr=Array( -' Process ' =$file _arr_process, the' Name ' + =$file _arr_name, -' Refresh_flag ' =$refresh _flagWuyi ); the $file _arr_json= Json_encode ($file _arr); - Echo $file _arr_json; Wu}View Code

V. Download file

Download the file to say, since the success has been generated, the download method is as follows:

1      Public functionexecscript_download () {2         $filename=$_request[' filename '];3         $uid=$_request[' UID '];4         $file _dir= Statistic_export_file_dir.$uid.' /'.$filename;5         if(!file_exists($file _dir)){6             Header("content-type:text/html; Charset=utf-8 ");7             Echo"File not found!";8             Exit; 9}Else {Ten             Ini_set("Memory_limit", "500M");  One             Header(' Content-description:file Transfer '); A             Header(' Content-type:application/octet-stream '); -             Header(' Content-disposition:attachment; Filename= '.basename($file _dir)); -             Header(' Content-transfer-encoding:binary '); the             Header(' Expires: '.gmdate(' d, D M Y h:i:s '). ' GMT '); -             Header(' Cache-control:must-revalidate,post-check=0, pre-check=0 '); -             Header(' Pragma:public '); -             Header(' Content-length: '.filesize($file _dir)); +             ReadFile($file _dir); -         } +  A}

Vi. problems arising after the launch

The Local has already been tested, can go online, but there are strange problems;

    Description of the phenomenon:
        When you click the Generate file in the background, jump to the download page, because the download page is the page showing the progress of the file,
        Unexpectedly appears sometimes has just clicked the file progress, sometimes does not have, the feeling did not produce the corresponding file;
    Workaround:
        Because both the data file and the progress file are generated in a folder file of the program, so the reading is the file under the Read folder, thus judging the display progress;
        Later, because the background program has two servers, resulting in reading and downloading when the corresponding folder can not be found, two servers corresponding folder to get a shared directory on it

Vii. corresponding follow-up optimization

Due to the number of files downloaded, resulting in more and more files under the folder, and the original generated files are worthless, so added a regular deletion of the file function, only the last seven days of the file

Of course, you can use crontab, but I am more lazy, is in the click to generate files, determine the folder of outdated files, thereby deleting

1      Public functionexecscript_process_show () {2         $this->load->library (' Smarty ');3         $uid=$_request[' UID '];4         $url _dir= Statistic_export_file_dir.$uid.' /';//@todo5         if(!Is_dir($url _dir)){6@mkdir($url _dir, 0777);7         }        8         $files=Scandir($url _dir);9         if(!Empty($files)){Ten             foreach($files  as $key=$value) { One                 if($value! = '. ' &&$value!='..'){ A                     foreach($files  as $key=$value) { -                         if($value! = '. ' &&$value!='..'){ -                             if(substr($value, 0, 5)! = "Flag_"){ the                                 $filenamedate=substr($value, 0,10); -                                 $today=Date(' y-m-d ', Time()); -                                 $filenamedate=Date(' y-m-d ',Strtotime($filenamedate) + (statistic_file_expire_day-1) *24*3600); -                                 if($today>$filenamedate){//File Expiration +@unlink($url _dir.$value); -@unlink($url _dir. ' Flag_ '.$value); +                                 } A                             } at                         } -                     }                     -                 } -             } -         } -  in         $this->smarty->assign (' uid ',$uid); -         $this->smarty->display (' Interact/statistic/execscript.tpl '); to}

Eight, PostScript

Large file export is basically this appearance, welcome everybody spit groove, common exchange;

At that time in the command line to execute the method, also refer to the corresponding information, record;

http://blog.csdn.net/yysdsyl/article/details/4636457http://www.codesky.net/article/201202/163385.htmlhttp:// www.cnblogs.com/zdz8207/p/3765567.htmlhttp://blog.163.com/mojian20040228@126/blog/static/ 4112219320097300922992/http://php.net/manual/en/features.commandline.phphttp://blog.csdn.net/yangjun07167/ article/details/5603425http://blog.csdn.net/yunsongice/article/details/5445448http://www.cppblog.com/amazon/ archive/2011/12/01/161281.aspxhttp://blog.51yip.com/tag/proc_openhttp://www.justwinit.cn/post/1418/http:// Limboy.me/tech/2010/12/05/php-async.html

http://www.bkjia.com/PHPjc/1048777.html www.bkjia.com true http://www.bkjia.com/PHPjc/1048777.html techarticle I have experienced the large file data export (background execution, auto-generated), data export automatic Generation One, preface record a previous do the background Excel format Export statistics function, also ...

