How to derive large amounts of data in PHP

In the process of PHP development, sometimes there will be data to export, you can use some of the various framework of the package to get the data directly and then throw into the corresponding processing method, but when we encounter a lot of data need to export, we need to use other methods to achieve.

The general processing of big Data has the following solutions: by adding or modifying the event trigger script, generating data, timed task execution script to generate data, directly using Phpexcel to export a large amount of data, using fputcsv real-time data stream way write. A picture is shown as follows:

The main record here is to export the data using the Fputcsv method.

Public Function ExportData () {set_time_limit (0);                Ini_set (' Memory_limit ', ' 1024M ');        $columns = [' Column name 1 ', ' Column Name 2 ', ' Column name 3 '//need a few columns, define the column name];        Set up to tell the browser to download the Excel file Headers header (' Content-description:file Transfer ');        Header (' Content-type:application/vnd.ms-excel '); Header (' content-disposition:attachment; filename= ' Export data-'. Date (' y-m-d ', Time ()) '.        CSV "');        Header (' expires:0 ');        Header (' cache-control:must-revalidate ');        Header (' Pragma:public ');        $fp = fopen (' php://output ', ' a ');//Open output stream mb_convert_variables (' GBK ', ' UTF-8 ', $columns);        Fputcsv ($fp, $columns);//Format the data in CSV format and write to the output stream//Add a query condition to obtain the required data $query = Model::class ()->where ();        Get total, paging cycle processing $accessNum = $query->count ();        $perSize = 1000;                $pages = Ceil ($accessNum/$perSize); for ($i = 1; $i <= $pages; $i + +) {$db _data = $query->limit ($perSize)->offset (($i-1) * $perSize)->get ();                foreach ($db _data as $key = + $value) {$rowData = [];                Get each column of data, conversion processing to need to export data//need format conversion, otherwise it will garbled mb_convert_variables (' GBK ', ' UTF-8 ', $rowData);            Fputcsv ($fp, $rowData);            }//Release the variable memory unset ($db _data);            Refresh output buffer to browser ob_flush ();            You must use both the Ob_flush () and flush () functions to flush the output buffers.        Flush ();                } fclose ($FP);    Exit (); }

This will generate while writing the download file to improve processing speed.
before using the package processing, generate a 25M of data, to 20 minutes. This approach takes more than 1 minutes to process (also related to the actual processing environment).