Ask you about automatic exit when processing large data volumes in the phpfor Loop

{Code ...} the basic principle is to read the database data in the order of nssbh and id, and then split the data into a specified number of files (used for front-end ajax requests to simulate multi-thread operations, js sends a request to the backend to receive the name of the uploaded data shard file after processing, and then...

/*** @ Function getSplitCache fragment cache of basic data of excelId * @ param type $ excelId: id of the excel file tax_excel * @ param type $ beforeAndAfterInsertid details_id before and after excel insertion * @ return cache files without unified excel are as follows: EXCEL_ID_NUM */public function getSplitCache ($ excelId, $ beforeAndAfterInsertid) {processTip ('123456 ', 'Start data sharding in Excel ..................... '); $ where ['id'] = array ('gt', $ beforeAndAfterInsertid ['beforeid']); $ where ['excel _ id'] = array ('eq ', $ excelId); // check whether two tables can be inserted simultaneously, if this item $ allExcelData = $ this-> modelDetailsExcel-> where ($ where)-> group ('nssbh, id')-> select (); $ totalNum = count ($ allExcelData); $ this-> _ setAllRow ($ totalNum); // $ interval = ceil ($ beforeAndAfterInsertid [1]-$ beforeAndAfterInsertid [0]) /$ this-> excelDataCacheNum); // an error may occur when multiple users Insert Files simultaneously. $ interval = ceil ($ totalNum/$ this-> exceldatachenum ); for ($ I = 0; $ I <$ this-> exceldatachenum; $ I ++) {$ excelData [$ I] = array_slice ($ allExcelData, $ I * $ interval, $ interval);} unset ($ allExcelData ); // 1 a time difference at the beginning and end can be no longer subdivided // 2 must be divided together and the following $ I = 0; for ($ I; $ I <$ this-> excelDataCacheNum-1; $ I ++) {if ($ excelData [$ I] [$ interval-1] ['nssbh'] ===$ excelData [$ I + 1] [0] ['nssbh']) {$ j = 0; while ($ excelData [$ I] [$ interval-1] ['nssbh'] ===$ excelData [$ I + 1] [$ j] ['nssbh']) {$ excelData [$ I] [] = $ excelData [$ I + 1] [$ j ++]; P ($ j );} array_splice ($ excelData [$ I + 1], 0, $ j);} // output progress processTip ($ I/$ this-> exceldatachenum * 100 ), 'excel data sharding '. $ I. '..................... '); // specifies the location and name of the F cache method of Thinkphp F ('excel _'. $ excelId. '/'. $ I, $ excelData [$ I]);} F ('excel _'. $ excelId. '/'. $ I, $ excelData [$ I]); processTip ('20140901 ', 'excel Data Partition ends ..................... '); return $ this-> exceldatachenum ;}

The basic principle is to read the database data in the order of nssbh and id, and then split the data into a specified number of files (used for front-end ajax requests to simulate multi-thread operations, js sends a request to the backend for receiving the name of the uploaded data Shard, and then processes and changes the data in the file)
The memoryliimit win (win10 64bit 4g memory) allocated by apche php In the system environment is 4 GB for 2g linux (ubuntu14.04 64-bit 8g memory)
After testing, when the data volume is 3500 pieces, 1000 pieces are executed normally.
This problem occurs in linux and Windows when there are 11770 million data records.
When 978 pieces of cache are generated, the php program automatically jumps out and the process exits.

When the part size is 500, the system will get stuck when the part size reaches 488, and the output will be 489 in about two minutes. Then the process jumps out.

Reply content:

/*** @ Function getSplitCache fragment cache of basic data of excelId * @ param type $ excelId: id of the excel file tax_excel * @ param type $ beforeAndAfterInsertid details_id before and after excel insertion * @ return cache files without unified excel are as follows: EXCEL_ID_NUM */public function getSplitCache ($ excelId, $ beforeAndAfterInsertid) {processTip ('123456 ', 'Start data sharding in Excel ..................... '); $ where ['id'] = array ('gt', $ beforeAndAfterInsertid ['beforeid']); $ where ['excel _ id'] = array ('eq ', $ excelId); // check whether two tables can be inserted simultaneously, if this item $ allExcelData = $ this-> modelDetailsExcel-> where ($ where)-> group ('nssbh, id')-> select (); $ totalNum = count ($ allExcelData); $ this-> _ setAllRow ($ totalNum); // $ interval = ceil ($ beforeAndAfterInsertid [1]-$ beforeAndAfterInsertid [0]) /$ this-> excelDataCacheNum); // an error may occur when multiple users Insert Files simultaneously. $ interval = ceil ($ totalNum/$ this-> exceldatachenum ); for ($ I = 0; $ I <$ this-> exceldatachenum; $ I ++) {$ excelData [$ I] = array_slice ($ allExcelData, $ I * $ interval, $ interval);} unset ($ allExcelData ); // 1 a time difference at the beginning and end can be no longer subdivided // 2 must be divided together and the following $ I = 0; for ($ I; $ I <$ this-> excelDataCacheNum-1; $ I ++) {if ($ excelData [$ I] [$ interval-1] ['nssbh'] ===$ excelData [$ I + 1] [0] ['nssbh']) {$ j = 0; while ($ excelData [$ I] [$ interval-1] ['nssbh'] ===$ excelData [$ I + 1] [$ j] ['nssbh']) {$ excelData [$ I] [] = $ excelData [$ I + 1] [$ j ++]; P ($ j );} array_splice ($ excelData [$ I + 1], 0, $ j);} // output progress processTip ($ I/$ this-> exceldatachenum * 100 ), 'excel data sharding '. $ I. '..................... '); // specifies the location and name of the F cache method of Thinkphp F ('excel _'. $ excelId. '/'. $ I, $ excelData [$ I]);} F ('excel _'. $ excelId. '/'. $ I, $ excelData [$ I]); processTip ('20140901 ', 'excel Data Partition ends ..................... '); return $ this-> exceldatachenum ;}

The basic principle is to read the database data in the order of nssbh and id, and then split the data into a specified number of files (used for front-end ajax requests to simulate multi-thread operations, js sends a request to the backend for receiving the name of the uploaded data Shard, and then processes and changes the data in the file)
The memoryliimit win (win10 64bit 4g memory) allocated by apche php In the system environment is 4 GB for 2g linux (ubuntu14.04 64-bit 8g memory)
After testing, when the data volume is 3500 pieces, 1000 pieces are executed normally.
This problem occurs in linux and Windows when there are 11770 million data records.
When 978 pieces of cache are generated, the php program automatically jumps out and the process exits.

When the part size is 500, the system will get stuck when the part size reaches 488, and the output will be 489 in about two minutes. Then the process jumps out.

You gophp.iniFile to see if there is any configuration item with the maximum variable length, find it, modify it and then restart the server.

I have encountered this problem before.

• Enable error_reporting
  You can add

  phperror_reporing(E_ALL);ini_set('display_errors');

  Whether the memory usage exceeds the limit when the monitoring data volume is large

• If it is confirmed that the memory overflows
  Transform your program to reduce memory usage
  For example, generator
  Based on your business, try not to take too much data into the memory.
  Make sure that you do not increase the memory size as the data volume increases while running the program.
  The ideal situation is that the increase in the processing data volume will not result in a sudden increase in the program running memory.

Set_time_limit (0): Set the timeout time. It is not a memory issue. It may be timeout.