For batch data insertion in php + mysql, I wrote a code for batch data import. The process is as follows: read data from a data table and insert it into a new table. For the first time, I first read all and then insert them together. The result is incorrect because the data volume is too large. The second time, I modified the code and imported the first 150 items. when the user clicked "continue", the next 150 items were imported, an error is reported when 450 to 600 entries are imported. If the number of php + mysql batch data inserts after 300 entries are imported
I wrote a code for batch data import.
The process is as follows: read data from a data table and insert it into a new table.
For the first time, I first read all and then insert them together. The result is incorrect because the data volume is too large.
The second time, I modified the code and imported the first 150 items. when the user clicked "continue", the next 150 items were imported, an error is reported when 450 to 600 entries are imported. If 300 entries are imported, wait a few seconds before clicking "continue. Import will be normal.
Are you sure you want to solve this problem by reading a large amount of data from one data table and importing it to another data table ???
------ Solution --------------------
Find max_execution_time in the php. ini file and increase the value.
Estimate the approximate import time of your data and set the corresponding value.
I did this some time ago. I set 30000.
------ Solution --------------------
Generally, you can use the mysql command line to export data, and then use the mysql command line to import data. (mysqldump, mysqlimport, source ...)
You can also use phpmyadmin code to run and export the data. (when encountering large texts, pay attention to some settings of php. ini, such as memory_limit and upload_max_filesize ..)
------ Solution --------------------
Connect to mysql and exit immediately after the insert operation is executed.
It may be related to the number of mysql connections.
------ Solution --------------------
Learning !!
------ Solution --------------------
Learning
------ Solution --------------------
This is not the case, maybe your code is not fixed,
Set the PHP timeout value to 0.
Set_time_limit (0 );
Insert one entry for each read,
But remember to control the memory during the loop,
In addition, if there are large fields in the record, it is best to generate an SQL file and then export it using source,
I have processed a maximum of 10 million records in the case of Loop reading and re-import, and it is basically okay. If you have hundreds of errors, you will certainly not.
------ Solution --------------------
Try the MySQL Enterprise Edition Manager, which supports large data volume operations