For a Web page that imports csv data, which of the following solutions should be used to import tens of millions of data records or what problems should be considered?

If you want to use PHP to create a web page with questions, you don't know how to start with it. Before that, you have made a simple upload, but there are only a few thousand pieces of data. now you have to process millions of data imports, how to solve the problem. But Python is too difficult to use. May I ask you ~ If you want to use PHP to create a web page with questions, you don't know how to start with it. Before that, you have made a simple upload, but there are only a few thousand pieces of data. now you have to process millions of data imports, how to solve the problem. But Python is too difficult to use. May I ask you ~

Reply content:

If you want to use PHP to create a web page with questions, you don't know how to start with it. Before that, you have made a simple upload, but there are only a few thousand pieces of data. now you have to process millions of data imports, how to solve the problem. But Python is too difficult to use. May I ask you ~

For efficiency issues. Concatenate a string similar to insert into... value (), value ()> transaction> loop insert. Note that you need to modify the configuration of the maximum data size that mysql or other databases can insert at a time. I just tried to insert 0.1 million entries at a time. The first method is about 4 seconds (a table with 5 fields)

The data volume is large and can be imported in batches. you can use ajax to repeatedly request the upload interface and send different parameters to each request.
For example:

Www.xxx.com/upload.php? Offset = 0 & length = 1000

Www.xxx.com/upload.php? Offset = 1000 & length = 1000

This will not cause the PHP script to run to time out.

Optimization of inserted statements and table structure are highly required. Otherwise, the system will be slow and the method of merging data, transactions, and ordered data will be more efficient.

Upload the CSV table to the server first.
Then, PHP imports the CSV table into the MySQL database:

  Query ('set AUTOCOMMIT = 0'); $ db-> query ('Start transaction'); // import CSV table: CSV to array $ fp = fopen('file.csv ', 'R'); while ($ row = fgetcsv ($ fp ))! = FALSE) {// read a row from the file pointer and parse CSV $ stmt = $ db-> prepare ('Insert INTO posts (id, post_title, post_content) VALUES (?,?,?) '); $ Stmt-> bind_param ('iss', $ row [0], $ row [1], $ row [2]); // assume that the content in each row is ID, title, and content $ stmt-> execute (); // if the insertion fails, update if ($ stmt-> affected_rows = 0) {$ stmt = $ db-> prepare ('update posts SET post_title = ?, Post_content =? WHERE id =? '); $ Stmt-> bind_param ('ssi', $ row [1], $ row [2], $ row [0]); $ stmt-> execute (); if ($ stmt-> affected_rows = 0) {echo 'import '. $ row [0]. 'failed! '. "\ N" ;}}} fclose ($ fp); // submit the transaction $ db-> query ('commit '); // The failed operation has been output by echo, ROLLBACK $ db-> query ('set AUTOCOMMIT = 1 ');

Asynchronous, millions of synchronization won't work.