To make a Web page that imports CSV data, what are the options or considerations for dozens of of millions data?

Want to use PHP to do a title Web page, novice do not know how to start, before doing a simple upload, but the data only thousands of, now to deal with the millions data import, how to solve. Python But Python is too frustrating to use PHP. Let me ask you the great God ~

Reply content:

Want to use PHP to do a title Web page, novice do not know how to start, before doing a simple upload, but the data only thousands of, now to deal with the millions data import, how to solve. Python But Python is too frustrating to use PHP. Let me ask you the great God ~

Efficiency issues. Stitching into strings like insert Into...value (), value () such as > Transactions >> looping inserts. Note that a lot of data is inserted at one time to modify the configuration of the maximum amount of data that MySQL or other databases allow to be inserted at one time. I just tried it once. Insert 100,000 in the first method about 4 seconds (5 fields such as table)

A large amount of data can be imported into batches, with Ajax repeatedly request upload interface, each request to pass different parameters.
Such as:

www.xxx.com/upload.php?offset=0&length=1000

www.xxx.com/upload.php?offset=1000&length=1000

This does not allow the PHP script to run out of time.

For the inserted statement optimization, as well as the table structure of the larger requirements, it would be slow to be scary, merging data + transaction + ordered data is more efficient way

Upload the CSV form to the server first.
Then PHP imports the CSV table into the MySQL database:

 !--? php//open transactions, BULK Insert $db = new mysqli (' 127.0.0.1 ', ' user ', ' Pass ', ' dbname ') , 3306), $db--->query (' SET autocommit=0 '), $db->query (' START TRANSACTION ');//import CSV table: CSV to array $fp = fopen (' File.csv ', ' R '), while (($row = Fgetcsv ($fp))!== FALSE) {//reads a line from the file pointer and resolves the csv $stmt = $db->prepare (' INSERT into posts (ID,    Post_title, Post_content) VALUES (?,?,?) '); $stmt->bind_param (' ISS ', $row [0], $row [1], $row [2]);    This assumes that the contents of each line are ID, title and content $stmt->execute (); If the insert fails, update the IF ($stmt->affected_rows = = 0) {$stmt = $db->prepare (' update posts SET post_title =?, Post_ Content =?        WHERE id =? ');        $stmt->bind_param (' SSI ', $row [1], $row [2], $row [0]);        $stmt->execute (); if ($stmt->affected_rows = = 0) {echo ' Import '. $row [0]. ' failed! '. "        \ n "; }}}fclose ($FP);//COMMIT Transaction $db->query (' Commit '); The failed operation already has the echo output and does not need to roll back the Rollback$db->query (' SET autocommit=1 ');  

Asynchronous, sync millions of can't stand it.

