How to import massive txt data into the database using php

How to import massive txt data into the database using php

Column 1 column 2 column 3 Column 4 column 5 A 00003131 0 0 adductive #1 adducting #1 adducent #1 A 00003356 0 0 nascent #1 A 00003553 0 0 emerging #2 emergent #2 A 00003700 0.25 0 dissilient #1 -- 0.1 million data records in total -- Import the data to the database. The data table structure is: Automatic word_id increment Word [adductive #1 adducting #1 adducent #1] This TXT record must be converted to three SQL records Value = Third Column-fourth column; if it is 0, this record is omitted from the data table. The code is as follows: $ Li) {$ arr = explode ("", $ li); $ senti_value = $ arr [2]-$ arr [3]; if ($ senti_value! = 0) {if ($ I >= 20000 & $ I <25000) // batch import to avoid failure {$ mm = explode ("", $ arr [4]); foreach ($ mm as $ m) // [adductive #1 adducting #1 adducent #1] This TXT record must be converted into three SQL records {$ nn = explode ("#", $ m ); $ word = $ nn [0]; $ SQL. = "(\" $ word \ ", 1, $ senti_value, 2),"; // note that word may contain single quotes (such as jack's ), therefore, we must use double quotation marks to include word (escape) }}$ I ++ ;}// echo $ I; $ SQL = substr ($ SQL, 0,-1 ); // remove the last comma // echo $sql1_file_put_contents('20000-25000.txt ', $ SQL); // import the database in batches, 5000 entries at a time, about 4 0 seconds; too many max_execution_time imports at a time will not be enough, resulting in failure?> Note: 1. when importing massive data, pay attention to some restrictions of PHP. you can temporarily adjust them; otherwise, an error will be reported. Allowed memory size of 33554432 bytes exhausted (tried to allocate 16 bytes) 2. use PHP to operate the TXT file file_get_contents () file_put_contents () 3. during massive import, it is best to import data in batches, with a lower probability of failure. 4. before importing a large volume of data, the script must be tested multiple times before use, for example, use 100 pieces of data to test 5. after the import, if PHP's mem_limit is not enough, the program will still not run (we recommend that you modify php. ini method to improve mem_limit, rather than using temporary statements) That's all. if anyone has the guts, you can find a large amount of data in the txt test.