Python Django Bulk Import data

Preface:

During this time, we studied how to bulk import data into a database during Django Web page authoring.

This process is really appalling, made a lot of low-level mistakes, which will be said in the text. Furthermore, the import data is using the Py script, the script content is referenced to the self-Improvement Academy-Intermediate Tutorial-data import.

Note: This article mainly introduces your own learning experience summary, but not the tutorial!

text: First of all, the use of Django Bulk_create () function to implement data Bulk Import function, why choose it?

1 bulk_create () is to execute a SQL to deposit multiple data, making the import faster;

2 bulk_create () reduces the number of SQL statements;

Then, we prepare the data source to be imported, the data source format can be xls,csv,txt and other text documents;

Finally, write the Py script and run it!

The py script is as follows:

#coding: Utf-8 import os os.environ.setdefault ("Django_settings_module", "www.settings") ' DJANGO When the version is greater than or equal to 1.7, you need to add the following two clauses import djangodjango.setup () otherwise it will throw an error django.core.exceptions.AppRegistryNotReady:Models aren ' t Loaded yet. " Import Djangoimport Datetimeif Django. VERSION >= (1, 7): #自动判断版本 Django.setup () from keywork.models import lorderf = open (' cs.csv ') worklist = []next (f) #将文件标记 Move to the next row for lines in f:parts = Line.replace (' "', ') #将字典中的" replace empty parts = Parts.split (';') #按; Slice the string worklist.append (Lorder (Serv_id=parts[0], serv_state_name=parts[1], acct_code=parts[2], acct_name=parts[3], acc_nbr=parts[4], user_name= PARTS[5], frod_addr=parts[6], mkt_chnl_name=parts[7],mkt_grid_name=parts[8], com_chnl_name=parts[9],com_grid_  NAME=PARTS[10], product_name=parts[11],access_name=parts[12], completed_time=parts[13],remove_data=parts[14], SERVICE_OFFER_NAME=PARTS[15], org_name=parts[16], staff_name=parts[17],staff_code=parts[18], Handle_time=par Ts[19],finish_TIME=PARTS[20], prod_offer_name=parts[21],eff_date=parts[22], exp_date=parts[23],main_flag=parts[24], party_ NAME=PARTS[25]) F.close () LOrder.objects.bulk_create (worklist)

According to the above Py script source code mainly speaking of their own learning process encountered problems

Issue 1: the first row in the data source that needs to be imported is usually the field name, starting with the second row is the data, so the script uses next (f) to move the text marker to the second row, otherwise there will be problems, such as the field name is generally English, the default is string formatting, Script execution will meet VALIDATIONERROR:YYYY-MM-DD Hh:mm[:ss[.uuuuuu]][tz] This models data format does not match the import data format Error!

Issue 2: Note parts = parts.split (';') #按; Slice the string, because we import data there are spacers between each column of data in each row, such as comma in CSV, XLS Hollow lattice, and various text default interval symbols, The Split function uses the following example:

The following example shows how the split () function is used:

The result of the above example output is as follows:

[' Line1-abcdef ', ' line2-abc ', ' LINE4-ABCD ']
[' Line1-abcdef ', ' \nline2-abc \NLINE4-ABCD ']
Issue 3: If the import data source exceeds 10M and then the database defaults to maximum 10M, then the above script will not run successfully. Take MySQL, for example, if the import data size exceeds the data set, then the 2006 go away error is reported and needs to be in the My.ini in MySQL. Under [Mysqld], add the following statement:

max_allowed_packet=300m--Maximum allowable packet size 300mwait_timeout=200000  --Connection time 200000sinteractive_timeout = 200000-- Disconnect Time 200000s

The above is the whole content of this article, I hope you learn Python bulk import data to help.

