Django Bulk import of XML data

Django Background Bulk Import data

In the production environment, often data is not a few or hundreds of, then for example, the company all employee employee number or account password into the background, it is not recommended that you go backstage a record to add

How to Bulk Import SVN records from XML

The first step:

Building Models for data

@python_2_unicode_compatibleclass Svnlog (models. Model):  vision = models. Integerfield (verbose_name=u "revision", Blank=false, Null=false,)  author = models. Charfield (verbose_name=u "Author", Max_length=60, Blank=true, null=true)  date = models. Datetimefield (verbose_name=u "revision Time", null=true)  msg = models. TextField (verbose_name=u "comment message", Blank=false, Null=false, Default=u "")  paths = models. TextField (verbose_name=u "Affected file", Blank=false, Null=false, Default=u "")  Created_time = models. Datetimefield (verbose_name=u "Creation Time", Auto_now_add=true,)  Update_time = models. Datetimefield (verbose_name=u "Modify Time", Auto_now=true,)  class Meta:    ordering = [' revision ']  def __str__ ( Self):    return u ' r%s '% (self.revision or U "",)

Now that we've built the model, we're going to build a models that accepts our XML files.

@python_2_unicode_compatibleclass Importlogfile (models. Model):  LogFile = models. Filefield (upload_to= ' LogFile ')  FileName = models. Charfield (max_length=50, verbose_name=u ' filename ')  class Meta:    ordering = [' filename ']  def __str__ (self):    return self. FileName

OK, the above code we define the data and upload the file model

Synchronizing databases

Python manage.py Makemigrationspython manage.py Migrate

Then we go to modify admin.py so that we can upload files from the background,

Class Importlogadmin (admin. Modeladmin):  list_display = (' LogFile ', ' filename ',)  list_filter = [' filename ',]  def save_model (self, Request, obj, form, change):    re = super (ydimportlogadmin,self). Save_model (Request, obj, form, change)    update _svn_log (self, request, obj, change)    return re

Note the Save_model in the above code, here is the key, here I rewrite the modeladmin in the Save_model method
Because we want to upload files, read files, parse files, operation database in one step to operate, we can open debug, when uploading files, the return parameter of obj includes the file upload path, this path is the next step of our operation parsing file key, All right, let's create a new utils.py to manipulate our files and databases in this app folder, and for the sake of simplicity I wrote the function as follows
Put the XML file we want to test first.

qwert2016-09-27t07:16:37.396449z/aaa/readme20160927 151630VisualSVN server2016-09-20t05:03:12.861315z/branches/ Tags/trunkhello Word

Output result format

R2 | Qwer | 2016-09-27 15:16:37 +0800 (two, 27 9 2016) | 1 linechanged paths:a/xxx/readme20160927 151630------------------------------------------------------------------ ------R1 | VISUALSVN Server | 2016-09-20 13:03:12 +0800 (two, 20 9 2016) | 1 linechanged paths:a/branches a/tags a/trunkinitial structure.from. Models Import svnlogimport xmltodictdef update _svn_log (self, request, obj, change): headers = [' R ', ' a ', ' d ', ' m ', ' P '] filepath = obj. Logfile.path xmlfile = xmltodict.parse (open (filepath, ' R ')) Xml_logentry = Xml.get (' log '). Get (' logentry ') info_list = [ ] PathList = [] Sql_insert_list = [] Sql_update_list = [] for j in xml:data_dict = {} # Get path paths = J.G ET (' paths '). Get (' path ') if Isinstance (paths,list): for path in paths:action = Path.get (' @action ') p Athtext = Path.get (' #text ') Pathtext = action + ' + pathtext pathlist.append (pathtext) _filel ist = u ' \ n '. Join (pathlist) _paths = u "Changed paths:\n {} ". Format (_filelist) print _paths else: _filelist = paths.get (' @action ') + ' + paths.get (' #text ') _p aths = u "Changed paths:\n {}". Format (_filelist) Print _paths # get revision vision = J.get (' @vision ') # get Auth author = j.get (' author ') #get Date date = J.get (' date ') #get msg msg = j.get (' msg ') Data_dict[heade RS[0]] = Int (vision) data_dict[headers[1] [= author data_dict[headers[2]] = Date Data_dict[headers[3]] = msg D ATA_DICT[HEADERS[4]] = _paths info_list.append (data_dict) _svnlog = SVNLog.objects.filter (). order_by ('-vision '). First () _last_version = _svnlog.vision If _svnlog else 0 for value in info_list:vision = Value[' r '] Author = valu E[' a '] date = value[' d '] msg = value[' m '] paths = value[' P '] print vision,author _svnlog = ydsvnlog.objects . filter (). order_by ('-revision '). First () _last_version = _svnlog.revision If _svnlog else 0 if vision > _last_vers Ion:sql_insert_list.append (Svnlog (revision=revision, Author=author, date=date, msg = msg, paths = paths)) Else:sql_update_list.append (SVN  Log (Revision=revision, Author=author, date=date, msg = msg, paths = paths)) SVNLog.objects.bulk_create (sql_insert_list) SVNLog.objects.bulk_create (Sql_update_list)

We used Xmltodict, a third-party repository to parse the XML, and he parsed the content into a high-efficiency orderdict type, which is a dictionary of sequences.
This XML is more complicated is the path in the paths, because this XML contains two elements, the first element of the path contains only one path, the second element paths contains three paths, so we need to determine when parsing the acquisition

paths = J.get (' paths '). Get (' path ') if Isinstance (paths,list):  Pass

We determine if this path is a list type, and if so, then we will do the list, if not, then we will handle it in a single way, get the results in the output format and then get the other content.

Revision = J.get (' @vision ') # Get Authauthor = J.get (' author ') #get datedate = j.get (' date ') #get msgmsg = J.get (' msg ')

Finally we will get to the element that exists in the dictionary
In the loop, determine the current version number and the version number in the database.
If it is smaller than the original, then we perform the update operation, and vice versa, insert operation

Finally, Bulk_create is used to operate the database, which avoids the waste of resources caused by the database operation every time in the loop.

