Django Bulk Import XML Data _python

Django Background Bulk Import data

In the production environment, often the data is not a few or hundreds of, so for example, the company all employee employee number or account password into the background, it does not suggest you go backstage a record to add

How to Bulk Import SVN records from XML

First step:

Building a model for data

@python_2_unicode_compatible
class Svnlog (models. Model):

  vision = models. Integerfield (verbose_name=u "revised Version", Blank=false, Null=false,)
  author = models. Charfield (verbose_name=u "Author", Max_length=60, Blank=true, null=true)
  date = models. Datetimefield (verbose_name=u "revision Time", null=true)
  msg = models. TextField (verbose_name=u "annotation Message", Blank=false, Null=false, Default=u "")
  paths = models. TextField (verbose_name=u "Affected Documents", Blank=false, Null=false, Default=u "")
  Created_time = models. Datetimefield (verbose_name=u "Creation Time", Auto_now_add=true,)
  Update_time = models. Datetimefield (verbose_name=u "Modify Time", Auto_now=true,)

  class Meta:
    ordering = [' revision ']

  def __str__ ( Self): return
    u ' r%s '% (self.revision or U "",)

Now that we've built the model, we're going to build a models that accepts our XML files.

@python_2_unicode_compatible
class Importlogfile (models. Model):

  LogFile = models. Filefield (upload_to= ' LogFile ')
  FileName = models. Charfield (max_length=50, verbose_name=u ' filename ')

  class Meta:
    ordering = [' filename ']

  def __str__ (self): return
    self. FileName

OK, the above code we have defined the data and upload the file model

Synchronizing databases

Python manage.py makemigrations
python manage.py migrate

Then we go to modify admin.py so that we can upload files from the background,

Class Importlogadmin (admin. Modeladmin):

  list_display = (' LogFile ', ' filename ',)
  list_filter = [' filename ',]

  def save_model (self, Request, obj, form, change):

    re = super (ydimportlogadmin,self). Save_model (Request, obj, form, change)
    update _svn_log (self, request, obj, change) return
    re

Note that the Save_model in the code above, here is the key, where I rewrite the modeladmin Save_model method
because we want to upload files, read files, parse files, operation database together for one step to operate, you can open debug, In the upload file, the return parameter obj included the file upload path, this path is the next step we operate the key to parse the file, OK, we have a new utils.py in this app folder to operate our Operations files and Database tool class, in order to simple I wrote a function as follows
Paste the XML file we want to test first

<?xml version= "1.0" encoding= "UTF-8"?>
<log>
<logentry
  revision= "2" >
< author>qwert</author>
<date>2016-09-27T07:16:37.396449Z</date>
<paths>
<path
  action= "A"
  prop-mods= "false"
  text-mods= "true"
  kind= "file" >/aaa/readme
  </ path>
</paths>
<msg>20160927 151630</msg>
</logentry>


< LogEntry
  revision= "1" >
<author>visualsvn server</author>
<date> 2016-09-20t05:03:12.861315z</date>
<paths>
<path
  action= "A"
  prop-mods= " False "
  text-mods=" false "
  kind=" dir ">/branches</path>
<path
  action=" A "
  Prop-mods= "false"
  text-mods= "false"
  kind= "dir" >/tags</path>
<path
  action= "A"
  prop-mods= "false"
  text-mods= "false"
  kind= "dir" >/trunk</path>
</paths>
<msg>hello word</msg>
</logentry>
</log>

Output result format

R2 | Qwer | 2016-09-27 15:16:37 +0800 (two, 27 9 2016) | 1 line Changed paths:a/xxx/readme 20160927 151630------------------------------------------------------------------ ------R1 | VISUALSVN Server | 2016-09-20 13:03:12 +0800 (two, 20 9 2016) |
1 line Changed paths:a/branches a/tags a/trunk Initial. Import svnlog Import xmltodict def update_svn_log (self, request, obj, change): headers = [' R ', ' a ', ' d ', ' a ', ' models ', ' m ', ' P '] filepath = obj. Logfile.path xmlfile = xmltodict.parse (open (filepath, ' R ')) Xml_logentry = Xml.get (' log '). Get (' LogEntry ') info_list = [] PathList = [] Sql_insert_list = [] Sql_update_list = [] for j, xml:data_dict = {} # Get path P Aths = J.get (' paths '). Get (' path ') if Isinstance (paths,list): for path in paths:action = Path.get (' @acti
        
      On ') Pathtext = Path.get (' #text ') Pathtext = action + ' + pathtext pathlist.append (pathtext) _filelist = U ' \ n '. Join (PATHLIST) _paths = u "Changed paths:\n {}". Format (_filelist) print _paths else: _filelist = Paths.get (' @ac tion ') + ' + paths.get (' #text ') _paths = u "Changed paths:\n {}". Format (_filelist) Print _paths # Get Rev
    Ision vision = j.get (' @vision ') # get Auth author = j.get (' author ') #get Date date = J.get (' Date ') #get msg msg = j.get (' msg ') data_dict[headers[0]] = Int (vision) data_dict[headers[1]] = author Data_dict

  [headers[2]] = Date Data_dict[headers[3]] = msg data_dict[headers[4]] = _paths info_list.append (data_dict) _svnlog = SVNLog.objects.filter (). order_by ('-vision '). A () _last_version = _svnlog.vision If _svnlog else 0 for VA Lue in info_list:vision = value[' r '] Author = value[' a '] date = value[' d '] msg = value[' m '] paths = V alue[' P '] print vision,author _svnlog = YDSVNLog.objects.filter (). order_by ('-revision '). (a) _last_version = _svnlog.revision If _sVnlog else 0 if vision > _last_version:sql_insert_list.append (Svnlog (revision=revision, Author=author, date= Date, msg = msg, paths = paths)) Else:sql_update_list.append (Svnlog (revision=revision, Author=author, date=date

 , msg = msg, paths = paths)) SVNLog.objects.bulk_create (sql_insert_list) SVNLog.objects.bulk_create (sql_update_list)

We use the Xmltodict this Third-party library to parse XML, and he parses the content into an efficient orderdict type, a dictionary with sequences.
This XML is more complex than the path in that paths, because this XML contains two elements, the first element of the path contains only one path, the second element of the paths contains three path, so we need to determine when parsing acquisition

paths = J.get (' paths '). Get (' path ')
if Isinstance (paths,list):
  Pass

We're going to determine if this path is a list type, and if so, then we'll do it in the list way, if not, then we'll do it in a single way, get the results in the output format, and then get something else.

Revision = J.get (' @vision ')
# get auth
author = j.get (' author ')
#get date
date = J.get (' date ')
#get msg
msg = j.get (' msg ')

Finally we will get the element to exist in the dictionary
In the loop, determine the current version number and the version number in the database,
If it is smaller than the original, then we perform the update operation, instead of performing the insert operation

Finally, the bulk_create is used to manipulate the database, which avoids the waste of resources caused by database operation every time in the loop.