About the download Gae high Replication datastore data [actual combat article]

Through the bulk loader can upload the download data, GAE support xml,csv format data bulk upload, as well as xml,csv and text format download.

You can choose to automatically generate a bulkloader.yaml, or manually code to write a bulk loader. Google does not recommend manual writing, the individual also feel that automatic generation is kingly.

How to generate Bulkloader.yaml automatically

Bulkloader.yaml is a configuration file that describes the format of a data, such as XML or CSV. Bulk loader needs this profile to convert external data (XML or CSV) to intermediate data--python dictionary types for data import, and then converts the intermediate data--python dictionary to Engine data in Google datastore entity. When downloading data, the conversion order is reversed.

Perform the following command to automatically generate BULKLOADER.YAML:

$ appcfg.py create_bulkloader_config--filename=bulkloader.yaml--Url=http://your_app_id.appspot.com/_ah/remote_api

This command will create a Bulkloader.yaml file in the current directory, and generally you will have to configure the original Bulkloader.yaml file to meet your data transfer requirements.

Configuring Bulkloader.yaml Files

The Bulkloader.yaml file consists mainly of two parts, Python_preamble and Transformers.

Python_preamble:
-import:google.appengine.ext.bulkload.transform
-import:google.appengine.ext.db
- Import:re
-Import:base64

In general, you don't have to change this part unless you want to use other modules or methods in the conversion format, for example, I added a datetime here.

Python_preamble:
-import:base64
-import:re
-Import:google.appengine.ext.bulkload.transform
- Import:google.appengine.ext.bulkload.bulkloader_wizard
-import:google.appengine.ext.db
-Import: Google.appengine.api.datastore
-import:google.appengine.api.users
-Import:datetime

The second part transformers is used to describe the data type, which contains one or more kind depending on the number of kind you have in datastore. Each kind includes the following:

Connector: Used to specify the input and output format, with a value of xml,csv or text.

Some properties of the connector_options:connector.

Property_map: The mapping relationship for each field (that is, the transformation relationship).

The first step in configuring Bulkloader.yaml is to set up connector, the default is CSV format, regardless of the input output requires the first row of CSV is the field name. Connector is: simplexml, CSV, SimpleText.

Connector configuration parameters can be found in Google Docs.

In the case of property:

The name of the field in the Property:datastore.

The tag name of the External_name:xml, or the field name in the first line of CSV.

Import_transform: Data format when uploading, or data conversion rules.

Export_transform: Data format when downloading, or data conversion rules.

Property and External_name are necessary, and if you just want to upload or download, you can comment out Export_transform or import_transform.

About SimpleXML

The SimpleXML in Bulkloader.yaml currently supports only simple XML constructs, such as:

<book>
  <article>
    <author>john kerry</author>
    <title>how is it, china< /title>
    <content>yes, there is a huge population.</content>
  </article>
</book >

Configured in Bulkloader.yaml to:

Connector:simplexml
connector_options:
  xpath_to_nodes:/book/article
  style:element_centric