About the download Gae high Replication datastore data [actual combat article]

Objective

In part of this article Google's official document does not state that, according to Google Docs and actual combat experience, Google's current Replication datastore data download support is not stable, but Google said in the future will improve.

From writing this article to the implementation of this article, Google Engine Launcher to the Google has an upgrade to 1.7.7.695. So there is no guarantee that all content in this article will apply to the latest Google engine.

[note] Google's data upload download document is targeted at Master/slave Datastore, and this article is based on high Replication datastore. Since I have not used master/slave datastore, I do not know what Google's original description of Master/slave datastore is effective. If you use Master/slave Datastore, please refer to the original text first, then refer to this article.

[note] Google's commands for Master/slave Datastore, some of which are not successful at execution, are referenced in this article using red Fonts for commands that are invalid for the high Replication datastore.

Body

Google to use a call bulk loader tool data upload download, to want to upload from Google's datastore download data, the key is to configure bulk loader. Federated (OpenID) authentication is not supported when passing data through the bulk loader, and is currently mainly supported by Master/slave Datastore, for high Replication Datastore, Google said that users can get data from the high Replication datastore, but currently do not support, if you try to download data from the high Replication datastore will appear in the Administrator console High_ Replication_warning error, and does not contain the most recently saved entities.

However, according to personal experience, in fact, currently can be downloaded from the high replication datastore data, there is no high_replication_warning error, but sometimes can be successful, sometimes failed, sometimes downloaded data format is not correct, Sometimes the download data is incomplete, so it can be seen that Google's support for downloading data from high replication datastore remains precarious.

Uploading or downloading data from Gae Datastore is divided into three steps:

Open the data transfer interface REMOTE_API: Must open the Remote_api interface to carry out data transmission, open the interface can be all the data as a whole upload download. (Do not configure Bulkloader.yaml)

Configuration data Structure description file Bulkloader.yaml: You must configure and use Bulkloader.yaml if you want to download the kind specified in the Datastore, and then deliver it in XML or CSV format.

Upload or download data: Through the command line, follow the data format description in Bulkloader.yaml to upload the download data.

Open Remote_api

Google engine via the Remote_api interface to communicate with apps deployed in Google Engine, and Google engine allows legitimate requests to remotely access app Remote_api via the Datastore interface.

There are two configuration methods for opening REMOTE_API, using the built-in instruction (Builtins directive) or the URL directive (URL directive).

1. Open Remote_api with built-in instructions

In the app folder, edit the App.yaml and add the following code:

Builtins:

-Remote_api:on

This command directs the app to look for the Remote_api interface in the Include.yaml file and maps this interface to/_AH/REMOTE_API, which is accessible only to administrators.

But Google has not told you that your app folder is likely to have no include.yaml files. So you need to create a Include.yaml file that reads as follows:

Handlers:

-url:/_ah/remote_api (/.*)?

Script: $PYTHON _lib/google/appengine/ext/remote_api/handler.py

Once the above configuration is complete, use Google Engine launcher to redeploy once, and if launcher is not installed, you can use the following command to deploy the update. When the deployment is complete, the REMOTE_API is opened.

appcfg.py Update <app-directory>

2. Open Remote_api with URL instruction

The advantage of using a URL directive is that you can map Remote_api to a personalized URL, and your access address must be/_AH/REMOTE_API when you use the built-in instructions. or edit the App.yaml file, add the following:

-URL:/remote_api

Script: $PYTHON _lib/google/appengine/ext/remote_api/handler.py

Login:admin

In the first line of the URL, you can configure the URL you want as an interface access address. Of course, this interface is still accessible only to administrators. Once the configuration is complete, redeploy the application to update the server-side app.yaml files.

[Note] Google alerts using a personalized URL to access the Remote_api interface, you need to advance in the/_ah/remote_api path installed REMOTE_API handler. Because I started with the built-in command to open Remote_api, so REMOTE_API handler may be installed at this time, if you start using the URL instructions but failed to run, you may want to use the built-in instructions to open the REMOTE_API, Then use the URL directive instead to personalize your access address.