A simple incremental update strategy for data (Android/mongodb/django)

I encountered a problem when I was doing a personal App-caykanji:

How to incrementally update the kanji data from the server to the app side, that is, each time the user runs the update operation, only gets the version number higher than the locally cached content.

Data format

In order to be able to seamlessly integrate with MongoDB, and eliminate the hassle of writing background code, simply save the Chinese character data into a JSON file, uploaded to the server. To the web app to read and write to the database.

The kanji file is the normal JSON format.

{    "category": "Line for ー2",    "Contents": [        {            "kanji": "Search",            "On-yomi": "ジン",            "Kun-yomi": "たず-ねる",            "Examples": ["Unknown な point ねる", "the line to ねて back る", "The origin of ねる", "police officer が strange しい Male occupies to ask able", "often ではない action"]        },        {            "kanji": "Promote", "            On-yomi": "ソク", "            Kun-yomi": "うなが-す",            "examples": ["grow to promote Bamboo blind", "souel words to promote bamboo blind", "sellers ó to promote able", " Support Payment いを urging able "]        }    ]}

Database operations

Then use a different directory to make a queue, files that are not written to the database are placed in the pending directory. Put in the finished directory that has been written.

This way, when Webserver receives the first update request, it detects if there is a new JSON file in the Pengding directory and writes to the database. And move the JSON file to the finished directory.

The web side uses the Django framework, using the pymongo operational database.

Data Write

def write_to_db (self, Path): With    open (path) as My_file:        Self._db.kanji.insert (Json.loads (My_file.read ()))

def check_update (self):    for (path, dirs, files) in Os.walk (self. Pending_dir): For        file in Files:            self.write_to_db (os.path.join (path, file))            # move sync-over file into ' Finished ' directory            shutil.move (os.path.join (path, file), Os.path.join (self). Finished_dir, file))

Once the JSON data is synchronized to the database, it is possible to initiate a request on the Android side to fetch the data.

But. This does not enable incremental updates on the phone, and each request will download all the data in the database. In fact the solution is also very easy. adding a version number to each JSON data is done .

Version number control for incremental data

The JSON file before the change. Add a version field.

{    "category": "Line for ー2",    "version": 0,    "contents": [        {            "kanji": "Search",            "On-yomi": "ジン",            " Kun-yomi ":" たず-ねる ",            " examples ": [" Unknown な point of ねる "," the line to ねて back る "," Origin of ねる "," police が strange しい male occupies to ask able "," ではない Action "        },        {            "Kanji": "Promote", "            On-yomi": "ソク", "            Kun-yomi": "うなが-す",            "examples": ["grow to promote Bamboo blind", "souel words to promote bamboo blind", "sellers ó to promote able", " Support Payment いを urging able "]        }    ]}

Data read

RESTfullURL of the data request style-http://www.liangfeizc.com/catykanji/download/0

The number following the download indicates the minimum version of the request, and the server will return all data with a version greater than or equal to 0 when the request is received.

First, add a URL rule to url.py

URL (r ' ^download/(? p<version>\d+) $ ', ' Download_kanji '),

Then add the function to the view.py download_kanji

def Download_kanji (Request, version=0):    kanji = _mongo.get_kanji (int (version))    return HttpResponse (Kanji, Content_type= "Application/json")

Read data greater than or equal to version

def Get_kanji (self, version):    # Check pending DIR-see if there's any file    self.check_update ()    result = Self . _db.kanji.find ({"version": {"$gte": Version}})    return json_util.dumps (Result)

Fixed incremental update rules on the web side. There's nothing wrong with Android.

Android side

The Android HTTP request uses the AOSP Volley library, just to splice the URI well, add to the volley requestqueue can be.

Each time the request data is returned from the server, the value of the largest version is added 1 after parsing the JSON file and saved to preference. The version number is read directly from the preference when the request is sent again. Stitching into a URL that requests a specified version number will be possible.

Finally, the parsed JSON data is divided into three tables saved to the database on the big announcement. Source code can see

Https://github.com/LyndonChin

A simple incremental update strategy for data (Android/mongodb/django)