A simple incremental update strategy for data (Android/mongodb/django)

I encountered a problem when I was doing a personal App-caykanji:

How to incrementally update the kanji data from the server to the app side, that is, each time the user performs an update operation, only the content that is newer than the local cache is obtained.

Data format

In order to be able to seamlessly integrate with MongoDB, and eliminate the hassle of writing background code, simply save the Chinese character data into a JSON file, upload to the server, to the Web application to read and write to the database.

The kanji file is the normal JSON format.

{    "category": "Line for ー2",    "Contents": [        {            "kanji": "Search",            "On-yomi": "ジン",            "Kun-yomi": "たず-ねる",            "Examples": ["Unknown な point ねる", "the line to ねて back る", "The origin of ねる", "police officer が strange しい Male occupies to ask able", "often ではない action"]        },        {            "kanji": "Promote", "            On-yomi": "ソク", "            Kun-yomi": "うなが-す",            "examples": ["grow to promote Bamboo blind", "souel words to promote bamboo blind", "sellers ó to promote able", " Support Payment いを urging able "]        }    ]}

Database operations

Then use different folders to make a queue, files that are not written to the database are placed in the pending folder, and are already written in the finished folder. This way, when the Web server receives the first update request, it detects if there is a new JSON file under the Pengding folder, writes to the database, and moves the JSON file to the finished folder.

The web side uses the Django framework, using the pymongo operational database.

Data Write

def write_to_db (self, Path): With    open (path) as My_file:        Self._db.kanji.insert (Json.loads (My_file.read ()))

def check_update (self):    for (path, dirs, files) in Os.walk (self. Pending_dir): For        file in Files:            self.write_to_db (os.path.join (path, file))            # move sync-over file into ' Finished ' directory            shutil.move (os.path.join (path, file), Os.path.join (self). Finished_dir, file))

Once the JSON data is synchronized to the database, it is possible to initiate a request on the Android side to fetch the data. However, this does not enable incremental updates on the phone, and each request will download all the data in the database. In fact, the solution is also very simple, to each JSON data added to a version of the fix .

Version control of incremental data

Modify the previous JSON file to add a version field.

{    "category": "Line for ー2",    "version": 0,    "contents": [        {            "kanji": "Search",            "On-yomi": "ジン",            " Kun-yomi ":" たず-ねる ",            " examples ": [" Unknown な point of ねる "," the line to ねて back る "," Origin of ねる "," police が strange しい male occupies to ask able "," ではない Action "        },        {            "Kanji": "Promote", "            On-yomi": "ソク", "            Kun-yomi": "うなが-す",            "examples": ["grow to promote Bamboo blind", "souel words to promote bamboo blind", "sellers ó to promote able", " Support Payment いを urging able "]        }    ]}

Data read

RESTfullURL of the data request style-http://www.liangfeizc.com/catykanji/download/0

The number following the download indicates the minimum version number of the request, and the server will return all data with a version number greater than or equal to 0 when it receives the request.

First, add a URL rule in url.py

URL (r ' ^download/(? p<version>\d+) $ ', ' Download_kanji '),

Then add the function in the view.py download_kanji

def Download_kanji (Request, version=0):    kanji = _mongo.get_kanji (int (version))    return HttpResponse (Kanji, Content_type= "Application/json")

Read data greater than or equal to version

def Get_kanji (self, version):    # Check pending DIR-see if there's any file    self.check_update ()    result = Self . _db.kanji.find ({"version": {"$gte": Version}})    return json_util.dumps (Result)

Fixed the web side of the incremental update rules, Android is no problem.

Android side

The Android HTTP request uses the AOSP Volley Library, as long as the URI is spliced well, added to the volley requestqueue.

Each time the request data is returned from the server, after parsing the JSON file, the value of the largest version is added 1 and saved to preference, which is read directly from preference when the request is sent again. Stitching into a URL that asks for the specified version number is available.

Finally, the parsed JSON data into three tables saved to the database on the big announcement became, source code can see

Https://github.com/LyndonChin

A simple incremental update strategy for data (Android/mongodb/django)