Extend SQLite to enable it to read db from the APK file

The game will use a lot of configuration files, each project group according to their own different needs will choose different storage format, such as using JSON or SQLite to store data. Here we only discuss the situation of using SQLite. In general, you will choose to put it in a read-write directory, so that SQLite can directly use its original IO API to read the DB file. This is not a problem on a PC or iOS platform. However, if the game is installed on an Android platform, it will still be in the form of an apk file. If our data is in db, the IO features originally brought with sqlite are not readable. There are 3 ways to choose from:

1. When the program first starts, unzip all the files in the APK into the read-write directory, the problem is that the first time you open the program will be slow.
2. This file is extracted from the apk when it is needed to read a certain db, which may result in a lag, or use asynchronous operations such as a co-process, but it is expensive to write code for the logical layer.
3. Make certain changes to sqlite so that it can read the db file in the APK.

Generally you may choose the first method, there is nothing to say. Let's look at the possibility of a 3rd approach.

Theory

To achieve the above idea, we need two conditions:

1. Android provides the ability to read individual files in the APK.
2. SQLite provides the ability to quickly modify the IO layer (on different platforms), which requires SQLite to have a better abstraction with fewer coupling between modules.

Of course, the above two conditions are satisfied, let's look at these two conditions specifically.

Android's ability to read individual files in apk

Open a new file descriptor that can is used to read the asset data. If the start or length cannot is represented by a 32-bit number, it'll be truncated. If The file is a large, use AASSET_OPENFILEDESCRIPTOR64 instead. Returns < 0 if direct FD access is not a possible (for example, if the asset is compressed). int Aasset_openfiledescriptor (Aasset * Asset, off_t * outstart, off_t * outlength)

As you can see from this API, it can return a file descriptor for reading the current asset. However, if the current asset is compressed, it returns a value less than 0. If we want to read the DB, then it has to be uncompressed.

The sample code is roughly as follows:

aasset* asset = Aassetmanager_open (MGR, filename, aasset_mode_unknown);     if (NULL = = Asset)     {         //logd ("File not found! Stop preload file:%s ", filename);         return File_not_found;     }     //Open Asset as file descriptor     int FD = aasset_openfiledescriptor (Asset, &start, &length);     assert (0 <= FD);     aasset_close (Asset);

Note that this FD returns a handle to the entire APK, and start represents the offset of the file in the APK, length is the duration, and should be noted when using it.

SQLite support for the file IO layer

is the official SQLite architecture diagram:

Our main concern here is the OS interface layer, which uses the VFS object to provide a guarantee for portability between different systems. Specific details can be referred to the official website for the introduction of VFS. SQLite currently provides support for UNIX-like systems and Windows systems, which are implemented in OS_UNIX.C and OS_WIN.C, respectively. OS_UNIX.C provides support for Mac OS, IOS, Android, and Linux. If reading wants to have a more in-depth understanding of this piece, you can look at official documents and see some examples such as test_demovfs.c for more insight.

Realize

With the support of the above theory, we can begin to write code. We currently have two options to choose from:

1. Implement a VFS for Android alone.
2. Modify it on the basis of OS_UNIX.C.

The first scenario requires a lot of code to write, and requires readers to have a more in-depth understanding of SQLite. So we choose the second option here. Change on the basis of the original OS_UNIX.C.

SQLite makeover

By analyzing the os_unix.c file, we can conclude that it mainly uses the open () read () write () and other IO operations. This is exactly the perfect combination of the aasset_openfiledescriptor provided by our Android NDK above. We can do something like this just by using the handle it returns. Only in the case of Android is a special point.

So down, we basically need to change a few of the functions and structure as shown below

1. struct Unixfile We need to add the offset and size of the file in the APK.
2. Posixopen returns the file descriptor via Openfiledescriptor.
3. Seekandread to add a file offset when reading.
4. Unixfilesize returns the value of the file size recorded in the previous unixfile.

The above basic is needed to change the function, of course, depending on the implementation of the different may be specific needs to change the function is not the same. It's just a rough fix. Like we need to support reading from the APK and reading from a piece of prose, so there are some different places than the above changes, but the basic idea is to pass. Of course, because I do not understand sqlite, there may be a need to change the place did not notice, if there is a mistake to hope to correct in time. The method has been said to be more clear, here also do not put code.

The above example mentions that Aassetmanager_open needs a Aassetmanager object when it is opened, which can only be retrieved from Java. If you are using Android directly then this object is easier to get, so if you are using unity or UE4 to develop how to get this object.

Unity Implementation Details

The changes to SQLite are the same as above, but how do we get this object in unity? Readers can compare this class Androidjni Androidjnihelper androidjavaclass androidjavaobject androidjavaproxy.

The sample code looks like this:

IntPtr cls_activity = (IntPtr) androidjni.findclass ("Com/unity3d/player/unityplayer"); IntPtr fid_activity = Androidjni.getstaticfieldid (cls_activity, "currentactivity", "landroid/app/activity;"); IntPtr obj_activity = Androidjni.getstaticobjectfield (cls_activity, fid_activity); IntPtr obj_cls = Androidjni.getobjectclass (obj_activity); IntPtr Asset_func = Androidjni.getmethodid (Obj_cls, "Getassets", "() Landroid/content/res/assetmanager;"); jvalue[] Asset_array = new jvalue[2]; <-? IntPtr Assetmanager = Androidjni.callobjectmethod (obj_activity, Asset_func, Asset_array);

So we get this assetmanager, and this time we can pass this object to the SQLite library through C #.

UE4 Implementation Details

UE4 in C + + can directly get Aassetmanager object, specific implementation details UE4 has helped us, specifically can see the code in AndroidJNI.cpp. We get the Aassetmanager object and set it to SQLite.

Summarize

In this case, we have written the method for reading the DB in the APK extension to SQLite extensions. Since the Android NDK returned the file descriptor and the OS interface layer provided by SQLite makes it easier to implement the extended to SQLite. Because the author does not understand SQLite originally, so there are inevitably mistakes, if there are errors please correct me. If readers want to have a more in-depth understanding of SQLite, you can also look at reference article 6.

Reference article:

1. Http://sqlite.org/arch.html
2. Http://www.sqlite.org/vfs.html
3. https://developer.android.com/ndk/reference/group___asset.html#ga1af4ffd050016e99961e24f550981677
4. Https://docs.unity3d.com/560/Documentation/ScriptReference/AndroidJNI.html
5. Http://answers.unity3d.com/questions/205212/android-file-open.html
6. Http://huili.github.io/summary/READMEG1.html

Extend SQLite to enable it to read db from the APK file