Analysis on incremental upgrade of android applications

By
He minggui (http://blog.csdn.net/hmg25) reprinted please indicate the source

I haven't updated my blog for a long time. It's a fallen phenomenon. I tried to write a blog several times, but I was always delayed by all kinds of things. I will write more articles to record it later.

Background with the increasing volume of android applications and the changing release of application versions, the upgrade of users has become a problem. google is also aware of the loss of user traffic caused by constantly updating applications, the Smart App update mentioned on Google I/O, that is, the incremental update or differential update of an application, is supported in the new version of Google Play, chatting with a group of friends one day is a topic in this area. I just studied it with curiosity. The principle of incremental upgrade is to achieve incremental upgrade of similar applications today. In fact, the principle of incremental upgrade is very simple, that is, first, the old Apk version of the application is different from the new Apk version to obtain the updated part of the patch, for example, the old APK version has 5 MB, the new version has 8 M, and the updated part may only be around 3 M (it should be noted that the size of the differential package is not a simple subtraction, because it actually needs to include some context-related things), the advantage of using a differential upgrade is obvious, so you don't need to download the entire 8 M file, you just need to download the update part, the updated part may only be 3 or 4 M, which can greatly reduce the traffic loss. After the user downloads the differential package, they need to be combined on the mobile phone end. You can refer to the method of copying the old software (mostly under/data/) on the mobile phone to the SD card or cache and combining them with the previous differential patch, get a new version of the apk application. If nothing happens, the generated apk is the same as the one you used to make the difference. After learning about the basic principles of incremental upgrade, we will gradually solve various difficulties. The first is the generation of the patch of the differential package. If the Android mobile phone otaupgrade is completed, the patch folder in update.zip contains files with the same name as the system file but suffixed with xxx. p. They are the generated differential patch files. We can use the differential generation tool for OTA system upgrade to generate the differential patch file for a single application apk. The following command is used to create the OTA system differential package:

./Build/tools/releasetools/ota_from_target_files-n-I <old package> <new package> <differential package name>

According to the ota_from_target_files code, the difference package is generated in the writeincrementalotapackage function, and the common is created in this function. difference class, we continue to follow up, in the common. class difference (object): In py, you can see:

    diff_program = DIFF_PROGRAM_BY_EXT.get(ext, "bsdiff")

Now we can see that android provides the tool we use to create a differential incremental update package, "bsdiff", which is an open-source binary differential tool. introduction to the Code address related to the portal or \ external \ bsdiff is a binary difference tool under the android code directory, the corresponding bspatch is the patch synthesis tool. Note that the incremental update Patch must be completed on the server side, that is, on the PC side. The general process is as follows, call the bsdiff function when creating a patch, and generate a patch file based on two binary files of different versions.

Command: bsdiff oldfile newfile patchfile For example: bsdiff xx_v1.0.apk xx_v2.0.apk xx. patch

Place the generated patch package XX. patch on the upgrade server for users to download and upgrade. Different versions must be different for different versions. if the version span is large, we recommend that you upgrade the entire package. After downloading the XX. Patch patch package, you need to use the APK corresponding to the patch, that is, the old apk installed in the system and the bspatch tool integrated with the patch. The earlier version of the APK can be obtained through the APK file under the Copy System Data/APP directory, and the bspatch patch synthesis can be encapsulated into a so library by slightly modifying the bspatch source code, for mobile phone calls.

Bspatch Command Format: bspatch oldfile newfile patchfile

The difference is the same as that of time-sharing. A new APK can be merged for installation. The above is just a simple operation principle. The incremental upgrade also involves many other aspects, such as upgrading patch verification. For details, refer to the bootable \ recovery \ applypatch operations in the android source code, this article is only an analysis, not a table here. Insufficient incremental upgrade is not a perfect upgrade method. There are at least two shortcomings: 1. incremental updates generate patches based on the differences between the two application versions. You cannot guarantee that the updates are updated to the latest version every time, therefore, you must make a difference between each version you release and the latest version so that users of all versions can perform a differential upgrade. This operation is more complicated than the original full-package upgrade, however, it can be generated in batches using automated scripts. 2. the premise for successful incremental upgrade is that the user's mobile phone must have an APK that allows you to copy and is consistent with the version used by your server for the differential version. This will exist, for example, the built-in APK of the system cannot be obtained and incremental upgrades cannot be performed. For some of the APK versions that are consistent with your differential version but have modified the content (such as the cracked APK version), incremental upgrades cannot be performed, to prevent merging patch errors, it is best to verify the sha1sum of the old APK version before merging the patch to ensure the consistency of the basic package. The experiment is useless, and practice is king. The following is a simple practice to check the correctness of the previous theory. Download the lab package (http://download.csdn.net/detail/hmg25/4676737), unzip the file as follows

├ ── Bsdiff-4.3 // bsdiff source code path, official website to obtain │ ── bsdiff.1 │ ├ ── bsdiff. c │ ── bspatch.1 │ ── bspatch. c │ make── makefile ── bsdiff-4.3.tar.gz ├ ── bsdiff4.3-Win32 // Windows PC end test tool │ binary ── binary diff.txt │ ├ ── bsdiff.exe │ ── bspatch.exe │ └ ── license ── bspatch // test tool ── ireader1.6.2.0(v352.16.apk // The old APK └ ── ireader1.8.0.1(v402.16.apk // The New APK version

Run the following command in the shell to enter the test \ bsdiff4.3-Win32 Folder:

   bsdiff.exe   ../iReader1.6.2.0(v35).apk   ../iReader1.8.0.1(v40).apk   ../ireader.patch

In the original APK (2.94 m) and new version (3.24 m), the resulting patch file is 1.77 m. Users only need to download 1.77 m, saving a lot of traffic.
Next we will merge them on the computer.

  bspatch.exe  ../iReader1.6.2.0(v35).apk   ../new.apk    ../ireader.patch

After the execution, the app named new.apk is merged. I verify it under ubuntu and we can see that they are the same.

Category and ireader. patch, push through adb to the directory you have permission to operate, preferably under/sdcard/, and then set the bspatch execution permission, repeat the above command, you can combine the new version of apk, install and view the verification version later. Extended reading uses bsdiff for differential upgrade, which is not the optimal method yet. google optimized the difference algorithm in its Chromium project. The optimized version is called huagua Courgette, it is said that performance optimization is not an order of magnitude. An official example is as follows:
Full update 10,385,920 bsdiff update
704,512 Courgette update
78,848
You can study it.
I am a little busy recently. I will try to encapsulate the incremental upgrade later and compile the Merged Code into a lib library for java to call. If you are interested in shoes, perform the following operations on your own ~~~ By
He minggui (http://blog.csdn.net/hmg25) reprint please indicate the source of genuine, pirated copy must be studied ^_^

Supplement: many people do not know how to encapsulate it as Lib. In fact, this is the same as the general Android C library. If you don't understand it, look at JNI-related knowledge, the original test project cannot be found. Below is a simple example for your reference.

 

#include <stdio.h>#include "com_hmg25_newstart_BSpatch.h"#include <bzlib.h>#include <stdlib.h>#include <stdio.h>#include <string.h>#include <err.h>#include <unistd.h>#include <fcntl.h>#include <android/log.h>static off_t offtin(u_char *buf){    off_t y;    y=buf[7]&0x7F;    y=y*256;y+=buf[6];    y=y*256;y+=buf[5];    y=y*256;y+=buf[4];    y=y*256;y+=buf[3];    y=y*256;y+=buf[2];    y=y*256;y+=buf[1];    y=y*256;y+=buf[0];    if(buf[7]&0x80) y=-y;    return y;}int applypatch(int argc,char * argv[]){    FILE * f, * cpf, * dpf, * epf;    BZFILE * cpfbz2, * dpfbz2, * epfbz2;    int cbz2err, dbz2err, ebz2err;    int fd;    ssize_t oldsize,newsize;    ssize_t bzctrllen,bzdatalen;    u_char header[32],buf[8];    u_char *old, *new;    off_t oldpos,newpos;    off_t ctrl[3];    off_t lenread;    off_t i;    if(argc!=4) errx(1,"usage: %s oldfile newfile patchfile\n",argv[0]);    /* Open patch file */    if ((f = fopen(argv[3], "r")) == NULL)        err(1, "fopen(%s)", argv[3]);    /*    File format:        0   8   "BSDIFF40"        8   8   X        16  8   Y        24  8   sizeof(newfile)        32  X   bzip2(control block)        32+X    Y   bzip2(diff block)        32+X+Y  ??? bzip2(extra block)    with control block a set of triples (x,y,z) meaning "add x bytes    from oldfile to x bytes from the diff block; copy y bytes from the    extra block; seek forwards in oldfile by z bytes".    */    /* Read header */    if (fread(header, 1, 32, f) < 32) {        if (feof(f))            errx(1, "Corrupt patch\n");        err(1, "fread(%s)", argv[3]);    }    /* Check for appropriate magic */    if (memcmp(header, "BSDIFF40", 8) != 0)        errx(1, "Corrupt patch\n");    /* Read lengths from header */    bzctrllen=offtin(header+8);    bzdatalen=offtin(header+16);    newsize=offtin(header+24);    if((bzctrllen<0) || (bzdatalen<0) || (newsize<0))        errx(1,"Corrupt patch\n");    /* Close patch file and re-open it via libbzip2 at the right places */    if (fclose(f))        err(1, "fclose(%s)", argv[3]);    if ((cpf = fopen(argv[3], "r")) == NULL)        err(1, "fopen(%s)", argv[3]);    if (fseeko(cpf, 32, SEEK_SET))        err(1, "fseeko(%s, %lld)", argv[3],            (long long)32);    if ((cpfbz2 = BZ2_bzReadOpen(&cbz2err, cpf, 0, 0, NULL, 0)) == NULL)        errx(1, "BZ2_bzReadOpen, bz2err = %d", cbz2err);    if ((dpf = fopen(argv[3], "r")) == NULL)        err(1, "fopen(%s)", argv[3]);    if (fseeko(dpf, 32 + bzctrllen, SEEK_SET))        err(1, "fseeko(%s, %lld)", argv[3],            (long long)(32 + bzctrllen));    if ((dpfbz2 = BZ2_bzReadOpen(&dbz2err, dpf, 0, 0, NULL, 0)) == NULL)        errx(1, "BZ2_bzReadOpen, bz2err = %d", dbz2err);    if ((epf = fopen(argv[3], "r")) == NULL)        err(1, "fopen(%s)", argv[3]);    if (fseeko(epf, 32 + bzctrllen + bzdatalen, SEEK_SET))        err(1, "fseeko(%s, %lld)", argv[3],            (long long)(32 + bzctrllen + bzdatalen));    if ((epfbz2 = BZ2_bzReadOpen(&ebz2err, epf, 0, 0, NULL, 0)) == NULL)        errx(1, "BZ2_bzReadOpen, bz2err = %d", ebz2err);    if(((fd=open(argv[1],O_RDONLY,0))<0) ||        ((oldsize=lseek(fd,0,SEEK_END))==-1) ||        ((old=malloc(oldsize+1))==NULL) ||        (lseek(fd,0,SEEK_SET)!=0) ||        (read(fd,old,oldsize)!=oldsize) ||        (close(fd)==-1)) err(1,"%s",argv[1]);    if((new=malloc(newsize+1))==NULL) err(1,NULL);    oldpos=0;newpos=0;    while(newpos<newsize) {        /* Read control data */        for(i=0;i<=2;i++) {            lenread = BZ2_bzRead(&cbz2err, cpfbz2, buf, 8);            if ((lenread < 8) || ((cbz2err != BZ_OK) &&                (cbz2err != BZ_STREAM_END)))                errx(1, "Corrupt patch\n");            ctrl[i]=offtin(buf);        };        /* Sanity-check */        if(newpos+ctrl[0]>newsize)            errx(1,"Corrupt patch\n");        /* Read diff string */        lenread = BZ2_bzRead(&dbz2err, dpfbz2, new + newpos, ctrl[0]);        if ((lenread < ctrl[0]) ||            ((dbz2err != BZ_OK) && (dbz2err != BZ_STREAM_END)))            errx(1, "Corrupt patch\n");        /* Add old data to diff string */        for(i=0;i<ctrl[0];i++)            if((oldpos+i>=0) && (oldpos+i<oldsize))                new[newpos+i]+=old[oldpos+i];        /* Adjust pointers */        newpos+=ctrl[0];        oldpos+=ctrl[0];        /* Sanity-check */        if(newpos+ctrl[1]>newsize)            errx(1,"Corrupt patch\n");        /* Read extra string */        lenread = BZ2_bzRead(&ebz2err, epfbz2, new + newpos, ctrl[1]);        if ((lenread < ctrl[1]) ||            ((ebz2err != BZ_OK) && (ebz2err != BZ_STREAM_END)))            errx(1, "Corrupt patch\n");        /* Adjust pointers */        newpos+=ctrl[1];        oldpos+=ctrl[2];    };    /* Clean up the bzip2 reads */    BZ2_bzReadClose(&cbz2err, cpfbz2);    BZ2_bzReadClose(&dbz2err, dpfbz2);    BZ2_bzReadClose(&ebz2err, epfbz2);    if (fclose(cpf) || fclose(dpf) || fclose(epf))        err(1, "fclose(%s)", argv[3]);    /* Write the new file */    if(((fd=open(argv[2],O_CREAT|O_TRUNC|O_WRONLY,0666))<0) ||        (write(fd,new,newsize)!=newsize) || (close(fd)==-1))        err(1,"%s",argv[2]);    free(new);    free(old);    return 0;}JNIEXPORT jint JNICALL Java_com_hmg25_newstart_BSpatch_applyPatch(JNIEnv *env,        jobject obj, jstring old, jstring new , jstring patch){  int argc=4;  char * argv[argc];  argv[0]="bspatch";  argv[1]=(*env)->GetStringUTFChars(env,old, 0);  argv[2]=(*env)->GetStringUTFChars(env,new, 0);  argv[3]=(*env)->GetStringUTFChars(env,patch, 0);    int ret=applypatch(argc, argv);     (*env)->ReleaseStringUTFChars(env,old,argv[1]);   (*env)->ReleaseStringUTFChars(env,new,argv[2]);   (*env)->ReleaseStringUTFChars(env,patch,argv[3]);   return ret;}

Android. mk:

LOCAL_PATH:= $(call my-dir)include $(CLEAR_VARS)# This is the target being built.LOCAL_MODULE:= libbspatch# All of the source files that we will compile.LOCAL_SRC_FILES:= \  com_hmg25_newstart_BSpatch.c# No static libraries.LOCAL_STATIC_LIBRARIES := \     libbz# Also need the JNI headers.LOCAL_C_INCLUDES += \    $(JNI_H_INCLUDE) external/bzip2# No special compiler flags.LOCAL_CFLAGS +=include $(BUILD_SHARED_LIBRARY)