Introduction to automatic unpacking and dynamic loading of Apsara stack Android DEX

Introduction to automatic unpacking and dynamic loading of Apsara stack Android DEX
Overview

As an android developer, as the business scale develops to a certain extent, he constantly adds new features and adds new class libraries, and the code is rapidly expanding, the size of the corresponding apk package also increases sharply, so one day you will unfortunately encounter this error:

1. The generated apk cannot be installed on android 2.3 or earlier machines. The error "INSTALL_FAILED_DEXOPT" is displayed.
2. If the number of methods is too large and an error occurs during compilation, the following message is displayed:

    Conversion to Dalvik format failed:Unable to execute dex: method ID not in [0, 0xffff]: 65536

   The specific causes of the problem are as follows:

   1. Installation failure (Android 2.3 INSTALL_FAILED_DEXOPT) is caused by the LinearAlloc limit of dexopt. Different Android versions have gone through the 4 M/5 M/8 M/16 M limit, currently, the mainstream 4.2.x systems can all reach 16 M, and the allocated space of LinearAllocHdr in Gingerbread or the following systems is only 5 M, which is 8 M higher than that in Gingerbread systems. Dalvik linearAlloc is a fixed-size buffer. During application installation, the system runs a program named dexopt to prepare the application for running on the current model. Dexopt uses LinearAlloc to store the method information of the application. The buffer for Android 2.2 and 2.3 is only 5 MB, and Android 4.x is increased to 8 MB or 16 MB. When the number of methods exceeds the buffer size, dexopt crashes.

   2. The maximum number of methods is exceeded due to the format limit of the DEX file. The number of methods in a DEX file uses the native short type to index the methods in the file, that is, up to 65536 methods can be expressed in four bytes. This limit applies to the number of fields/classes. For DEX files, all the class files required by the project are merged and compressed to a DEX file, that is, the DEX package process of Android, the total number of methods that can be referenced by a single DEX file (self-developed code and the code of the referenced Android framework and Class Library) is limited to 65536;

      Plug-ins? MultiDex?

      To solve this problem, there are generally the following solutions. One solution is to increase Proguard to reduce the size and number of DEX methods. However, this solution is a temporary solution, with the addition of Business Code, the number of methods will eventually reach this limit. One popular solution is the plug-in solution, and the other is the MultiDex solution provided by google, before google launched MultiDex, the Android Developers blog introduced the user-defined class loading process, and the Dalvik patch developed for Android apps by Facebook. However, what was written in facebook's blog was not very detailed; we have also made explorations and attempts to deploy the plug-in solution. to deploy the plug-in solution, we need to first sort out and modify the code of each business line to decouple the code. The changes are relatively large in size, through some discussion and analysis, we think it is more reliable for us to adopt the MultiDex solution, so that we can quickly and concisely split the code, at the same time, code changes are acceptable. In this way, we use the Multi Dex is developed.

      The plug-in solution has different implementation principles in the industry. Here we will not list them one by one. Here we will only list the solution that Google provides official support for building an application that exceeds the limit of 65 K: MultiDex.

      First, upgrade Android SDK Manager to the latest Android SDK Build Tools and Android Support Library. Perform the following two steps:

      1. Modify the Gradle configuration file, enable MultiDex, and include MultiDex support:

        android {    compileSdkVersion 21 buildToolsVersion 21.1.0    defaultConfig {        ...        minSdkVersion 14        targetSdkVersion 21        ...        // Enabling MultiDex support.        MultiDexEnabled true        }        ...    }    dependencies { compile 'com.android.support:MultiDex:1.0.0'}

      2. Allow the application to support multiple DEX files. Three optional methods are described in the official document:

      Declare android. support. MultiDex. MultiDexApplication in the application of AndroidManifest. xml;
      If you already have your own Application class, let it inherit the MultiDexApplication;
      If your Application class has inherited from other classes and you do not want to modify it, You can override the attachBaseContext () method:

      @Override protected void attachBaseContext(Context base) {    super.attachBaseContext(base);    MultiDex.install(this);}

      Add the following statement to Manifest:

           
            
                             ...            
            

      If you already have your own Application, let it inherit the MultiDexApplication.

      Dex automatic package splitting and dynamic loading of MultiDex

      After the MultiDex solution was launched in the first version, the MultiDex solution in Dalvik brought about the following problems:

      1. When the DEX file needs to be installed during cold start, if the DEX file is too large, the processing time is too long, it is easy to cause ANR (Application Not Responding );
      2. Applications using the MultiDex scheme may not be started on machines lower than Android 4.0 (API level 14). This is mainly because of a Dalvik linearAlloc bug (Issue 22586 );
      3. Applications using the MultiDex scheme may crash the program because they need to apply for a large memory. This is mainly because of a limitation of Dalvik linearAlloc (Issue 78035 ). this restriction has been added to Android 4.0 (API level 14) and may be triggered on machines lower than Android 5.0 (API level 21;

         This problem does not exist in MultiDex in ART because Ahead-of-time (AOT) compilation technology is used in ART, during APK installation, the system uses the built-in dex2oat tool to compile the available DEX files in the APK and generate a file that can be run on the local machine, this can improve the application startup speed, because the installation process will affect the application installation speed. If you are interested in ART, refer to the differences between ART and Dalvik.

         The basic principle of MultiDex is to load Secondary DEX through DexFile and store it in the DexPathList of BaseDexClassLoader.

         The following code snippet is the process of BaseDexClassLoader findClass:

         protected Class
                  findClass(String name) throws ClassNotFoundException {    List
                 
                   suppressedExceptions = new ArrayList
                  
                   ();    Class c = pathList.findClass(name, suppressedExceptions);    if (c == null) {        ClassNotFoundException cnfe = new ClassNotFoundException(Didn't find class  + name +  on path:  + pathList);         for (Throwable t : suppressedExceptions) {            cnfe.addSuppressed(t);        }        throw cnfe;    }    return c;}
                  
                 

         The following code snippets show how to use DexFile to load Secondary DEX and put it in the DexPathList of BaseDexClassLoader:

         private static void install(ClassLoader loader, List
                 
                   additionalClassPathEntries,                            File optimizedDirectory)        throws IllegalArgumentException, IllegalAccessException,        NoSuchFieldException, InvocationTargetException, NoSuchMethodException {    /* The patched class loader is expected to be a descendant of     * dalvik.system.BaseDexClassLoader. We modify its     * dalvik.system.DexPathList pathList field to append additional DEX     * file entries.     */    Field pathListField = findField(loader, pathList);    Object dexPathList = pathListField.get(loader);    ArrayList
                  
                    suppressedExceptions = new ArrayList
                   
                    ();    expandFieldArray(dexPathList, dexElements, makeDexElements(dexPathList,            new ArrayList
                    
                     (additionalClassPathEntries), optimizedDirectory,            suppressedExceptions));    try {        if (suppressedExceptions.size() > 0) {            for (IOException e : suppressedExceptions) {                //Log.w(TAG, Exception in makeDexElement, e);            }            Field suppressedExceptionsField =                    findField(loader, dexElementsSuppressedExceptions);            IOException[] dexElementsSuppressedExceptions =                    (IOException[]) suppressedExceptionsField.get(loader);            if (dexElementsSuppressedExceptions == null) {                dexElementsSuppressedExceptions =                        suppressedExceptions.toArray(                                new IOException[suppressedExceptions.size()]);            } else {                IOException[] combined =                        new IOException[suppressedExceptions.size() +                                dexElementsSuppressedExceptions.length];                suppressedExceptions.toArray(combined);                System.arraycopy(dexElementsSuppressedExceptions, 0, combined,                        suppressedExceptions.size(), dexElementsSuppressedExceptions.length);                dexElementsSuppressedExceptions = combined;            }            suppressedExceptionsField.set(loader, dexElementsSuppressedExceptions);        }    } catch(Exception e) {    }}
                    
                   
                  
                 

         Dex automatic package splitting and Dynamic Loading

         By viewing the MultiDex source code, we found that MultiDex is prone to ANR bottlenecks during cold start. In the Dalvik VM version earlier than 2.1, the installation of MultiDex is divided into several steps, step 1 open the zip package of apk, and step 2 extract the dex of MultiDex (excluding Classes. dex, for example, classes2.dex, classes3.dex, etc.), because the android system only loads the first Classes when starting the app. dex. We need to manually install other DEX files. The third step is reflection-based. In fact, these three steps are time-consuming, to solve this problem, we should consider whether DEX can be loaded into an asynchronous thread, so that the cold start speed can be improved a lot and ANR in the cold start process can be reduced, for a defect (Issue 22586) and limitation (Issue 78035) of Dalvik linearAlloc, we consider whether manual intervention can be performed on DEX splitting, so that the size of each DEX is within a reasonable range, so as to reduce the defects and restrictions that trigger Dalvik linearAlloc. To achieve these purposes, we need to solve the following three problems:

         1. How to generate multiple DEX packages during the packaging process?
         2. If dynamic loading is performed, how does one determine which DEX is dynamic?
         3. If dynamic loading is performed in the work thread after startup, what should I do if the user needs to use the class in dynamic loading DEX for page operations without loading?

            First, we will analyze how to solve the first problem. When using the MultiDex solution, we know that BuildTool will automatically split the code into multiple DEX packages, in addition, the configuration file can be used to control which code is put into the first DEX package, which is the packaging process of Android:

            To generate multiple DEX packages, We can customize a Task in Ant or gradle to intervene in the DEX generation process in this step of generating the DEX file, thus producing multiple DEX files, it is used to intervene in the Custom tasks that generate DEX in ant and gradle:

            tasks.whenTaskAdded { task ->    if (task.name.startsWith('proguard') && (task.name.endsWith('Debug') || task.name.endsWith('Release'))) {        task.doLast {            makeDexFileAfterProguardJar();        }        task.doFirst {            delete ${project.buildDir}/intermediates/classes-proguard;            String flavor = task.name.substring('proguard'.length(), task.name.lastIndexOf(task.name.endsWith('Debug') ? Debug : Release));            generateMainIndexKeepList(flavor.toLowerCase());        }    } else if (task.name.startsWith('zipalign') && (task.name.endsWith('Debug') || task.name.endsWith('Release'))) {        task.doFirst {            ensureMultiDexInApk();        }    }}

            The previous step solved the problem of how to package multiple DEX files, So how should we decide which classes should be placed in Main DEX based on what, in Secondary DEX (the Main DEX here refers to the Classes that are automatically loaded when the android system starts the apk before the Dalvik VM of Version 2.1. dex, while Secondary DEX refers to the DEX that needs to be installed by ourselves, for example, Classes2.dex, Classes3.dex, etc.). This needs to analyze the class dependency in the Main DEX, make sure to include all the class dependencies in Main DEX. Otherwise, ClassNotFoundException will occur at startup, here, our solution is to put all the Code involved in the Service, javaser, and Provider into Main DEX, and split the Code involved in the Activity to some extent, and split the homepage Activity, Laucher Activity, and Huan The class on which the webpage Activity and city list Activity are dependent is put into Main DEX, and the Activity of level 2 and level 3 pages and the Code of Business channels are put into Secondary DEX, to reduce the maintainability and high risk caused by manual analysis of class dependencies, we have compiled a script that can automatically analyze Class dependencies, this ensures that the Main DEX contains the class and all the classes they depend on are in it, so that the script will automatically analyze all the Code involved in starting the Main DEX before packaging, ensure that Main DEX runs normally.

            With the solution to the second problem, we came to the tricky third problem. If we load Secondary DEX in the background, the user clicks the interface to jump to the class interface in Secondary DEX, so ClassNotFoundException will inevitably occur. How can this problem be solved, add in all Activity jump code to determine if Secondary DEX is loaded successfully? This method is feasible, but the workload is very large. Is there a better solution? By analyzing the Activity startup process, we found that the Activity was started by ActivityThread through Instrumentation. Can we do some work in Instrumentation? By analyzing the codes ActivityThread and Instrumentation, it is found that the Instrumentation methods related to Activity initiation include execStartActivity and newActivity, in this way, we can add code logic in these methods to determine whether the Class is loaded. If it is loaded, the Activity will be started directly, if the loading is not complete, start a waiting Activity and display it to the user. Then, wait for the background Secondary DEX loading to complete in this Activity. After the loading is complete, the Activity automatically jumps to the Activity that the user actually wants to jump; in this way, on the premise that the Code is fully decoupled and every business code can be granular, we can load Secondary DEX on demand. Below are some key codes added by Instrumentation:

                public ActivityResult execStartActivity(Context who, IBinder contextThread, IBinder token, Activity target,                                            Intent intent, int requestCode) {        ActivityResult activityResult = null;        String className;        if (intent.getComponent() != null) {            className = intent.getComponent().getClassName();        } else {            ResolveInfo resolveActivity = who.getPackageManager().resolveActivity(intent, 0);            if (resolveActivity != null && resolveActivity.activityInfo != null) {                className = resolveActivity.activityInfo.name;            } else {                className = null;            }        }        if (!TextUtils.isEmpty(className)) {            boolean shouldInterrupted = !MeituanApplication.isDexAvailable();            if (MeituanApplication.sIsDexAvailable.get() || mByPassActivityClassNameList.contains(className)) {                shouldInterrupted = false;            }            if (shouldInterrupted) {                Intent interruptedIntent = new Intent(mContext, WaitingActivity.class);                activityResult = execStartActivity(who, contextThread, token, target, interruptedIntent, requestCode);            } else {                activityResult = execStartActivity(who, contextThread, token, target, intent, requestCode);            }        } else {            activityResult = execStartActivity(who, contextThread, token, target, intent, requestCode);        }        return activityResult;    }    public Activity newActivity(Class
                       clazz, Context context, IBinder token,                                Application application, Intent intent, ActivityInfo info,                                CharSequence title, Activity parent, String id, Object lastNonConfigurationInstance)            throws InstantiationException, IllegalAccessException {        String className = ;        Activity newActivity = null;        if (intent.getComponent() != null) {            className = intent.getComponent().getClassName();        }        boolean shouldInterrupted = !MeituanApplication.isDexAvailable();        if (MeituanApplication.sIsDexAvailable.get() || mByPassActivityClassNameList.contains(className)) {            shouldInterrupted = false;        }        if (shouldInterrupted) {            intent = new Intent(mContext, WaitingActivity.class);            newActivity = mBase.newActivity(clazz, context, token,                    application, intent, info, title, parent, id,                    lastNonConfigurationInstance);        } else {            newActivity = mBase.newActivity(clazz, context, token,                    application, intent, info, title, parent, id,                    lastNonConfigurationInstance);        }        return newActivity;    }

            In practical application, we also encountered another tricky problem, that is, too many fields, which were introduced by the Code organization structure we currently use, to facilitate the development of multi-business lines and multi-team concurrency and collaboration, we adopt the aar Method for development and introduce a general business aar at the bottom of the aar dependency chain, the General Service aar contains a lot of resources. When the Library resources are processed in ADT14 and later versions, the R resources of the Library are no longer static final, for details, please refer to the official instructions of google. In this way, the R in the Library cannot be inline during the final packaging, which leads to the excessive number of R fields, resulting in the need to split multiple Secondary DEX, to solve this problem, we use scripts to replace the references of R field (such as ID, Layout, Drawable) in Libray with constants during the packaging process, and then delete R. the corresponding Field in the class.

            Summary

            The above is the DEX automatic unpacking solution that we evolved in the MultiDex process. In this way, we can use script control to automate the splitting of DEX, then, the Secondary DEX can be freely loaded during the runtime, which can ensure the cold start speed and reduce the memory usage during the runtime.