Analysis of Dex splitting (MultiDex) technology in Android

Analysis of Dex splitting (MultiDex) technology in Android
I. Preface

The subcontracting Technology in Android is no longer a new technology, and there is a lot of analysis on the Internet, but they all give theoretical knowledge and Principle Analysis, and there is no detailed case description, so here we will explain in detail the analysis of the dex splitting technology in Android. Before explaining this, let's take a look at why this technology has emerged? Why does google provide such a technology.

 

Ii. Background

When developing applications, as the business scale develops to a certain extent, the new functions and new class libraries are constantly added, and the Code expands rapidly, the size of the corresponding apk package also increases sharply, then one day, you will unfortunately encounter this error:
The generated apk cannot be installed on android 2.3 or earlier machines. The following message is displayed:INSTALL_FAILED_DEXOPT
If the number of methods is too large and an error occurs during compilation, the following message is displayed:
Conversion to Dalvik format failed: Unable to execute dex: method ID not in [0, 0 xffff]: 65536

Installation failure (Android 2.3 INSTALL_FAILED_DEXOPT) is caused by the LinearAlloc limit of dexopt. Different Android versions have gone through the 4 M/5 M/8 M/16 M limit, currently, the mainstream 4.2.x systems can all reach 16 M, and the allocated space of LinearAllocHdr in Gingerbread or the following systems is only 5 M, which is 8 M higher than that in Gingerbread systems. Dalvik linearAlloc is a fixed-size buffer. During application installation, the system runs a program named dexopt to prepare the application for running on the current model. Dexopt uses LinearAlloc to store the method information of the application. The buffer for Android 2.2 and 2.3 is only 5 MB, and Android 4.x is increased to 8 MB or 16 MB. When the number of methods exceeds the buffer size, dexopt crashes.
The maximum number of methods is exceeded due to the format limit of the DEX file. The number of methods in a DEX file uses the native short type to index the methods in the file, that is, up to 65536 methods can be expressed in four bytes. This limit applies to the number of fields/classes. For DEX files, all the class files required by the project are merged and compressed to a DEX file, that is, the DEX package process of Android, the total number of methods that can be referenced by a single DEX file (self-developed code and the code of the referenced Android framework and Class Library) is limited to 65536.

We know the reason, but we can see that google provides a solution: Multidex technology to solve this problem. To be compatible with the old SDK version, but let's think about it, is this technology used by all projects? The answer is definitely not. Not all projects will encounter this kind of problem. This technology is used only when your project is large enough and the number of methods exceeds the limit. Now that we need to use it, we will introduce it here. What knowledge points should we prepare before introducing this article?

1. Learn how to use Ant script to compile an apk package

2. Understand the process and steps for compiling an apk package.

3. Understand the usage of dx and aapt commands in Android

4. Understand the dynamic loading mechanism in Android

I have mentioned these materials in my previous articles, and those I don't know can fight:

You can refer to this article for the entire process and steps of compiling an apk package with the Ant script and package the apk package:

Http://blog.csdn.net/jiangwei0910410003/article/details/50740026

If you do not know about the dynamic loading mechanism in Android, refer to this article:

Http://blog.csdn.net/jiangwei0910410003/article/details/48104581

Only by understanding these four knowledge points can we see the content described below. So I hope you can take a closer look at these two articles.

 

Iii. Technical Principles

Now that the preparation has been completed, we will introduce in detail the technical principles of package splitting, which is actually a solution provided by google:

Step 1: Use dx to unpack

There are two categories: one is automatic unpacking and the other is manual unpacking.

1. Automatic unpacking

Starting with google SDK 5.0 +, the dx command is supported:-- Multi-dexParameters for direct automatic Subcontracting

For more information about how to unpack specific commands, I will explain the case in detail.

2. Manually unpack

Manual package splitting is actually used to solve the problem that the SDK of earlier versions does not support the -- muyun-dex parameter. Therefore, we have to find a solution. We know that, one step in android is to use the dx command to convert the class file to dex. Here we can do this. Before the dx command is executed, first, all the class files compiled by javac are classified, and then dx is used for specific classification to convert to the dex file. In this way, multiple dex files are generated. At this time, we may need to write an additional script for class classification. The specific number of dex types is determined by ourselves.

The above two methods of unpacking are described. In fact, these two methods have their own advantages and disadvantages. Of course, we advocate the first method, because the SDK is already 5.0 +, this method is also advocated by google and is also convenient.

There is a common concern about the two subcontracting technologies mentioned above.:

In Android, multiple dex entries are generated after subcontracting, but the system will first find classes by default. dex files are loaded and run automatically, so there is a problem here. We need to put some important initialization classes into classes. dex. Otherwise, an error or crash occurs.

Here we can see the differences between the two subcontracting technologies:

If the first subcontracting technology is used, we can use the -- main-dex-list parameter to specify which class files belong to the primary dex, that is, classes. in dex, the remaining dx will automatically classify the data from dex, such as classes2.dex... classes3.dex... classes1.dex is not found here. Pay attention to this. At the same time, subdex does not support class Classification and relies entirely on dx automatic classification. This is why it is called automatic package splitting.

However, if we use the second type of package splitting technology, it will be very random, because the classification is our own operation, so we can divide it as much as we want, in addition, the number of dex files and the class to which dex is supported can be achieved. The flexibility is good, but there is a bad way in this way that you need to write your own scripts for classification. At this time, you should be very careful, because it may miss a class that is not classified into a specific dex, an error is reported during running. This class cannot be found.

 

Step 2: Dex Loading

In the first step, we have explained how to split the class into multiple dex. Then the problem arises. As we mentioned above, when running in Android, only the classes will be loaded by default. dex is also called the primary dex file. how can other dex files be loaded to the system after the split? At this time, google provides a solution to use DexClassLoader for dynamic loading. I don't want to explain too much about dynamic loading of dex, because many of my previous articles have introduced it, specific can refer to this article: http://blog.csdn.net/jiangwei0910410003/article/details/48104581

So we only need to get all dex except classes. dex. We can load each dex to the system one by one. There are two problems to solve:

1. How can I obtain all subdex statements?

Here we can do this. In the attachContext method of the Application (because this method was the earliest), we can get the apk path of the program, and then decompress the apk file using ZipFile, you can obtain all the subdex files. For more information about the operations, see the case.

2. We use DexClassLoader to load subdex and then let the system know?

Here we use the reflection mechanism to merge the DexList variable values in the PathClassLoader and DexClassLoader of the system. This technology is also provided by google. The specific technology is also explained in detail in the following cases.

 

Iv. Case Analysis

Here we will introduce the two steps for unpacking: splitting dex and loading dex. Here we will use specific cases to practice it, this is also a lot of information about dex splitting technology on the Internet, but there is no particular case. Everyone knows the theoretical knowledge, but there is no case to explain it, I don't know what problems will be encountered in the actual process, and the detailed operation steps, so here we must use a case to describe in detail.

Our case uses the ant script for compiling. In fact, google recommends gradle for compiling, because it has already been integrated into AndroidStudio. To tell the truth, I am not familiar with gradle syntax, so I didn't use gradle to explain it. Of course, the gradle syntax and ant syntax are quite helpful. If you want to learn gradle, you can use gradle for one operation. Here I won't introduce the ant script syntax too much, instead, we will introduce the detailed command usage. In fact, all the compilation scripts theoretically have added some functions to the optimization of compilation commands. Well, let's not talk about it. Let's look at the case:

The main-dex-rule.txt here is the list of class files included in the main dex we need later, my. keystore is the signature apk File

Next, let's take a look at the compilation script build. xml.

I will not explain it all here again. This script is an article I wrote earlier:

Http://blog.csdn.net/jiangwei0910410003/article/details/50740026

You must first read this article ~~

First, let's take a look at how to use the dx command to automatically split dex:

 

 
  
   Generate multi-dex...
  
  
 

The parameters here are simple:

 

Parameter description:
-- Multi-dex: multi-dex package Switch
-- Main-dex-list = : The parameter is a file of the class list. Classes in this file will be packaged in the first dex.
-- Minimal-main-dex: only the classes specified in the -- main-dex-list file are packaged in the first dex, and the rest are in the second dex file.

Because the last two parameters are optional parameters, in theory, you only need to add the "-- multi-dex" parameter to dx to generate classes. dex, classes2.dex, classes3.dex, and so on.

Here we specify -- main-dex-list merge to merge the specified classkey into classes.dex. see the content of main-dex-rule.txt:

Here, we will merge the MyApplication class, the entry Activity of the program, and the three classes that need to load dex into the main dex, because these three are the earliest initialization time, it is also an important class for loading the subdex class, so it must be placed in the main dex. Otherwise, the program cannot run. Of course, you should also pay attention to the situation of internal classes and add them.

Okay, after running build, we found a problem, that is, we only saw one dex, that is, classes. dex. Why?

Looking At dx parameters, main-dex-list and minimal-main-dex will only affect the files contained in the primary dex, and will not affect the generation of files from dex, so it should be caused by other reasons.
If no data is found, analyzing the source code is a perfect solution to the problem. So I decompiled dx. jar and found the following key lines of code through analysis:

Obviously, when dx is used for multi-dex packaging, the maximum number of methods in each dex is 65536 by default. Check the number of dex methods currently used. The total number of methods is only 51392 (the number of methods is not exceeded, mainly because the number of LinearAlloc Methods exceeds the standard), but not 65536, so there is only one dex during packaging.
Continue to analyze the code and find the following key code:

This indicates that dx has a hidden parameter -- set-max-idx-number. This parameter can be used to set the maximum number of methods for each dex during dx packaging, however, this parameter is not listed in dx Usage !).

In the ant script, we set this parameter so that the maximum number of methods for each dex is set to 2000. Then, compile:

Okay, now there are two dex S. Here I can find that there is no classes1.dex, And the subscript starts from 2. This should be noted.

Here we can use IDA to view the dex file:

Classes. dex:

Classes2.dex:

It seems that the operation was successful, so let's continue to look at it. Now we have two dex files. How can we compile them into an apk file?

We know that we can use the apkbuilder command to package the compiled resource. arsc, asset, and dex files into apk files.

But here comes a problem:

Only one dex file can be included in the parameters supported by the apkbuilder command. To put it bluntly, only one dex file can be entered.

What should I do if it hurts? At that time, the verification results were urgently needed. It was a little trick to get it done first. We directly imported subdex using WinRar:

Then, you can sign the apk separately and run it. The mood is good, but it cannot be automated. It's too dark, so we have to find a way to think of the ZipFile class. We can decompress the apk, add the subdex, and then package the functions into an executable jar, then in build. run in xml. This is also a method, but it still feels bad and tedious. After thinking about it, I finally found a good method, which is also a very important knowledge point, that is, using the aapt command for operations. We may know that the aapt command is used to compile resource files, in fact, he also has an important purpose to edit the apk file:

We can see that we can delete a file in the apk or add a file. Okay, the method is finally found. Let's perform the following operations directly. First, we can run the following command to see the effect:

Now, let's take a look at the apk file:

Nima found that it was not correct. There is a pitfall here, that is, we didn't see classes2.dex, but we saw a directory structure. Oh, think about it. When the aapt command is added or deleted, the file path must be clear. It cannot be an absolute path, but a relative path. This is relative to the apk and directory. Now we want to put classes2.dex under the root directory of the apk, then it should be classes2.dex. We can put dex and apk under a directory for operations:

At this time, let's take a look at the apk file:

Well, it succeeded, so here we need to note the subdex path.

Next we will continue to look at the content of the build. xml script to see how to package it into an apk after the package is successfully split:

 

  
  
   
   
    
     
    
   
  
  
   
   
   
     
   
     
      
      
     
      
$ {Dexfile} is not handle
       
      
      
     
      
$ {Dexfile} is handle
     
     
      
    
    
    
  
  
   
    
     
      
     
    
   

The general process here is as follows:

 

1. traverse all dex files in the bin directory

2. Copy the dex file to the root directory of the project first, because our script file is under the root directory, and it is also under the root directory when running the script, therefore, you need to copy the dex file here. Otherwise, the above problem will occur. You cannot use an absolute path, but add the dex file to the relative path.

3. Because apk contains the classes. dex file by default, You need to filter the classes. dex file and add the aapt command to other dex files.
 

However, we also encountered a problem: how to use loops, regular expressions, and condition judgment tags in the ant script:

The specific labels are not mentioned. It mainly describes how to use these labels? By default, these labels are not available, so you need to import the jar package?

 

The following error is reported when the definition is imported:

 

At this time, we also need:Ant-contrib-1.0b3.jarPut it under the lib directory of ant. This jar is not available by default. There are many jar files on the Internet. You can download them by yourself.

Now we can run the build. xml script.

 

The preceding section describes how to package multiple dex files into an apk file.

 

The following describes how to implement the Code:

The Code defines four activities:

MainActivity is our entry Activity, and SecondaryDexEx is the core class for dynamic loading.

 

First, how to obtain the dex file from the apk:

 

ZipFile apkFile = null;try {apkFile = new ZipFile(appContext.getApplicationInfo().sourceDir);} catch (Exception e) {Log.i("multidex", "create zipfile error:"+Log.getStackTraceString(e));return;}Log.i("multidex", "zipfile:"+apkFile.getName());File filesDir = appContext.getDir("odex", Context.MODE_PRIVATE);Log.i("multidex", "filedir:"+filesDir.getAbsolutePath());for(int i = 0 ; i < SUB_DEX_NUM; i ++){String possibleDexName = buildDexFullName(i);Log.i("multidex", "proname:"+possibleDexName);ZipEntry zipEntry = apkFile.getEntry(possibleDexName);Log.i("multidex", "entry:"+zipEntry);if(zipEntry == null) {break;}msLoadedDexList.add(new LoadedDex(filesDir,possibleDexName,zipEntry));}

First obtain the path of the application apk, use ZipFile to decompress the apk, and obtain all dex files. Here we do not need to process classes. the dex file is loaded by default, so filter it here. In addition, the subdex subscript starts from 2 and does not have classes1.dex. Note. At the same time, obtaining the apk file of an application does not require any permission restrictions. It is similar to sharing a local app to a friend in QQ.

 

Next we get all the subdex, and then we can load it dynamically. We need to load our dex into the system. Here we mainly use the reflection technology to merge dexList.

 

/*** DexClassLoader needs to be injected here * Because GDT clicks an advertisement internally to download the App, and starts a DownloadService to download the App *, this is required here, otherwise, an exception * @ param loader */private static void inject (DexClassLoader loader, Context ctx) {PathClassLoader pathLoader = (PathClassLoader) ctx will be reported. getClassLoader (); try {Object dexElements = combineArray (getDexElements (getPathList (pathLoader), getDexElements (getPathList); Object pathList = getPathList (pathLoader); setField (pathList, ist, pathList. getClass (), "dexElements", dexElements);} catch (Exception e) {Log. I ("multidex", "inject dexclassloader error:" + Log. getStackTraceString (e);} private static Object getPathList (Object baseDexClassLoader) throws handle, NoSuchFieldException, IllegalAccessException, ClassNotFoundException {return getField (baseDexClassLoader, Class. forName ("dalvik. system. baseDexClassLoader ")," pathList ");} private static Object getField (Object obj, Class cl, String field) throws NoSuchFieldException, IllegalArgumentException, IllegalAccessException {Field localField = cl. getDeclaredField (field); localField. setAccessible (true); return localField. get (obj);} private static Object getDexElements (Object paramObject) throws IllegalArgumentException, NoSuchFieldException, IllegalAccessException {return getField (paramObject, paramObject. getClass (), "dexElements");} private static void setField (Object obj, Class cl, String field, Object value) throws NoSuchFieldException, IllegalArgumentException, IllegalAccessException {Field localField = cl. getDeclaredField (field); localField. setAccessible (true); localField. set (obj, value);} private static Object combineArray (Object arrayLhs, Object arrayRhs) {Class localClass = arrayLhs. getClass (). getComponentType (); int I = Array. getLength (arrayLhs); int j = I + Array. getLength (arrayRhs); Object result = Array. newInstance (localClass, j); for (int k = 0; k <j; ++ k) {if (k <I) {Array. set (result, k, Array. get (arrayLhs, k);} else {Array. set (result, k, Array. get (arrayRhs, k-I) ;}} return result ;}

 

Call the loadSecondaryDex method in the attachBaseContext method of MyApplication:

 

Here we have introduced the script and code. Let's start one script: Ant release

After running, we can see that a successful apk file is added to the bin directory:

The installation and running results are as follows:

As you can see, the MainActivity portal is started first, and then we click btn to enter different activities. The results run normally. You can also view the logs to obtain and load dex.

 

We were so excited that we finally had to solve the unpacking and encountered some problems, but we all solved them. Haha ~~

 

Project download: http://download.csdn.net/detail/jiangwei0910410003/9452599

 

5. Sort out the process

Next we will summarize what we have done:

1. First, we use the script to automatically split the package and get two dex files.

2. Because the apkbuilder command can only contain one dex file when creating an apk package, we need to use the aapt command to add other subdex files.

3. Obtain the apk file of the application and obtain all dex files. Use DexClassLoader to load subdex.

But for the second technique of package splitting we mentioned earlier, manual package splitting is not introduced here. In fact, it is very easy to copy files by category in the ant script, after obtaining multiple dex files, you can use the aapt command to add them to the apk file.

 

6. Knowledge Point Overview

What knowledge points have you learned:

1. More in-depth understanding of ant script syntax

2. I learned how to use the dx command for automatic unpacking.

3. reviewed DexClassLoader usage

 

7. Errors

We may encounter some errors in this process. The most common errors are: ClassNotFound and NotDefClass. This process is very simple. Let's check which class is used, then use IDA to check whether each dex file contains this class. If not, it indicates that this class is not classified.

 

VIII. Performance Analysis

During Cold Start, multiple DEX files need to be loaded. If the DEX file is too large and the processing time is too long, the ANR (Application Not Responding) is easily triggered );
Applications using the MultiDex scheme may not be started on machines lower than Android 4.0 (API level 14). This is mainly because of a Dalvik linearAlloc bug (Issue 22586 ); applications using the MultiDex scheme may crash the program because they need to apply for a large memory. This is mainly because of a limitation of Dalvik linearAlloc (Issue 78035 ). this restriction has been added to Android 4.0 (API level 14) and may be triggered on machines lower than Android 5.0 (API level 21.

After Dex subcontracting, synchronous loading during startup may affect the startup speed of the application, but the main impact is the first startup after installation. This is because the Android system performs Dexopt on the loaded dex and generates ODEX when it is started for the first time after installation. Dexopt is a time-consuming operation, therefore, the Initial Startup speed after installation is greatly affected. When the application is started for the first time after installation, it only needs to load ODEX. This process is fast and has little impact on the startup speed. In addition, the size of the slave dex directly affects the startup speed. That is, the smaller the slave dex, the faster the startup speed.

 

IX. Summary

There have been many cases and explanations on the Android package splitting technology online, but most of them are theoretical knowledge, not detailed, and there is no demo. Therefore, this article mainly introduces the technical points of subcontracting and uses a demo for case analysis, so as to gain a deeper understanding of the unpacking technology, although ant scripts are used for compilation in this article, gradle can also be used for compilation. As long as you understand the principles, commands are actually operating and have nothing to do with the script. From then on, we will not feel that the package splitting technology is far from clear. This is actually the case.