How to pack your IOS project more than 10 times times faster

How to pack your IOS project more than 10 times times faster

Too slow compilation speed has very obvious side effects. On the one hand, programmers may be distracted by the process of waiting for a package, such as brushing a friend's circle, watching a news story, and so on. This cognitive context switching can lead to a lot of invisible time wasting. On the other hand, most apps have their own continuous integration tools, and if they're packaged too slowly, they can affect the overall team's development progress.

Therefore, this article will discuss the daily development and continuous integration of the two scenarios, analysis of the slow packaging bottlenecks, and corresponding solutions. Using these schemes, the author successfully reduced the continuous integration time of the company's app from 9 min to 80%, and increased the efficiency by up to 10 times times. If you summarize in one sentence is:

In front of absolute strength (hardware), all skills (software) are clouds

Daily Development

In fact, the optimization of daily development space is not big, because by default, Xcode will use the last compile-time left the cache, that is, the so-called incremental compilation. Therefore, the main time of daily development is composed of three parts:

Total time Consuming = Incremental compilation + link + Generate Debug information (DSYM)

Incremental compilation takes a short time, even in my 14-year-old MacBook Pro (4 cores, 8 threads, 2.5GHz i7 4870HQ, hereinafter referred to as MBP), and it takes just 10 seconds to move up and down. Our application code volume of about more than 1 million lines, the industry more than this level of application should not be many. Linking and generating debug information is less than 20s each, so the time spent on one incremental compilation is about half a minute to about a minute, and we analyze it individually:

1. Incremental compilation: Because it takes a short time (about more than 10 seconds or less), there is little room for optimization, but it is very easy to get worse. Because only the header files can be cached, if a file is referenced by N files, and the file's header file has changed, then the n files will be recompiled. The layered architecture of the APP is generally done, but a typical misconception is that you use macro definitions in the header files of the base library, such as defining constants that can be read globally, such as whether to turn on debugging, the server's address, and so on. Once these constants change (for example, to debug or switch to certain branches), the application will be recompiled.
2. Links: Links are not cached and can only be done with a single core, so its time-consuming depends primarily on the performance of the single core and the speed of disk read and write. Given that our target files are generally small, the performance of 4K random reads and writes should be more important.
3. Debugging information: The daily development, do not need to generate DSYM files, this file is mainly used to find the call stack when the crash, easy to debug online applications, and the development process of the crash can be directly seen in Xcode, the shutdown of this feature will not have any negative impact on development.

The daily development of the optimization space is small, even if the large project, backward machine performance, close DSYM will take time to take about 30s. In contrast, the packaging speed can be optimized and discussed more places.

Continuous Integration

Caching is not recommended when using tools such as Jenkins for continuous integration. This is because Apple's cache is not stable enough and in some cases there are bugs. For example, obviously the local has fixed the bug, can be compiled through, but the last compilation cache was not properly cleaned, resulting in the packaging machine still cannot compile through. Or the local clearly wrote the bug, but also because of the cache problem, the packaging machine can still be compiled through.

Therefore, either manually deleting Derived Data the folder or invoking the xcodebuild clean command will erase the cache. or directly xcodebuild archive , the cache is ignored automatically. Recompiling all the time is the root cause of slow packaging. Take our project as an example, in a total of 45min packaging time, 40min is executing xcodebuild this line of command.

using the CCache cache

The most natural idea is to use the cache, since Apple's cache is not reliable, then find a reliable cache, such as CCache. It is based on the compiler level of the cache, according to the current feedback situation, there is no problem of cache inconsistency. According to the author's experiment, the use of CCache really can increase the packaging speed significantly, delete the cache and use CCache recompile, it takes only more than 10 minutes.

However, the most deadly problem with CCache is that PCH files and Clang modules are not supported. PCH is intended to optimize compilation time, we assume that a header file a depends on the M head file, where each dependent header file depends on the N head file, as shown in:

Because #import the essence is to copy the content of the dependent header file into its own header file, so the header file A actually contains the contents of M * N head file, it will require M * N file IO and related processing. The process of M * N complexity is repeated once for each additional file in the project that relies on the header file a.

The advantage of PCH files is that the header files in this file will only be compiled once and cached, and then added to all header files in the project. The problem is solved, but it is very retarded that all files implicitly depend on all the files in the PCH, and the files that really need to be globally dependent are actually very few. As a result, more people will take PCH as a fast import tool rather than as a compilation performance optimization. As explained earlier, once a PCH file has been modified, it can lead to thorough completion of a full project recompilation, thus reducing compilation speed. It is because of the side effects of PCH that it even offsets the optimizations it brings, and Apple has not used the PCH file by default.

To replace the PCH is the Clang modules technology, for projects that open this option, we can @import replace the past #import , such as:

@import UIKit;

Equivalent to

#import <UIKit/UIKit.h>

Aside from these small features of the auto-link framework, Clang modules can be understood as a modular PCH, which has the advantage of having the PCH cache header files while providing finer-grained references.

Say back to CCache, because it does not support PCH and Clang modules, which makes it impossible to apply in our project. Even if it can be used, it will slow down the technical upgrade of the project, in exchange for the cache at this price, I am afraid is not worth the candle.

DISTCC

DISTCC is a distributed compilation tool that can send files that need to be compiled to other machines to compile, and then receive the compilation artifacts. However, through the application of bar paste, bass chat, hand Q and other experiments, found that it is not suitable for iOS applications. Its principle is that multiple clients compile together, but the vast majority of files actually compile time is very short, it is not worth to send back and forth through the network, this scheme should only be suitable for a single file volume of very large projects. In our project, the use of distcc a large increase in packaging time, about 1 hours or so.

Locating bottlenecks

After looking for an external tool without a result, I began to try to optimize the compilation time directly. In order to figure out how this 40min is going to cost, I'll start with xcodebuild a detailed analysis of the output results.

Anyone who has used a xcodebuild command will know that its output is not friendly to the developer, it is almost unreadable, and xcpretty that the tool can format it:

gem install xcpretty

gemafter installation, simply pass xcodebuild the output to the pipeline xcpretty :

xcodebuild -scheme Release ... | xcpretty

Here is the Demo in the official documentation:

I am only interested in the compilation part, so we can get a highly uniform output of the format by simply doing the filtering:

Compiling A.mCompiling B.mCompiling ...Compiling N.m

At this point, we can finally do the most critical calculation, we can set the timer to calculate the interval between the two adjacent lines of output, this interval is the compilation time of the file. Of course, there are similar assistive tools to do this logic:

By simply sorting, you can see the most time-consuming top 200 files, as well as differentiate between file suffixes, calculate the total time spent, and so on. After troubleshooting, we found that half of the compilation time was spent on compiling the Protobuf file.

Project Setup

In addition to case-by-case analysis for ultra-long time-consuming files, another option is to adjust the engineering settings. In general, our continuous integration tools are primarily intended for use by product managers or testers, to experience functionality or to verify bugs, and do not need to be concerned about runtime performance unless the APP Store is on the shelves. However, the use of the Release mode on the phone, the default will turn on a variety of optimizations, these optimizations are sacrificing the performance of the compilation, in exchange for running speed, for the shelves of the package is understandable, but for those Daily Build package, it seems to outweigh the cost.

Therefore, the idea of accelerating packaging and the idea of optimization are completely reciprocal, and all we have to do is turn off all possible optimizations. An article is recommended here: The summary of the research work on Xcode's performance optimization can be said to be quite comprehensive.

After searching the data for each parameter and trying to close it, the ascending speed is arranged in descending order, it is simple to organize several:

1. Only the ARMV7 instruction set is supported. The instruction set on the phone belongs to the ARM series, from the old to the new armv7, armv7s and arm64. The new instruction set can be compatible with the old model, but the old model cannot be compatible with the new instruction set. By default, our packages will have ARMV7 and arm64 two instruction sets, the former responsible for backstop, and for models that support the arm64 instruction set, using the latest instruction set can achieve better performance. Of course, the cost of generating two sets of instructions takes more time. So in the Haste pack mode, we only generate the oldest instruction set of ARMV7, sacrificing runtime performance for compile speed.
2. Turn off compilation optimizations. The rationale for optimization is to sacrifice compile-time performance and pursue runtime performance. Common optimizations are compile-time removal of useless code, retention of debugging information, function inline, and so on. So the secret to increasing the speed of packaging is to do the opposite, sacrificing runtime performance in exchange for compile-time performance. The two most important optimizations I have made are to Optimize level change to O0, which means not doing any optimization.
3. Use virtual disks. The compilation process requires a lot of disk IO, which mainly occurs in the Derived Data directory, so if enough memory, you can consider the 4G of memory, build a virtual disk, which will optimize the disk IO to memory IO, thereby increasing the speed. Since the packaging machine is recompiled every time, there is no need to worry about cache loss after restarting the machine.
4. The Dysm file is not generated and has already been described in the previous article.
5. For some other options, refer to the previous recommended article.

In the above several operations, the function of the reduced instruction set is the largest, approximately can reduce the compilation time from three Min to 30min, with the Shutdown compilation optimization, can further reduce the packaging time to 20min. Virtual disks can reduce compilation time by approximately two or three minutes, DSYM takes about 20 seconds, and other options are less optimized, about a few seconds, without accurate measurements.

Therefore, in general, as long as the simplified instruction set and turn off optimization, the conditional machine can use the virtual disk, do not recommend other modifications.

Binary

The binary is mainly refers to the profit static library instead of the source code, to avoid compiling. The previous article has described how time-consuming it is to analyze files, so the proceeds from binary are very easy to calculate. Due to the problem of division of labor, the author has no experience of binary system, in general this optimization is more suitable for the infrastructure group to implement.

Hardware Acceleration

The above is mainly through the modification of software to speed up packaging, since the company applied for 2013 Years Mac Pro (Xeon-e5 1630 6 core 12 threads, 16G memory, 256G SSD Standard, hereinafter referred to as MAC Pro), no need to modify any configuration, just simple migration baler , you can reduce the packaging time to a maximum of three, and the last one in the previous section of the optimization, the final packaging time of about 10min.

On my black Apple (i7 7820x 8 Core 16 threads, 16G memory, Samsung PM 961 512G SSD, hereinafter referred to as Black Apple), even if no optimizations are turned on, the compilation from scratch is only required for 5min. If the Protobuf file binary, and the optimization of some engineering settings, I do not dare to imagine how long it takes, it is expected to be around 4min, the speed increased by about 11 times times.

Compilation is a test of multi-core performance, on my black Apple, compile can see 8 CPU load has reached 100%, so within a certain range (such as 10 cores), the increase in CPU cores is far more than the increase in the frequency of the single core of the compilation speed of the impact. For some 20 cores or more, the performance of a CPU with low single-core performance is expected to give feedback to experienced readers.

Summary of optimization points

The following table summarizes the speed improvements brought about by the various optimization methods mentioned in the article, with reference to the original time of each (packing machine: 13 inch MacBook Pro):

Scenario Number	Optimization Scenarios	time elapsed after optimization (min)	Time Reduction Percentage
1	Infrequently modified file binaries	25	44.4%
2	Streamlined instruction Set	27	60g
3	Turn off compilation optimizations	38	15.6%
4	Using Mac Pro	15	66.7%
5	Virtual Disk	42	6.7%
6	Company current program (2+3+4+5)	9	90D
7	Black Apples	5	88.9%
8	The ultimate solution (1+2+3+5+7)	4 (Expected)	91.1% (Expected)

Strictly speaking, the article is a bit of a title to the party, because one sentence is:

You can use hardware to solve the problem, do not use software to solve.

How to pack your IOS project more than 10 times times faster