IOS Continuous Integration

IOS Continuous Integration Series-Introduction

IOS development slowly stabilized after these years of savage growth. Whether the development language is OBJECTIVE-C or Swift, the type of project is Hybird or native, the idea of development is OOP or functional, as the project has become more and more in the face of the same problem: testing, publishing and other repetitive work accounted for a large part of the time, the return cost is higher. Continuous integration is inevitably put on the agenda.

This article mainly describes the continuous integration under iOS, with the goal, content, process, tools to start, I hope we can describe an iOS continuous integration blueprint. This may not be an article that allows you to follow step by step, but hopefully you can build a concept in your mind to go in the right direction when you are actually doing it.

We will explain in more detail what is and how to do it in a few of the following sections.

Goal

For us, reducing duplication of effort and improving team efficiency. For the company, save money!

The narrow sense of continuous integration refers to early integration early detection early detect problems early repair, and supplemented by some automated means, the goal is to reduce the cost of repair. Through its purpose, we can see that in fact, reducing duplication of effort through automation and reducing costs through early detection is the core concept of continuous integration. So I put the automation Code Review here too.

Content

Ongoing integration under IOS includes automated Code reivew, automated unit testing, automated packaging, and automated distribution. The automation code Review ensures that the team adheres to code specifications and reduces bugs during the coding phase. In a company with a large turnover, the same set of code is used to ensure a smoother handover of the project. Automated unit tests detect bugs during the integration phase to reduce regression costs. Automated packaging and automated distribution are less repetitive work, after all, engineers ' time is money.

Automated Code Review

As the name implies, automated code Review uses automated means to Review the code of the team members to ensure code quality. From a practical point of view, the automated code Review is more static analysis of the codes, by scanning the code and contrasting the rules that are produced to produce the desired results. The desired result can be a quantitative quality report for the project as a whole, or a warning to be displayed in Xcode??。 This depends on what role the user is in.

In practice, there are generally two roles that will focus on the results-engineers and management. Engineers need to be aware of the code errors during the development process in order to correct them in a timely manner. Management needs to understand the overall code quality of the project in order to master the risks associated with the project. At the same time, it can also be one of the basis of engineer performance.

To complete this piece of content, we need these tools:

• Jenkins
• SonarQube
• Oclint OR Swiftlint

Through these three collaborations, we can implement the following processes:

工程师通过 `Git Commit` 提交代码 → Web Hook 触发 `Jenkins` 构建 → `OCLint` 扫描代码生成PMD格式报告 → `Sonar-runner` 读取报告并展现到 `SonarQube`。

As for Code Review, it is necessary to point out that automation tools have limitations. It is unable to analyze the more "intelligent" rules. For example, the following article

4.7 Method name with lowercase verb as the beginning

There is the following code, although there are many easy to cause the flash back of the BUG, but also can escape the analyzer's eyes

if result?.bindPhone == "true" {      let range = (result?.loginId?.startIndex.advancedBy(3))!...(result?.loginId?.endIndex.advancedBy(-5))!    let phoneNumber: String? = result?.loginId?.stringByReplacingCharactersInRange(range, withString: "****")    self.phoneNumLabel.text = phoneNumber!}

Therefore, in order to achieve the purpose of "Guaranteed code quality", it also requires the combination of manual Review and automated Review.

Automated Unit Testing

Automate unit testing, interrupt integration in the event of a test failure, and notify the relevant person that this piece of work is the content.

To do this, we need the following tools:

• Jenkins
• Xctool

We can implement the following processes:

`Git Merge`  → Web Hook 触发 `Jenkins` 构建 → xctool 执行单元测试 → 如果失败则发邮件给相关人员。

Of course, if your company's project is hosted on GitHub, the industry has two excellent Jenkins alternatives Travis-ci and Circle-ci. Their support for GitHub is perfect, and it's free for open source projects. Private projects require a paid trial.

In unit testing this piece, the highest cost is still the time cost of writing unit tests. If you can implement a tool that automatically generates test code for your business code, that's a big boost in productivity, says the brain hole.

Automated packaging and distribution

A complete package, you need to go through the configuration certificate, switch environment, adjust parameters (build version number, etc.), Archive, export. According to the complexity of the project, the above process is fast five minutes, slow is half an hour. A lot of time is wasted by engineers when it is necessary to make frequent distributions.

We can actually use some tools to implement a process like this:

Git Merge to Master (usually this means a Release) → triggers Jenkins build to build ipa→ automated distribution.

Automatic distribution includes the following tasks:

• Automatically upload App Store
• Automatic distribution with TestFlight
• Automatically upload to the Dandelion/fir and other platforms
• Automatically upload to the Enterprise App Store

We can configure the relevant scripts separately to implement the test distribution, and release.

To implement such a process, we need these tools:

• Jenkins
• Fastlane

Fastlane is a tool made up of small components that include uploading to iTunes Connect, creating provisioning file, Managing TestFlight testers, distributing, etc. The tool can use almost all of the features of the command line to package upload and distribute, and it is increasingly becoming the default industry continuous integration standard tool. Just like CocoaPods in the management of dependency.

Summarize

Continuous integration can be seen as a workflow through a version control system, a CI platform (such as Jenkins) and related toolchain to reduce repetitive work on the team and ensure that the software can iterate without too much error. IOS under the continuous integration is basically the above several pieces of content, we have encountered a lot of holes in the process of implementation, the following pieces of the article in detail.

IOS Continuous Integration Series-automated Code Review

Code Review is a very important part in order to ensure the quality of the codes. It is * within the scope of this work to see whether the code's structure conforms to some of the basic principles of software development, in the right place.

Limited to the reality, most teams do not have enough time to Code Review, then only a part of the CR work to the computer to complete. We just need to set a reasonable process, use code to tell the computer what needs to be done, and leave the rest to our reliable partners.

With the automated code Review applied, Xcode will not be happy if you write poorly.

If you ignore Xcode's mood, then the quality management platform will silently record all this.

This helps developers to write higher-quality code, and gives managers a cut-off on the quality of the project, while at the same time requiring less human maintenance, which sounds like an itch:)

Process

The overall workflow is very simple,

The key point is the local review and remote review two steps. The former provides developers with an immediate code quality feedback that developers can modify to avoid getting a lower score in the next remote Review. The latter is to generate related reports for project managers to track the quality of the project to provide a basis. In many large companies, this is one of the developers ' performance references.

The rest is some glue steps, how to make the process more automated, is the glue step to do. For example, the use of WebHook automatically trigger remote Review, using Git hooks for incremental check instead of full-volume calibration. Let's take a look at the process of local calibration before we talk in the back.

Local Review

In the local Review link, the developer only need to press CMD + B as usual, and then just wait for the progress bar to finish reading, full screen?? It will indicate exactly which rule the code of a line violates. At this point the developer can make the corresponding changes according to the code specification.

There are a few things that have happened from the press of a button to a warning:

• Generate Compile_commands.json File
• Oclint read the relevant Rules and scan the. m files in Compile_commands.json one by one
• Oclint the generated report on Xcode

The core of implementing local Review is Oclint and Compile_commands.json files

Oclint

工欲善其事, its prerequisite

Oclint is an open source, based on Clang, written in C + + and can be used in C, C + + and objective-c static code analyzers. It can dynamically load rule files during the scanning process, thus enabling very flexible, highly customizable code analysis scenarios. It can be seamlessly integrated with most systems, such as Cmake, Bear, Xcodebuild, Xctool, Xcode, Xcpretty, Jenkins ci, Travis ci, and more. Here you can find out how to use it with Xcode.

The latest version of Oclint has brought 71 rules, which are basically valuable experiences for the ancestors, such as the rule that disables the Goto statement, which is a manuscript from Edsger W. Dijkstra for 1968 years.

These 71 rules have helped us avoid some of the problems caused by writing habits and language myths, but clearly not enough for companies with full coding specifications. We have to develop Rules for ourselves.

Fortunately, Oclint has prepared everything for us.

Oclint provides a layer of encapsulation of the Clang and AST (abstract Syntax tree) so that we do not have to parse the abstract syntax tree, just focus on the logical development of the rules-related. We can clearly see this from the interfaces it provides.

// encountered unary operator BOOL Visitunaryoperator (Unaryoperator *node)//  encountered two-dollar operator BOOL visitbinaryoperator ( Binaryoperator *node)//  encountered objective-c function declaration BOOL visitobjcmethoddecl ( Objcmethoddecl *node)  

After developing the relevant rules, packaging them into dylib, you can load our own rule at the time of analysis.

Compile_commands.json

Compile_commands.json is a specification of the Clang definition, which stores a set of working directories, target files, commands that need to be executed, and helps the tool to convert the source code files to AST and do the corresponding things independently of the compiled system.

Looking at the contents of the file is more intuitive:

[{  "Directory":"/path/to/project/",   "Command":"/applications/xcode.app/contents/developer/toolchains/xcodedefault.xctoolchain/usr/bin/clang-x ...",   "file":"/PATH/TO/PROJECT/XXXVIEWCONTROLLER.M"},...]

Oclint can check the source code files in bulk according to the contents of the Compile_commands.json.

Xcpretty

One more point to watch out for is, how do I generate a Compile_commands.json file?

The most convenient way is oclint-xcodebuild to use to generate. First, generate the Xcodebuild.log file using Xcodebuild.

xcodebuild | tee xcodebuild.log  

Then use the oclint-xcodebuild build Compile_commands.json

oclint-xcodebuild  

As of Xcode 8.1, this practice can generate JSON files correctly. Because the Oclint team has claimed to no longer maintain oclint-xcodebuild, this method may no longer apply in a future version of Xcode.

Another recommended approach is to use Xcpretty.

Xcpretty can generate a JSON file in one sentence.

xcodebuild | xcpretty -r json-compilation-database --output /path/to/compile_commands.json  

Using local Review

Knowing these tools makes it easy to understand how local automation Code Review works and how it can be easily understood:

1. First install the Oclint locally on the computer and get the company custom Rules file
2. Configure the project on Xcode
3. Build project and wait for the results to appear on Xcode.

Attach a configuration script for our team for your reference:

SOURCE ~/. Bash_profile  cd ${srcroot}  xcodebuildClean | tee xcodebuild.log  oclint -xcodebuild  oclint-json-compilation-database \  ----- max-priority-1100000-max-priority-2100000- max-priority-3100000-report--r/path/to/rules

Distal Review

The remote Review and local Review are broadly similar, and the object that distinguishes the script that is built with references from Xcode to Jenkins CI, the report's display from Xcode becomes the SonarQube. The process is this:

The engineer git push submits a code →web Hook trigger Jenkins to build →oclint scan code to generate a PMD format report →sonar-runner read the report and present it to SonarQube.

CI Environment

In order to realize the remote Review, the server must have a CI environment first. Given the peculiarities of IOS, the server must be a MacOS system. CI We directly choose the open source Jenkins, the quality management platform chooses the open source SonarQube. Jenkins famous people are very familiar with, SonarQube is relatively small person understand.

SonarQube is a quality management platform, on SonarQube, you can see the number of lines of code, number of files, code repetition rate, violation of code specifications, technical debt time, and so on. SonarQube support for Java is extremely friendly and provides the Sonarscanner to scan Java source code directly. Objective-c was not so fortunate. Although SonarQube also provides support for Objective-c's presentation, static analysis relies on Oclint.

Sonnar-runner

We ran oclint on Jenkins to generate a report. Need an intermediary to parse the report into a SonarQube understandable format and transfer it to the SonarQube platform. This middleman is Sonnar-runner. Sonnar-runner only plays the role of the porter in our system. From here you can learn how to install and use Sonnar-runner on Jenkins.

Sonnar-runner can only parse reports in PMD format, so we need to export the report format to PMD format after we use Oclint to parse the code.

oclint -report-type pmd -o ./report.xml  

Rules in Sonar

SonarQube has a set of rules to divide code problems into 5 levels according to severity, and different levels of issues will affect the project quality score with different weights. This set of rules and the rule name in the report generated by Oclint must correspond to each other and SonarQube correctly categorize and score the issues in the report.

If you use Oclint native Rules to check your code, simply install the SonarQube Plugin for Objective C plug-in on the SonarQube and the related reports will be correctly identified.

If you are using a self-developed Rules, just Clone the above plug-ins and add the relevant rule name to Profile-oclint.xml and Rules.txt, then package and install the plugin on the SonarQube.

As an example:

When we checked the code with the self-developed Rule, the Report.xml was generated, with the following content:

<?xml version="1.0"encoding="UTF-8"?> &LT;PMD version="oclint-0.11"> <file name="/PATH/TO/TERRIBLECODE.M"> <violation rule="binary operator space (Ht_ios_coding_style 2.8)"begincolumn="9"endcolumn="157"Beginline=" the"Endline=" the"priority="3"ruleset="Ht_ios_rules">at least one space is required between the multivariate operators and their operands</violation> </file></pmd>

Which binary operator space (HT_iOS_Coding_style 2.8) is the error we defined in rule name. On SonarQube, it is also necessary to have a rule name in order to correctly identify the error.

At this point we only need to add a section in the rules.txt of the above plugin

operator 2.8 )  ----------  2 Category:hengtian iOS Coding standard  

Add another piece of code to the profile-oclint.xml of the plugin above

<rule>      <repositoryKey>OCLint</repositoryKey>      operator2.8) </key >    </rule>

Then the plugin is packaged and installed on the SonarQube, SonarQube can correctly identify our problem and classify it.

Using the remote Review

Before using it, make sure your MacOS server has the latest version of XOCDE, Oclint, Jenkins, Sonnar-runner, installed the relevant plugins for Jenkins, and places the custom Rule on the server, if any.

Review and generate reports

Create new projects on Jenkins and configure git, build triggers, and more. Add a step Execute Shell to the build step and fill in the script below

 cd yourprojectdir xcodebuild clean xcodebuild -workspace Myproject.xcworkspace-scheme HTMARKET-SDK Iphonesimulator | Tee Xcodebuild.log |  Xcpretty oclint -xcodebuild oclint - Json-compilation-database-e Pods \ -v --< Span style= "COLOR: #000000" > -max-priority-1  100000   -max-priority-2  100000   -max-priority-3  100000   -report-type PMD -r/path/to/diy-rules -o/path/to/report.xml 

The script is roughly the same as the local Review, and there are three places to look at.

1. xcodebuildCommand adds a -sdk iphonesimulator parameter to avoid the problem of the build requiring Code sign.
2. -report-type pmdThe output format must be in PMD format
3. -o /path/to/report.xmlNote the path to the output report, which is used when the next sonnar-runner is read.

Read to SonarQube

Add one more step at the bottom of the previous step, Invoke Standalone SonarQube analysis, and choose your sonnar-runner. and add the following configuration to the analysis Properties : (If this is not the case, you may need to install SonarQube related plugins.) ）

sonar.projectkey=your_project_name Sonar.projectname=your_project_name sonar.projectversion=1.0Sonar.language=OBJC sonar.projectdescription=your_project_description# Path to source directories sonar.sources=/path/to/source/directories# Xcode project configuration (. xcodeproj or. xcworkspace) #-If you had a project:configure only sonar.objectivec.project#-If you had a workspace:configure sonar.objectivec.workspace and sonar.objectivec.project# and use the later to spec Ify which project (s) to includeinchThe analysis (comma separated list) Sonar.objectivec.project=your_project_name.xcodeproj Sonar.objectivec.workspace=your_project_name.xcworkspace# Scheme to build YOUR applicationsonar.objectivec.appScheme=your_project_namesonar.sourceencoding=utf-8# Oclint Report generated by run-sonar.sh isStoredinchsonar-reports/oclint.xml# Change it onlyifYou generate the file on your own sonar.objectivec.oclint.report=your_report_file_path

Note the comments and modify YOUR_PROJECT_NAME , YOUR_PROJECT_DESCRIPTION and the YOUR_REPORT_FILE_PATH values for your project.

If all goes well, build immediately on Jenkins and you'll be able to see the code Quality Report on your Sonar platform.

With the WebHook capabilities of building triggers and Git platforms, you can automatically trigger builds at key points such as developing commit code or merging branches.

Troubleshooting

Why the generated Compile_commands.json is empty

Check if log is empty, and if log is empty, the build fails. The cause of the failure can be generated normally.

The Jenkins build encounters the following issues

?  Code signing is required for product type ‘Application‘ in SDK ‘iOS 10.0‘

In this case, the release version was built, and the project opened Automatic Code sign on xcode8+. Here's how to fix it:

1. If you only need to check the code specification, add a parameter after the Xcodebuild command to indicate that it is -sdk iphonesimulator built in Debug mode.
2. If you want to build the release version, turn off automatic signing and manually configure the certificate and proversion profile on the CI system. or retain the automatic signature, refer to this answer use the sed command to modify the relevant configuration before building.

Reference links

• Oclint
• Jenkins
• SonarQube
• Edsger W. Dijkstra
• AST (Abstract Syntax Tree)
• JSON compilation Database format specification
• Xcpretty
• QMe
• Xcode8 and IOS10 post-upgrade issues focused discussion posts--testerhome

• [IOS Continuous integration-automated unit testing]
• [IOS Continuous integration-automated packaging and distribution]

IOS Continuous Integration