Continuous integration: Accelerated test execution from six levels

In the field of continuous integration, the release of a product often has its own process cycle (lifecycle), the general will be divided into: Build-> deployment-> test-> released several important stages, including testing is to release the important stage before the product is the guarantee of product quality. While continuous integration works, the test scripts are required to run faster and better, in addition to the need for more robust test scripts. This may seem irrelevant to small projects, after all, most of the small project test scripts but hundreds, the verification point is not thousands of "point"; but for a large project, the test code source files may be hundreds of thousands, all the tests may have to wait a long time, and the result of the bitter wait may be the failure, So if the test execution speed becomes an urgent need to solve the problem, imagine the test phase from 2 hours to 1 hours, and then compressed from 1 hours to 30 minutes, each time compression brings not only the technical staff itself, but also the whole process of product release experience improvement.

So how do you speed up the test execution? Lifting speed, we may immediately associate with the "Performance" Tuning steps: Advanced line tuning, and then find the bottleneck of the problem, and finally one by one. This article does not discuss how to do these steps, but based on C + + and Java for language cases, testng and Google test for the testing framework, Jenkins for the continuous platform analysis, from the following six levels to improve the general method of test execution:

Hardware resource Hierarchy

工欲善其事 its prerequisite, improve the hardware (CPU, memory, disk, etc.) configuration is to improve the implementation speed of the "hard" method, the optimization of hardware resources should not only be limited to the single individual indicators to promote, in the case of increasing demand, you can consider the implementation of virtual machines, Distributed clustering and other ways to further obtain better hardware resources, of course, involving distributed implementation, you can use the following continuous integration platform level of "soft" implementation to work together.

In addition, in the case of hardware resource constraints, different projects or different teams may have to reuse a test environment, resulting in more tension in the availability of resources, you can stagger the time test (such as a project team test scheduled to start at 0 o'clock in the morning, B project group scheduled at 2 o'clock in the morning) to improve speed.

Implementation level of language coding

The test code itself is also code, it is obvious that if the code is written to pay attention to efficiency, speed will certainly be profitable. This may require "entanglement" with some everyday coding details: such as StringBuffer and StringBuilder comparisons in Java, and C + + i++ and ++i comparisons. This language level improves the efficiency of the article books a lot, here do not do too much description. The most important thing at the language coding level is not the details of these languages, but the design of the test code that avoids some consumption time, reducing unnecessary time-consuming operations, such as the following:

(1) Redundant log information, unreasonable logging level settings, etc.

The disk frequent access to the output log is bound to reduce the speed, so in the guarantee of sufficient log information, minimize the log, or only the failure of the test log (after all, for testers rarely pay attention to the success of the log), you can make the test faster.

(2) Unreasonable waiting

The user performs an action, must wait for a condition to occur (for example, insert a new data in db) and then perform the following action is often in the test of the scene, then wait for how long to be considered as a problem, assuming the TimeUnit.MINUTES.sleep (1) Wait a minute, In 10 seconds to meet the conditions of the scene to waste is 50 seconds, so here must be considered reasonable sleep time to take into account the consumption of resources and the impact of the speed of the operation, while waiting for the way can also be considered is the use of a short cycle of time to wait or asynchronous notification way to proceed.

(3) The process of a use case

Run through all the test steps first, then do all the steps to do a one-time check, or to do a one-step calibration step, the speed of the two ways in different scenarios, so need to weigh; Similarly, for use cases that require a DB connection, each performs a fetch DB connection and then releases the connection, Or all use cases get the connection before execution, and releasing the connection after all case execution can also have an effect on the speed of execution.

Build test Script Hierarchy

For a large project, the large number of source files or dependent dependency too much lead to code compilation takes a lot of time, how to improve the speed of compiling code? In addition to using better disks, focus on the impact of code writing on the speed of compilation, you can also take different effective strategies for different languages, for example, for C + +, when you compile the project with the Make command, you can add parameter-j to compile the project in parallel. The meaning of the-j parameter can be referenced below:

-j [Jobs],--jobs[=jobs]

Specifies the number of jobs (commands) that are running synchronously. If you have more than one-j option, then only the last one is valid. If the-J option has no parameters, the compilation process does not limit the number of jobs that can be run synchronously.

It is necessary to note that the compilation process may require a particular order to cause parallel compilation to fail, and if this problem is encountered, it can be repeated once in parallel, serially (minus-j) to resolve.

For Java,maven 3 to support concurrent builds, there are several common ways to do this:

Mvn-t 4 Clean Install # builds with 4 threads
mvn-t 1C Clean Install # 1 thread/CPU core
mvn-t 1.5C Clean i Nstall # 1.5 thread per CPU core

While using MAVEN to manage Java projects often takes time on downloads that rely on jars, you can check for redundant repository configurations, longer download timeout time settings, the connection speed of the selected repository, etc. Even in different test environments you can use profile to manage repository to speed up test script construction.

Test framework Support Level

At the support level of the test framework, we should make full use of the rich functions of the framework itself to improve the test execution speed, taking the Java test Framework testng as an example:

(1) Using timeout control to fail to wait

If a test case waits for a long time to wait for a condition to be triggered, the long wait time is often invalidated for the use case itself, especially if the condition is never satisfied. It is therefore necessary to control the maximum timeout time allowed for use case execution. TestNG can set timeout time for test or test suite to control the maximum allowable time for the execution of one or a group of (testing.xml) automated test cases, respectively:

1. @Test (timeout = 1000)
2. Testng.xml: <suite name= "Module Test" parallel= "None" time-out= "200000" >

(2) using @beforetest, @BeforeClass and other conditions annotation, reduce meaningless test

The successful completion of the test needs to meet a number of basic conditions, such as the need to test the environment is ready, if you do not use the @before class tag, when the condition is not available, will still perform all the use cases, the inevitable time wasted, so using the @before class tag can avoid meaningless testing, @ Once the before tag method fails, subsequent tests will not continue.

(3) Multi-threaded support with frame self

For example, for testng itself, you can set the parallel parameter in testng.xml to specify whether concurrency and concurrent levels: Methods|tests|classes, in addition to the test framework itself, software project management tools can provide multithreaded support, For example, the test component Maven-surefire-plugin of MAVEN provides settings for concurrent parameters:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId> maven-surefire-plugin</artifactid>
<version>2.16</version>
<configuration>
<parallel>methods</parallel>
<threadCount>10</threadCount>
</ Configuration>
</plugin>