Continuous integration and test automation

I. background

I have worked in three companies of different sizes since I graduated: private, foreign, and listed companies. However, in the past, most of them were engaged in projects and software development, and most companies did not pay much attention to testing. Even if there was a scale, they did not build a testing system. In short, the management philosophy of light development and testing has continued in China for decades and will continue, look at the low salaries they offer to test engineers, and when they talk about tests in class, they will know that the tests are ignored by Chinese bosses.

In the past two years, I have been engaged in testing and Project Management of CRM software products. Because the company has high requirements on software quality, this will inevitably attract everyone's attention to the testing work, not only requires a strong testing team, the team must have a professional level in terms of business and testing skills, and the software development process is constantly adjusted due to testing.

Fortunately, with the improvement of software development technologies and tools and the promotion of software engineering and software process practices, software testing is becoming increasingly valued and specialized. During my testing work, I have been researching CMM, testing theory, automated testing tools, and established a complete testing system.

We will not introduce the entire test system here, but the most noteworthy part of the test: continuous integration and test automation. The purpose is to make progress together with everyone. Of course, there are already many introductions on continuous integration and automated testing, but what I want to introduce is not just continuous integration, but automation testing.

Ii. test automation

Automated testing is to use automated testing tools or other means to carry out automated testing according to the testing engineer's schedule. The purpose is to reduce the labor cost of manual testing and improve the software quality. Automated Testing aims to discover old defects. The purpose of manual testing is to discover new defects.

Testing automation involves integration of testing processes, testing systems, automated compilation, continuous integration, automatic release of testing systems, and automated testing. That is to say, automation of testing is not only a problem of technology and tools, but also a cultural issue of companies and organizations. First, the company will support you in terms of capital and management, and secondly, we will have a dedicated testing team to establish a testing process and system suitable for automated testing; the second step is to extract, compile, integrate, and release the original code from the controlled database to run the system, perform automated unit tests, and perform automated functional tests.

(1) benefits of Automated Testing

1. Perform a regression test on the new version-test each feature

Every time a new version is released for product software, most of the features and interfaces are similar or identical to the previous version. This part of the function is particularly suitable for automated testing, this allows testing to test each feature.

2. More frequent tests-dull and time-consuming

The release cycle of our products to the market is three months, that is, our development cycle is only three months in short, during the test, a version is released every day/every two days for testing by testers. There are thousands of functional points in a system. Manual testing is very time-consuming and tedious, this will inevitably lead to low testing efficiency.

3. Difficulties in replacing manual tests-some non-functional tests for 300 users: stress testing, concurrency testing, large data volume testing, and collapse testing are impossible. Before introducing the automated testing tool, for the purpose of testing concurrency, the passwords of No. 1 and No. two hundred developers in the R & D center are as follows: 1-, 2-, and 3 !, You can press the same button at the same time. It is also interesting to recall this scene.

4. Consistency and Repeatability

Since the scripts run in each automated test are the same, it is difficult for people to ensure consistency in each test. due to the consistency of automated testing, it is easy to find any changes to the software under test.

5. Make better use of resources-weekly/evening

The ideal automated testing can run completely automatically as planned. When developers and testers cannot perform three shifts, the automated testing can be competent for this task and can perform tests on weekends and evenings. this makes full use of the company's resources and avoids waiting between development and testing.

6. Resolve conflicts between testing and development

Usually at the end of the development stage, it enters the integration test phase. Because there are fewer errors in the test system at the early stage of each release, developers have to wait for the tester to test the error. in fact, there are more contradictions in the development mode with a short stack generation cycle, but automated testing can solve the main contradiction.

7. Increase software Trust

In short, the benefits and benefits of automated testing are obvious, but only the fact that automated testing is successful can benefit from it.

(2) Automated Testing-misunderstandings and restrictions on automated testing have many advantages, but there are also many limitations, because many bosses have too high expectations for automated testing, therefore, there are many examples of failed automated tests.

1. It is expected that automated testing can replace manual testing.

Automated tests cannot be expected to replace manual tests. Manual Tests are the main task.

2. Expect automatic testing to discover a large number of new defects

Likewise, automated testing cannot be expected to discover more new defects. As it turns out, the more new defects there are, the more likely the automated testing will fail. Discovering more new defects should be the main purpose of manual testing. Test expert James Bach concluded that 85% of the defects were discovered manually, while automated tests only found 15% of the defects.

In fact, I think automated testing can well detect old defects.

3. The tool itself is not imaginative

After all, tools are tools. There are some tests that require consideration, experience, and beautiful UI. Automated testing tools are powerless.

4. Technical Issues, organizational issues, script Maintenance

There is a lot of resistance to the implementation of automated testing, such as whether the Organization attaches importance to it, whether such a testing team is set up, whether there is such a technical level, and the maintenance workload for the testing script is quite large, whether maintenance is worthwhile or not must be considered.

(3) Not Suitable for automated testing

Automated testing is not suitable for all companies and projects.

1. customized project (one-time)

For projects customized for the customer, the maintenance period is borne by the customer, and even the development language and running environment used by the customer are also the special requirements of the customer, that is, the company has little experience in this area, such projects are not suitable for automated testing.

2. Projects with short project cycles

The project cycle is short and the test cycle is short, so it is not worth the effort to invest in automated testing. It is unrealistic to establish a test script that cannot be reused.

3. complex objects of business rules

Objects with complex business rules have many logical and operational relationships, making it difficult to test tools.

4. beautiful appearance, sound, and usability tests

In terms of human perception: the appearance of the interface, sound experience, and ease of use tests are only tested by people.

5. Few tests run: only once a month

Testing rarely runs, which is a waste of automated testing. Automated testing is to make it more efficient and repetitive.

6. Software instability

If the software is unstable, the automated test fails due to these unstable factors. Automated testing can be started only when the software is relatively stable and there are no serious interface errors or interruption errors.

7. Physical interaction

It is difficult for tools to interact with physical devices, such as card swiping tests.

(4) What kind of situation is suitable for automated testing? The reason why automated testing can be implemented in many large companies is that it has the characteristics suitable for automated testing and high ROI.

1. Product Projects

For product projects, each project only improves a small number of features, but each project must repeatedly test those functions that have not been modified. This part of the test can completely be undertaken by automated testing, and new function tests can also be slowly added to automated testing.

2. incremental development and continuous integration projects

Because this development mode frequently releases new versions for testing, automated testing is required for frequent testing to free people from testing new functions.

3. systems that can be automatically compiled and released

To fully implement automated testing, you must be able to perform automated compilation and automated release of the system for testing. Of course, automated testing can also be performed under manual intervention if this requirement cannot be met.

4. regression testing

The strength of the regression test automation test, which can ensure that you have introduced new defects and the old ones have been modified. To some extent, automated testing tools can be called regression testing tools.

5. Repeated and mechanical actions

Automated testing is a favorite test: repeated and mechanical operations. Such a test will never fail. For example, you need to input a large amount of similar data to the system to test the pressure and report.

6. Frequent tests are required.

In a project, you need to run the test frequently. the test cycle is calculated by day to maximize the use of the test script and improve work efficiency.

7. Converting cumbersome tasks into automated testing

Iii. Continuous integration and automatic compilation

The concept of "continuous integration" comes from a practice of XP (eXtreme Programming). Our development model is based on CMM and introduces some XP concepts, therefore, our idea is to take the essence of all aspects to suit ourselves.

Continuous integration is the process of automatically integrating submitted (check-in) code until it is released to the test server for testing.

1. To achieve automated daily building, you need to do the following:

2. Save all source code on a single development server, so that everyone can obtain the latest source code from it (use configuration management tools to store Source Code such as VSS/CVS/clearcase ).

3. Make the creation process completely automated, so that anyone can enter only one command to create the system.

4. Make the test completely automated, so that anyone can run a complete system test with only one command.

5. Ensure that everyone can get the latest and best executable files.

6. Automated Compilation: To provide automated testing, all code must be able to implement automated compilation. In fact, many companies that are engaged in continuous integration have implemented the change function: for example, Java program % 8 E6 continuous improvement, the development of continuous integration and testing automation of software projects is inevitable, its role will become more and more obvious. Different technologies and development environments have different requirements on How to automate tests. There are still many areas worth studying.