Regression test best practices-regression test case Optimization Selection and coverage analysis

Level: elementary

Dengjn@cn.ibm.com (mailto? Subject = regression testing best practices), software engineer, IBM
Mailto: huangssh@cn.ibm.com? Subject = regression testing best practices), Researcher, IBM
Chen Yun (mailto: agile158@gmail.com? Subject = regression testing best practices), Intern

March 13, 2009

This article introduces an effective solution to improve the efficiency and quality of regression testing. It solves two main problems in regression testing: how to optimize regression testing cases and analysis coverage.

Regression testing is of great significance for ensuring software quality. To achieve effective regression testing, two main problems in regression testing must be solved: (1) Optimization of test cases and (2) coverage rate analysis. The former determines the efficiency of regression testing. a good selection of test cases can accurately cover as many changes as possible in the new version with a small number of test cases. The latter is an important indicator of measurement testing. It ensures the quality of regression testing by achieving good test coverage.

This article discusses how to optimize the selection of test cases to maximize coverage at the minimum cost, so as to find an effective solution for regression testing.

Problems

Optimization of Test Cases

There are usually two methods for selecting test cases. One is to select the relevant or all test cases of the module for execution. The other is to test the fixed apar/defect only, there are few test cases or new test cases for this fixed are developed. Both methods are insufficient. The first method is to execute all the test cases when the test time is limited. It will test a lot of test cases that no longer need to be tested, resulting in a waste of test resources; second, it is difficult to ensure that the tested system is not affected after apar/defect changes, and it is difficult to ensure the test quality.

Because of the Bug fix or feature update, before the new version is released, make sure that the changes do not have a negative impact on the existing functional modules. Using all the test cases for regression testing, there is a problem of high labor and time costs; relying on human experience to select regression test cases, the selection is inaccurate or the test coverage for program changes is incomplete.

Figure 1. Optimization and selection of Test Cases

Regression test coverage rate analysis

The test case optimization option can effectively solve the coverage problem of existing test cases. However, during the actual test, we still find that there is a problem of incomplete coverage. That is, some implementations in the new version cannot be covered by existing test cases. In this way, testers need to develop new test cases to test and overwrite new functions. This is not practical for testers. It is difficult for them to find out the areas not tested in the code based on their testing experience. So how can we get the test coverage rate during the test and the test coverage rate of the entire software after the test is completed, so that the tester can design new test cases for areas not tested. Here we specifically aim at the updates that are not covered during the version update process.

Back to Top

Solve the problem

Principle

Figure 2. Optimization and selection principles of Test Cases

Figure 3. Example of test case Optimization

As shown in, all test cases have a function call path. Let's take down these call paths one by one. For changes made to the new version, all test cases related to upper-layer calls can be accurately selected, in this way, we can use these accurate test cases to cover the impact of this change. Unrelated test cases are not selected. Therefore, the regression test required for this change is completed at a lower cost, which saves time and effort and ensures a high testing quality.

Figure 4. Coverage analysis example

As shown in, test cases affected during version update are test case1, test case2, and test case3 covers 12 nodes. The update point during version update is 4, 3 is covered, and 1 update point is not covered. The update coverage capability of the existing test case set is 75%. In this way, we know that the test cases of the previous version are not fully designed and the program modules are not covered by any existing test cases. Because code updates are most likely to cause program defects or even system crashes, you need to add new test cases to ensure coverage of updates to reduce the risk of running programs.

The coverage rate of software nodes is refined to the function granularity. We learned about functions through binary code analysis of software deployment. The source code is not required. Therefore, the test coverage rate analysis of software is less demanding. the test case coverage feedback and analysis are completed at a lower cost. It saves time and effort and ensures high test quality.

Solution

Based on the principles of Question 1) and question 2, we have designed and implemented a regression testing solution, as shown in. It contains three main steps. One is the entry of test cases; the other is the change analysis of the new and old versions; the third is the test case Optimization Selection and coverage analysis, and the corresponding test case Optimization Selection Report is obtained, and coverage analysis report.

Figure 5 regression testing solution

Step 1: trace test case records test cases and stores the captured runtime trace of test cases in the database;

Runtime trace in the background of a test case is an important part of dynamic analysis. The actual running information creates conditions for the optimization selection and coverage analysis of test cases. The following is a framework diagram of test case tracking:

Figure 6. Framework of test case tracking

We can see that after the tester triggers the trigger, the agent controller will be started. The agent controller constantly monitors jvmti In the JVM to obtain the tested applications deployed on the JVM. These agent controllers are also responsible for transmitting collected data to data collector. Data Collector writes these runtime traces to the database table shown in the following table.

Case ID	Package	Class	Method	Signature
001	Com. IBM. CRl. orts. Action	Deletecommodityaction	Delete	([Ljava/lang/string;) V
001	Com. IBM. CRl. orts. Action	Deleteorderaction	Delete
002	......
003	......	......	......	......

Note: The signature information of the function must be recorded as the parameter identifier of the function. To differentiate functions with different parameters of the same name.

Step 2: change analysis compares the old and new versions to obtain the change report, that is, the program change report, which can be accurate to the method granularity. Generally, there are four levels of code changes: Package, class, function, and statement ).

For the package level and class level, the comparison is too rough, which will affect the quality of regression test optimization. Function level and statement level can play a good role in regression testing. The statement-level analysis results are the most accurate and the regression test quality is the highest because of the finest granularity. However, compared with the function-level change analysis, the quality of regression testing is quite different, but it causes too much execution time and affects the efficiency of regression analysis. Therefore, we use function-level change analysis as the change granularity of regression testing.

After determining the comparison granularity, you can select the analysis and comparison method. The simplest common comparison method is text comparison. Including source code and executable file (Binary Code) text comparison. Based on the object-oriented features of Java, you can also use a comparison method based on object-oriented analysis. The analysis granularity obtained by the latter is finer, but the cost is also higher.

Step 3: Obtain the specific runtime trace information of the test case through test case recording, and obtain the change information of the new and old versions through change analysis, we can solve the problem of test case optimization and coverage analysis.

In test case prioritization, test cases are obtained by matching test cases with running runtime trace. Test Cases are sorted by the test case optimization Sorting Algorithm to obtain the optimization results. There are many types of optimization sorting algorithms. We will not discuss the optimization sorting algorithms here.

In coverage analysis, the coverage report based on code update is obtained by analyzing the number of methods that have been covered by the relevant test cases and the number of methods that have not been covered by the test cases. After obtaining the report, the tester can find the method that is not covered and develop new test cases accordingly. To overwrite new features.

We can see that step 1 and Step 2 jointly serve the test case Optimization Selection and coverage analysis modules. With the runtime trace and Change Report of the test case, we can match the change result with the runtime trace to find the test case that can overwrite the program change. Similarly, for the Change Section not covered by the test case. Because no relevant test cases are found, our existing test cases are insufficient. We need to develop new test cases for the unoverwritten change section.

As an important indicator of software testing, the coverage rate can be solved based on the methods that have been covered and not covered, that is, the ratio of the number of covered change methods to the total change method.

Result

Based on the above regression testing solution, we perform regression testing for a program with 30 test cases. The following test case Optimization Selection analysis report is obtained:

Change Analysis Report

Total Functions	Change Function	Overwrite count	Not covered	Coverage
108	12	10	2	83.3%

Table 1 Optimization and selection of Test Cases: 3 (of total 30)

Case name	Cover changed Method
Testcase001	5	Details
Testcase005	3	Details
Testcase013	2	Details

The analysis report shows that a total of 12 functions have been changed in this code change. Three of the 30 test cases are associated with these changes, covering 10 of the 12 code changes. The other 27 test cases are irrelevant to the 12 code changes.

Table 2 regression test result analysis

All test cases	Optimization Selection	Function Change	Coverage Analysis
			Overwritten	Not covered	Coverage
30	3	12	10	2	83.3%

From the table, we can see that after the test case optimization selection, we have selected three test cases related to function change, with a coverage rate of 83.3%. At the same time, because 27 test cases unrelated to function changes do not need to be retested, 90% of regression test resources are saved.

Figure 7. Coverage Rate Analysis

We can clearly see the number of test cases based on each function change and test coverage. For example, there are two change methods in the class managecommodityaction. Only one changed method is overwritten by one existing test case, and the test coverage rate is 50%.

In the above analysis report, a total of 12 functions were modified, and 2 of them were not covered by any test cases. The change method that is not covered requires the tester to analyze it and add new test cases to improve our test coverage and ensure the test quality.

Back to Top

Summary

Optimize the selection of regression test cases and accurately overwrite the changes with the least test cases, which greatly improves the test efficiency and quality of regression test.

The coverage rate feedback analysis in the automatic testing process is used to obtain the current test completion status at the lowest test cost and the most accurate analysis. This improves the analysis report for testers, this allows testers to improve and add new test cases. This greatly improves the efficiency and quality of regression testing.

Disclaimer

This document does not provide any express or implied warranty. The suggestions or best practices provided by the article are shared only as common experiences. The author does not guarantee that these suggestions or best practices are effective under any circumstances. Any subjective statement in this article only represents the author's personal opinion and does not represent the official position of IBM. For details, consult the author directly.

References

Learning

• "Rational edge: accelerating regression testing: getting started with cumulative test analysis and target testing" (developerworks, October 2006): This article is from rational edge: the "cumulative test analysis" technology provides software testing teams with a more rational approach to automated testing, especially in the field of regression test sets. Understand how CTA improves your testing efficiency.

• "Cross-platform automated regression testing based on rft and staf without manual intervention" (developerworks, August 2008): This article combines rational functional tester (rft) with the degree of automation of regression testing) perform cross-platform concurrent automated regression tests without manual intervention.
• Browse the technology bookstore to get books on these and other technology topics.
• Developerworks Java Technology Zone: find hundreds of articles on Java programming.

Discussion

• Join developerworks blogs and join the developerworks community.

Author Profile

		Deng junning, software engineer, ibm cdl taas (test as a service) Competency Center Department. He has been engaged in IT software development and design since 2000 and has rich experience in Java and web development. Currently, the IBM software development center is engaged in software testing and is one of the core members of the taas automated testing workgroup.

		Huang Sheng, Researcher, ibm crl Service Building Technology (SBT) department. Currently, the ibm crl sbt testing team is one of the core members of the testing team. The main design personnel and Development Team Leader of the regression testing solution.

		Chen Yu, an intern at IBM China Research Institute. One of the main members of the regression testing solution development team.