Automated regression test case evaluation criteria merest

The automated regression test case evaluation criteria try to define different dimensions to evaluate the advantages and disadvantages of automated cases, which serves as the basis for subsequent evaluation and discussion of the test framework, test technology, and test case writing mode. So what is a good automated regression test case? In short, it is a case with a high Roi. Because the return of a case is relatively stable, the main feature of a high ROI test case is low cost, this cost should be the full lifecycle cost of a test case:

Test Case full-cycle cost =

Initial Development Cost

+
Execution cost * E

+
Average error locating cost * F

+
Average reconstruction cost * m

 

Here:

LInitial Development CostIt refers to the cost of case development, debugging and optimization. In fact, it is mainly determined by the testing framework and testing tools used;

LAverage execution costIt refers to the case execution cost, which is mainly caused by temporary computing resources,

LEIt refers to the number of executions, which should be thousands or tens of thousands of times;

LError locating costIt refers to the cost of discovering, locating, and solving problems after an error is reported in an automated case,

LFIs the number of errors, generally dozens of times;

LAverage reconstruction costIt refers to the average cost of restructuring the case to cope with changes in demand;

LMIs the number of refactoring times, which may be dozens of times, depending on the frequency of changes required.

 

 

With the above cost model, we can discuss the evaluation criteria for automated regression testing cases (merest standards ):

1.Maintainable (maintainability), primary Metric): The case should be easy to maintain and expand to cope with future demand changes. In the full-cycle cost of the test case, the total weight of a normal case constitutes a big share. Therefore, maintainability of test cases is crucial;

2.Efficient (efficiency), secondary indicator (cocould): The case should be protected as many risk points as possible in the shortest time; improving the efficiency of the case helps reduce the number of test cases, without affecting other indicators, using fewer cases as much as possible to cover the same risk points helps reduce the total cost of the test case set;

3.Readable (easy to use), primary Metric): The case should be simple and easy to understand. In the case development stage, the case communication and communication will be difficult, the initial development cost will be affected, and more importantly, it is also an important factor affecting the reconstruction cost, because the person who maintains the test case may not be the person who originally developed the case, and the premise for reconstruction is to be clear;

4.Easy-to-use and secondary metrics): Cases should be very easy for testers to use, for example, easy to debug, easy to locate problems, and easy to re-run failure cases; this indicator can affect development costs to a certain extent, and the cost of mispositioning and reconstruction can be achieved;

5.Sensitive (sensitivity), required (must): The purpose of a case is to protect key requirements and prevent specific risks. Therefore, the case must have sufficient risk sensitivity and provide early warnings when a problem occurs. A test case that never fails is not a good case; sensitivity is actually the minimum functional requirement of the case. A case that does not have sufficient risk sensitivity is an invalid case. For example, unit test cases without assert or just assert function return values are not sensitive cases;

6.Trustable (reliability), required (must): The case should avoid random failures caused by the case. If the case fails, it is best to indicate that the corresponding system has problems. reliability is an important indicator because random failures occur if the case is unreliable, this greatly increases the number of errors and leads to the total cost of test cases.

 

After analyzing the merest principles, we can discuss the testing framework, testing technology, and test case writing mode.