Introduction to test metrics

Introduction to test metrics 23:32:32

Four metrics are defined during the cmmi4 system test: test coverage rate, test execution rate, test execution pass rate, and test defect resolution rate. To enable dedicated/part-time testers to understand these four metrics and how to use existing resources to collect metric data, this article introduces the meaning of these four metrics and the data collection method. 1. Test Coverage test coverage refers to the requirements of test cases. Calculation formula: number of requirements/total number of requirements for tested cases. Test Coverage covers both breadth and depth in terms of dimensions. Test Coverage covers user scenarios, feature coverage, feature combinations, and system scenarios. First of all, whether or not each requirement item in the requirement specification is designed in the test case. Secondly, let's talk about depth. In layman's terms, it is not to let our test design flow on the surface. Can we find out the potential problems through the customer's requirement documents. For example, you can click a button 10 times, or add, delete, or delete a record of the same data in sequence. In my actual work, I have encountered an example where a system written in PL/SQL repeatedly clicks the query button six times on a query interface, the query function becomes invalid. After debugging, developers found that the GDI resources were not completely released. When designing test cases, we seldom design test cases in terms of breadth or depth separately, but generally design them together. To cover test cases in both breadth and depth, we need to consider designing various test cases, such as user scenarios (identifying the most commonly used 20% operations), Function Points, function combinations, system scenarios, performance, statements, branches, etc. During execution, the test must be executed in a certain order based on the ample test time. Generally, the test case of the user scenario is executed first, and then the test of specific function points and function combinations is executed. We can use the requirement tracking matrix RTM to collect test coverage data. In the requirement tracking matrix, the tester fills in the data in the "System Test Cases" column. The tester can quickly calculate the test coverage rate by calculating the number of requirements listed by RTM and the number of requirements for the designed test cases. Through RTM, testers, including project team members, can quickly understand the test coverage of the current project. Note: In this RTM example, the "Outline Design", "Detailed Design", and "encoding" columns are hidden to show only the content related to the test coverage rate calculation. 2. test execution rate: as the name suggests, it refers to the ratio of tested cases that have been executed in the actual execution process. Calculation formula: Number of executed test cases/total number of designed test cases. Readers may find it strange that all the test cases we designed must be executed. Even if the test is executed by module, the test execution rate of this module is certainly 100%, why do we need to set this indicator? Actually not. In the actual test process, the following situations often occur. One case is that the system uses iterative development, and each build has a different focus, including different content. The second case is that due to limited test resources, it is impossible to test all the design tests each time. Because of these two situations, we will arrange test activities according to different test focuses and content during each test execution, therefore, the "test execution rate" indicator exists. Generally, our purpose is to ensure that 100% of test cases are executed, that is, the execution rate is 100%. However, as mentioned above, there may be a non-100% execution rate. If the test execution rate cannot reach 100%, we need to develop different test execution rate standards based on different situations-mainly Considering risks, importance, and acceptable test execution rate. When considering the acceptable test execution rate, the test case execution sequence is involved. When designing test cases, we need to cover the requirements as much as possible in terms of breadth and depth. Therefore, we need to design various test cases, such as normal test cases, abnormal test cases, and interface test cases. However, when executing the test, the tester needs to execute the test cases in a certain order based on the project progress and test time adequacy, and according to the test execution rate standard. Generally, the test case corresponding to the user scenario is executed first, and then the test of specific function points and function combinations is executed. After these tests are completed, other tests are performed, such as system scenarios, performance, and statements. For example, a project has a total of 280 test cases. In a certain phase of the project, the test is divided into four versions. If one version executes 134 test cases, the test execution rate of this version is 47.9%. 3. The test execution pass rate introduces the execution result definition. Let's look at the test execution pass rate. Test execution pass rate refers to the ratio of test cases in which the execution result is "passed" in the actually executed test cases. Calculation formula: Number of test cases whose execution result is "passed"/total number of actually executed test cases. We can measure the test cases of all planned executions and refine them to specific modules to compare the test cases of each module. To obtain the test execution pass rate data, we need to record the execution results of each test case in the test case copy during the test execution, and then complete the execution in the current version, or regularly (such as weekly) Statistics on the current test execution data. Through record and statistics of raw data, we can quickly get the test execution pass rate of the current version or the current stage. 4. defect resolution rate: the ratio of closed defects to the total number of defects in a certain stage. A defect closure operation can be performed in either of the following situations: the defect is fixed and has been verified by the tester; forced closure: Repeated defects; defects caused by external reasons; defects that are not processed temporarily; invalid defects. After such defects are confirmed, they can be forcibly disabled. Calculation formula: the total number of closed defects/defects in the project process, the defect resolution rate increases slowly at the beginning, with the development of the test work, the defect resolution rate gradually increases, before the release of the version, the defect resolution rate tends to be 100%, as shown in figure 2. Generally, when each version is released, the defect resolution rate should reach 100%. That is to say, except for the repaired defects that need to be verified, other defects that need to be forcibly closed must be confirmed and corresponding countermeasures should be taken. The defect resolution rate can be used as a standard for testing and version release. If some defects are still on or processed, this version is not allowed to be released in principle. Defect closure data. You can use the defect tracking tool to periodically collect the number of defects in the current system and the number of closed defects, the defect resolution rate curve of the entire project process or a certain stage can be drawn. Of course, the test metrics include not only the above four indicators, but also indicators such as verification failure rate and defect density in actual work. This data is collected to quantitatively manage the test process. However, a simple collection of metric data is not an aim. It is an aim to reduce risks by analyzing, preventing, and taking corrective measures.