28th back to software testing process and quality measurement

There are many Process metrics or projects in the test phase, including the Software Test Progress, test coverage, test defect appearance/arrival curve, test defect accumulation curve, and test efficiency. During the testing process measurement, the measurement should be based on software scale measurement (such as function points, object points, etc.), complexity measurement, project measurement, and other methods, complete measurement of the test process state from three different measurements:

1. The number of requirements provided by the measurement of the test breadth (among all requirements) has been tested at a certain time point to measure the execution and Test Progress of the test plan;
2. The test depth is a measure of the percentage of independent basic paths covered by the test to the total number of basic paths in the program. The measurement of the basic path quantity can be calculated using the McCabe ring computing complexity method.
3. Measurement of the number of defects collected during the process, and the differences and development trends among the number of detected, corrected, and closed defects, it provides an important basis for Process Quality, additional investment in development resources, and software release prediction.

As mentioned above, the measurement of the test process can combine the measurement of the process state with the measurement of the process result to make the measurement of the test process more effective.

In the testing phase, the main process quality measurements include:

• Defect Measurement or Distribution Measurement
• Depth, quality, and effectiveness of Test Cases
• Test execution efficiency and quality
• Quality of defect reports
• Test Coverage (test overall quality)
• Stability or effectiveness of the test environment

Defect measurement is the main measurement content in the test phase, including product Defect Measurement and defect process measurement. The product defect measurement will be detailed in the next time. The stability or effectiveness measurement of the test environment is measured using MTTF just like the software effectiveness. So next we will briefly introduce other metrics, such as software defect arrival mode, PTR appearance/backlog model, test case measurement, requirement-based test coverage evaluation, and code-based test coverage evaluation.

1. Time-based defect arrival Mode
The defect density of the product or the defect rate in the test phase is a general indicator. The defect arrival mode can provide more process information, even if the overall defect rate is the same, however, the quality difference may be large because the defect arrival mode is different. The more defects the earlier they arrive, the better the quality of the test process. From the perspective of Test Progress and customer rediscoveries, defect process tracking is very important, A large number of serious defects in the development cycle may prevent testing progress, and will directly affect the quality and performance of software products.
Compared with the product release time and the defect level of the previous version, project managers or development experiences often ask:

• When does a defect reach its peak? What is the peak value sometimes?
• How long does it take to reach a low and stable level after reaching the peak?
• How long does the low and stable level last? Can the current version be released?

Answering these questions is exactly the goal of the defect fulfillment model. Qualitative analysis is easier. The more mature the test team is, the sooner the peak value reaches. Sometimes the peak value can be reached on the first weekend or the second week. The value of this peak value depends on the code quality, the design quality of the test case, and the testing execution strategy and level. In most cases, it can be pushed based on the baseline (or historical data. It takes a long time to reach a low and stable level from a peak, at least 4-5 times the time used to reach the peak. This time depends on the peak value, defect removal efficiency, and so on.

2. PTR cumulative Model
The purpose of the test is to detect Software defects as early as possible. Through the test case, the Software defects can be detected more effectively and faster, and the software defects are identified through PTR (problem tracking report, problem tracking report). Therefore, the number of PTR represents the quality of the software to a certain extent. Each defect/PTR has a life cycle from where the tester discovers the problem and forms a report (called PTR appearance, also known as defect arrival ), developers/designers need to reproduce, correct this PTR/defect, and build and submit a new build containing the corrected PTR/defect to the test group, the corrected issue is verified until it passes the test (called PTR close), the number of PTR hold at a specific time during the test (the difference between all newly discovered PTR and closed PTR) -- PTR accumulation/backlog value. The PTR appearance/accumulation model is based on the two types of data in the problem tracking report-the PTR appearance value in a certain time unit and the PTR accumulation value at a certain time to measure the defect change process found in the test, that is, the process of changing the software product quality status.

3. Depth, quality, and effectiveness of Test Cases
Test cases are the foundation of test execution. The quality of test cases is directly related to the quality of test, which affects the process of ensuring software quality. The measurement of a test case includes the depth, quality, and effectiveness of the test case, and the measurement of the degree of automation, that is, the proportion of the test cases has been automated.
The test case depth measurement can be expressed as the number of test cases per kloc or the number of test cases per function point/object point, the efficiency of test cases can be measured by the number of defects found in each 100 or 1000 test cases. Different test stages are different and different versions of the same stage should be compared, it is not appropriate to compare different stages of the same version. The quality of test cases (TCQ, Test Case Quality) can be measured by the number of defects found in test cases, that is

TCQ = number of defects found in test cases/total number of defects

Some defects can be detected through ad-hoc testing (random and free testing) and work-through testing) and fire-drill testing (similar to fire-training user stress/acceptance testing) and other means to detect defects

4. test execution efficiency and quality
The quality of test execution can generally be measured by the ratio of Software defects left after software release to the total number of defects, generally less than 0.5%, or by seed formula or cross-test. Test execution efficiency can be measured using the following methods:

• Number of test cases per person per day
• Number of Defects Detected by each person per day
• Number of test cases run for each modified kloc

5. Quality of defect reports
One of the ways in which the quality of defect reporting can be evaluated by testers, such as measurable indicators:

• The effectiveness of the defect report, the ratio of All corrected/closed (high-grade) defects to all (high-grade) Defects reported by the tester. The closer this value is to 1, the higher the effectiveness, if a high-grade defect is examined, its normal value is approximately 0.92-0.96.
• The quality of the defect report can be measured by the number of defects with intermediate states being "supplemental information required" and "not a defect". Generally, it is normal to count between 3% and 5% of the total number of defects, higher than or below this value may be abnormal, higher than 5%, may indicate that the quality of the defect report is low; lower than 3%, may indicate that the tester is not skeptical.

6. Test Coverage Evaluation Based on Requirements
The requirement-based test coverage evaluation relies on the verification and analysis of executed/running test cases. Therefore, the requirement-based test coverage evaluation is transformed into the evaluation test case coverage rate: the purpose of the test is to ensure that 100% of the test cases are successfully executed. Generally, the test plan defines the test workload, number of test cases, and test case coverage rate (98%-100%). Based on the pre-determined test schedule, the test plan value can be made into a curve, and then the actual value curve can be drawn on a regular basis (daily or weekly) based on the actual execution results, so as to monitor and predict the entire test process.
During the test activity, the coverage rate of test cases can be estimated in two types:

• Determine the test case coverage rate that has been executed, that is, the number of test cases that have been executed in all test cases. Assume that the number of test processes or test cases executed by Tx is the total number of test requirements:
• Tested overwrite = Tx/rft
• Confirm successful test coverage, that is, there is no failed test during execution, if there is no defect or unexpected result of the test, assume that ts is the number of fully successful and defect-free testing processes or test cases that have been executed.
• Successful test coverage = TS/rft

7. Code-based test coverage evaluation

Code-based test coverage evaluation analyzes the coverage of tested program code statements, paths, or conditions. If the application is code-based, the test policy indicates the number of source code that has been executed in the test. This test coverage policy type is very important for security-oriented systems.
Evaluate the code coverage rate. Determine the expected and total number of test code lines of the test target, the number of actual code lines executed in the test and their percentages, and record the results in the test evaluation report. The amount of code that has been executed during the test is relative to the amount of remaining code to be executed. Code coverage can be based on the control flow (statement, branch, or path) or data flow. The purpose of control flow override is to test the code lines, Branch Conditions, paths in the Code, or other elements of the software control flow. The purpose of data stream coverage is to test whether the data status is valid through software operations. For example, whether data elements have been defined before use.
Code-based test coverage is calculated using the following formula:
Tested overwrite = tc/TNC
TC indicates the number of executed projects represented by code statements, condition branches, code paths, data status vertices, or data element names. TNC (total number of items in the Code) is the total number of projects in the code.

To predict the future, please read the next decomposition: 29th back to software quality measurement

Link: http://www.csdn.net/community2006/vote/index.rails? Id = 1
Copyright, software testing

-For the series of discussions, see:Software testing romance-intermediate and advanced series (sequence)