Defect Leakage Test Analysis: Testing process Improvement

First, the definition of leakage test

The so-called leakage test, refers to the SOFTWARE PRODUCT defects have not been detected by the test group and left out to the user, but eventually discovered by the user. If the product has a problem with the user, the consequences are very serious. In the software development process, the sooner the defect is discovered, the smaller the cost of discovering and resolving the defect. If the flaw is found in test group tests rather than being used by the user, the cost will be much smaller. If the defect was discovered by the development team during the development process, it would cost less. Therefore, it is very meaningful to carry out leakage analysis, prevent leakage test, and make the defect be discovered in the early development process, which is helpful to reduce the cost of software products and improve the quality of software products.

Second, the purpose of leakage test analysis

The purpose of the leakage analysis is to promote the continuous improvement of software quality and development test process. Specifically, it is through the analysis of the development and testing process of missing defects, to develop appropriate preventive measures to avoid the recurrence of similar leakage test in the future. Continuous improvement of the test process will improve the effectiveness of the test environment and the efficiency of test execution, reduce the number of defects left to the user and defect resolution costs, thereby improving software quality, reputation and sales. In the SOFTWARE product development process to pay attention to the leakage test analysis and participate in the leakage test analysis of the team, the more the effect of leakage analysis is better. If the development and testing team attach importance to the leakage analysis, and closely cooperate with the leakage analysis work, leakage analysis will achieve very good results.

In the actual work, the leakage analysis process should focus on those common, serious and cost-effective problems. Specifically, the objectives of the leak-test analysis are:

Classification of missed measurements to facilitate further in-depth analysis

Statistics of classified data

Identify and change the whole process on the basis of statistical analysis

Based on the analysis of some special missing items, some parts of the process are identified and changed.

Using metric data to illustrate the effects of process changes

Third, how to carry out the leakage test analysis

The leakage analysis activities can be carried out according to the following recommendations. After familiar with the leakage analysis process, it is necessary to determine the frequency of the leakage analysis activities. In order to achieve better results, it is best to follow a timetable for periodic leakage analysis activities, one months is a more appropriate frequency.

Plan

This process is set up for a multi-project group joint leak-test analysis, which is most effective when implemented in a joint project team. If it is not possible to set up a joint project team for leak analysis, you can also modify the process to be implemented only within the test group.

The plan will be difficult to implement effectively if there is no definite point of concern. In order to achieve the desired results, it is necessary to plan the exact number of people and the time of activity of the leakage analysis activities. The effect of the process execution depends entirely on how it is executed, and if you do not make a good plan, your process will not get much improvement.

When the leakage analysis activity is actually carried out, only a subset of the missing-test classification is selected for analysis, which will be helpful for the more effective leak-test analysis. Before you can triage, you need to determine which subset of subsets to select in your plan. For example, if the severity level of the leak is divided into one to four levels, the first severity is the highest, and the four severity is the lowest, then perhaps only the analysis of the secondary leakage test is the most appropriate, so as to avoid the user is not insignificant leakage test too much work, or can only analyze those closed and repaired leak detection defects, Because if you analyze flaws that have not been shut down and repaired, you may miss out on critical information, and you can also eliminate duplicate defects and defects caused by user error operations before you perform a leak analysis, so that you only need to analyze those effective leak detection defects, They can really provide information that needs to be improved in the development and testing process.

Classification of missed test

The next step is to classify all the missing defects according to a meaningful attribute, of course, provided there is a missing list that contains all the details of the missing flaw. You then need to determine which attribute classifications are useful for analysis. Here are some examples of the properties that are missing from the flaw test:

Leakage of yield monitor life (most likely to find the missing defect activity)

Development phase (original requirement review, concept review, design review, Code view, unit test , module linking, information development)

Test phase ( functional test , system test , local language test, device driver test, installation test, performance test , anomaly test)

Product Modules (code modules that generate leak detection defects)

Modules A, B, C, etc.

Impact of defects (impact on user use)

System crashes, business aborts, data integrity, command invalidation, installation failure, device/drive, performance, documentation, availability, etc.

Introduction Version (Code version with leak-tested defects)

Platform

Severity level

Found the missing version of the test

The leak detection activity is that in the software development testing process, a certain category of missing defects should be found in the activity. The purpose of this classification is to facilitate further refinement of the activities that have resulted in the leakage test.

The product module refers to the code module where the defect is missing.

Defect impact refers to the type of problem that can be caused by the leakage of defects to the user.

The introduction version is the version of the code that was introduced when the flaw was leaked, and it should be the first time the code introduced the bug.

Platform refers to the platform or operating system that produces the missed test.

Severity is a measure of the severity of a defect, such as: fatal, severe, general, hint. If your defect tracking process does not contain this attribute of the defect, then the leak analysis process should explicitly give the severity level of each defect.

The missing version is found to be the version of the software that was first discovered and reported at the time of the leak test.

In the case of missed classification activities, it is best to bring together representatives from the relevant departments in development, testing, technical support and all other product lifecycles to discuss recent leaks, especially if the technical support staff can provide a lot of very detailed information about the missing flaw, which is very helpful for the missed classification.

When the project team does the actual leak analysis activities, it may not be necessary to classify some of the attributes suggested above, and some other classification criteria should be used, it is best to brainstorm in the project group to determine which classifications are most applicable.

After the leakage of the defect classification activity, it is necessary to make statistical analysis of the classification result data. For example, each missing flaw corresponds to a leak-generated activity, which can be considered for further improvement.

　　Analytics activity: Tracking Tools

It is difficult to perform leak analysis without the support of the defect tracking tool. Tools should be used to maintain the missing defect data for all different classifications. The Lotus Notes database is a great tool that makes it easy to split data into a variety of different ways, so you can create a variety of views of the same batch of data, enabling you to perform statistical analysis from every angle.

　　Analysis Activity: Statistics

Statistical analysis is to guide the process improvement of the whole process. Statistical analysis of the first to determine the frequency of statistical analysis, generally a quarterly statistical analysis is more appropriate. For statistical analysis, it is necessary to compare the data one by one of each categorical item of a classification with all other categorical item data for that category, and to do so for all classifications. For those with relatively large number of categories should be further subdivided, and further statistical analysis work.

　Analysis activities: Process improvement throughout the process

In the statistical analysis, the leak analysis team needs to be assembled together to discuss the statistical analysis results. Based on statistical analysis results, various trend graphs can be obtained, and the analysis team can discuss the suggestions and proposals that need to be improved in the whole development process, then make formal improvement recommendations to those areas where improvements are needed, formulate an improvement implementation plan, and discuss the change implementation process at subsequent meetings. Task assignment and tracking can be done through the leak-test analysis database or other tools. Here are two examples of the whole process improvement based on defect analysis: The first example, if the system in the fault processing of the detection of a lot of leakage defects, then the whole process of development process improvement, you can consider increasing the anomaly test group, strengthen the anomaly test; the second example, If a user discovers a large number of defects in the process of using software on a hardware platform and the test group does not have the hardware platform, then it is necessary to consider improving the hardware acquisition process and increasing the hardware platform for testing. The overall process improvement will have a huge impact on the software business, so be sure to get management support and approval.

Analysis activity: Local Process Improvement

In the case of the joint project team for the leakage analysis, each of the activities that produced the leakage test will be elected representatives (such as: development activities, representatives of test activities, document writing representatives, etc.). For example, when classifying the "leak-generated activity" attribute, if a missing flaw is classified as "unit Test", then the leak-test defect should be further analyzed by the development activity representative for the local process. All of these defects are listed in the leak Analysis database, and each representative of the classification activity should list all the missing defects that belong to the activity, and then propose a local improvement plan for those activities. For example, the test activity representative should list all the leak detection defects that were "tested" for "leak test generation activities" and subdivide them, then assign them to the test engineer for analysis, and the test engineer will conduct a detailed analysis of the assigned leakage defects to find out the cause of the missed test, Then propose a targeted improvement plan to prevent the same kind of defects from being missed again. These improvement plans should be implemented after the audit is passed, and the entire improvement process should be tracked in the database, each improvement plan should be able to correspond with the individual flaw leakage analysis results, the test representative should promote the completion, review and implementation of the improvement plan. In particular, these improvement plans are not intended to be used to fix defects, because the analyzed leak defects should have been fixed, and the improvement plans are simply to redefine the test process (or the development process) based on the analysis of the cause of a defect leak, and it is concerned with how to prevent such problems from recurring in the future. , rather than caring whether that particular flaw will appear again in the future (because it has been repaired). For example, a local process improvement plan could be to supplement a use case that was not previously considered, or to add a specific hardware in a test environment to make the test environment closer to the user's environment. Creativity should be encouraged when considering improvement plans.

Measure

The final step of the leak-test analysis process is to measure the effect of phased implementation of the improved process. This is discussed in more detail later in this article.

Example of leak test analysis

Figure-1

* APAR is a term that describes a defect attribute.

Figure 1 is an example of missing a defect database from Lotus Notes for a virtual product. In this example, three label items are used to track the characteristic data for a leak detection defect. The first tab item is used to describe the details of the defect, which comes from the defect tracking tool, which is best done automatically from the Defect tracking tool to the input and conversion of the leak analysis tool. In addition, there is a larger domain used to describe the history and profile of the defect, which is not shown in this figure.

Figure-2

Figure 2 shows the "Development Analysis" tab entry. For the development process, the leakage of defects can be different classification. In this example, the missing flaw is categorized as a "design review process" omission. This example demonstrates the process of creating a "concept review" of product development process changes. The "Capture probability" item is used to assess the possible effects of a change implementation, and it can answer "how likely is this flaw to be captured if the change plan has been implemented in the development process?" , this article will be explored in detail later in the "Measurements" section. The table also designs the expected date items for the change plan and the expected date items for implementing the change plan, which automatically sends a reminder to the change plan's assignee on that date.

Figure-3

Figure 3 shows the "Test Analysis" tab item. Here you can analyze whether a user-reported flaw is really missing, in other words, determine if there is a real flaw in the testing process or whether the flaw is really worth solving. This is very important. Some users find that the environment of the problem is very special or uncommon, and some defects are costly to solve and difficult to detect, and the leak analysis group needs to determine whether it is necessary to improve the testing process for this leak test defect. If the user who discovers the problem is a very important customer, and the user will often use the environment, even if the environment to find the problem is very special, you need to improve the test environment to meet the user's environment as much as possible. In the example above, the flaw is categorized as "the type of test not executed" and the test type has been missed. After a statistical analysis of hundreds of leak-detected defects, it may be necessary to increase the type of test in the entire development process if the "test type not executed" ratio is found to be high. The "catch probability" item here is the same as the one described in the previous section.

Measurement

This section will give some suggestions on measurement. First of all, for the need to improve the point, will be able to guide the leak test Analysis group to develop appropriate improvement plan of the measurement point; next, we will give some methods to evaluate the effect of the missed analysis process. You can use some or all of these suggestions to build your own measurement system.

1. Measurement Drive Improvement

Compare the previous categorical data with the total number to get the proportions of each category. Here are some examples:

Figure 4 shows each code module (module A-Z) the percentage of the total number of missed test. It is clear from the pie chart that more than 50 of the leakage is from the B, C, D, E four modules, and this measurement can help the leak analysis group decide whether to implement improvements in the development process of these four modules.

Figure-4

Figure-5

Figure 5 analyzes the different effects on users caused by leak detection, such as business interruption, system crash, or device-related problems. For example, if "Impact 1" is a device-related problem, the hardware platform on which the software is being tested may need to be improved, and similarly, if the blue part is the type of impact of a high severity rating, then you can see how much the leak test is for the type of impact of the high severity rating.

With the database tools from the previous example, you can also output a large number of other charts, the two examples above are just two of the most commonly used analysis diagrams.

2. Evaluation of leakage test analysis effect

The effect of the evaluation improvement needs to have accurate data and consistent analysis report, the following data will be used:

Tfvud is the number of defects found by the user (total Field Valid Unique Defects), which is a confirmed, non-duplicated, non-user-operated, non-recommended type of defect found by the user;

The PDD is a test of the number of defects found (post development Defects), that is, the number of defects found during the test cycle after development, but does not include those found after posting to the user;

KLOC represents thousands of lines of code.

The following analysis data can be obtained using the above data:

Tfvud

-------------------

Thousand lines change Code & new Code

The above values should be measured throughout the product life cycle and used as an evaluation indicator for comparison at the same time checkpoint for one version and another version. For example, in a quarter, the 2.0 version of the measurement should be lower than the 1.0 version. The purpose of this measurement is to reduce the number of effective issues that users find in the unit code size.

PDD

-------------------

Pdd+tfvud

The above values should also be measured throughout the product life cycle as evaluation indicators for comparison at the same time checkpoint as a version and another version. For example, in a quarter, the 2.0 version of the measurement should be higher than the 1.0 version. The purpose of this measurement is to facilitate early detection of defects in the development process, thereby reducing the cost of repairing defects.

Capture probability

There is a "catch probability" attribute item in the database (explained in detail in the preceding section), which is an estimate of the effect of preventing similar problems from re-leaking after the implementation of the process changes. This estimate is the basis for the expected effect of the plan. By averaging the acquisition probability of each change, we can get the overall expected effect after the process change, so that we can reasonably anticipate the decrease of the number of user problems after the product release.

Figure-6

, the acquisition probability of module B in the development process is 30, and the acquisition probability of the test process is. If the development process in the code generated 100 defects, then according to the capture probability in the development phase may find 35 defects, there are 65 defects may be omitted to the test phase, according to the test process 30 of the capture probability, in the testing phase will likely find 65*30%= 19.5 defects, then the development test phase A total of about 55 defects can be found. This 55 probability is the development of the test process after the change of the comprehensive effect estimate. The above process is represented by an equation (. 35) + (1-.35) (. 30) or d+ (1-d) (t), where D is the acquisition probability of the development process, and T is the probability of the test process capturing. This figure is based on an example of a code module, and other classifications can perform the same evaluation work , as shown in Figure 7 below.

Figure-7

The final step is to estimate the reduction of the number of effective user defects by averaging all the combined capture probabilities. First, select a set of focus-missing groups that are relevant to the expected effect. In this example, assuming that the focus of the leak test group contains 76 leakage defects, if the combined acquisition probability for the 76 defects is 52.5%, then it will be possible to prevent about 40 defects leakage test. Assuming that there will be 250 leaks in a year, the first 52.5% of the capture probability is a relatively accurate data, then will be able to prevent 250 leakage of 16%―― about 40 leak test defects, this is the next version will reduce the number of leakage of the final prediction, and this is the smallest prediction, because we are just the focus of the leak test group Predictions, which may not apply to other types of issues. If we do not make that assumption, then the predicted drop in the number of missed measurements may be unrealistic 52.5%.

Iv. Summary

The main purpose of the defect analysis is to improve product quality and customer satisfaction, and reduce the cost of repairing defects. This is achieved by facilitating the early detection of defects in the software development process as much as possible. Software Test teams that conduct leak analysis activities will help the software development team improve quality, their testing process will be more complete, and the test environment will be more consistent with the user's actual environment. The data collected from the leak analysis process provides sufficient justification for improving activities such as hardware additions to the test environment. In addition, the leak-test analysis process encourages communication and collaboration between the project teams to develop higher quality software products. It can also predict the number of missed defects in the future, evaluate its own effectiveness, to prove that the resources invested is worthwhile.

Defect Leakage Test Analysis: Testing process Improvement