Defect missing Test Analysis: test process improvement

2005.12.19 from: 51 testin

Definition of missed tests

The so-called missed test means that the software product defects are not found by the test group but missed by the user. If the product has problems with the user, the consequences will be very serious. In the software development process, the earlier the defects are discovered, the less costly the defects are to be discovered and solved. If the defect is found in the test group test rather than in use, the cost will be much lower. If defects are discovered by the development team during the development process, the cost will be reduced. Therefore, it is very meaningful to conduct missed test analysis, prevent missed tests, and encourage defects to be discovered as early as possible during the development process. It is conducive to reducing software product costs and improving software product quality.

Objective of missed Test Analysis

The purpose of the missed test analysis is to promote the continuous improvement of software quality and the development and testing process. Specifically, it is to analyze the defects of missed tests during development and testing, and formulate corresponding preventive measures to avoid similar missed tests in the future. Continuous improvement in the testing process will improve the testing environment performance and test execution efficiency, reduce the number of defects left on the user and the defect resolution cost, and thus improve the quality, reputation and sales of the software. The more teams that focus on missed test analysis during software product development and participate in missed test analysis, the better the effect of missed test analysis. If both the development and testing teams attach importance to the missed test analysis and closely cooperate with the team for missed test analysis, the missed test analysis will produce very good results.

In actual work, the missed test and analysis process should focus on the common and serious problems to solve high costs. Specifically, the objective of the missed test analysis is:

• Classify missed tests for further analysis
• Collect statistics on classified data
• Identify and change the entire process based on statistical analysis
• Based on the analysis of some special missed test items, some parts of the process are marked and changed.
• Use metric data to describe the effects of process changes
• How to conduct missed Test Analysis

For the missed test analysis activity, refer to the following suggestions. After you are familiar with the missed test analysis process, you need to determine the frequency of missed test analysis activities. In order to achieve better results, it is best to follow a schedule to conduct regular missed tests and analysis activities. One month is a more appropriate frequency.

• Plan

This process is set for joint missed test analysis by multiple project teams. It is most effective to implement this process in the joint project team. If it is not possible to set up a joint project team for missed test analysis, you can modify the process and only implement it within the test group.

Developing a plan will be difficult to effectively implement if you are not sure about the focus. To achieve the desired results, you need to plan the exact number of persons and time for the missed test and analysis activities. The effect of process execution is entirely dependent on the way it is executed. If you do not plan well, your process will not be improved much.

In actual missed test analysis activities, only a subset of the missed test categories is selected for analysis, which will facilitate more effective missed test analysis. Before classification of missed tests, you need to determine which subset to select in the Plan for analysis. For example, if the Severity Level of the missed test is classified into one to four levels, the severity level is the highest, and the Severity Level 4 is the lowest, then it may be most appropriate to analyze only one or two missed tests, this can avoid spending too much effort on missed tests that are irrelevant to users. You can also analyze closed and repaired missed tests only, because if you analyze defects that are not closed or fixed, some critical information may be missed. In addition, you can also eliminate repeated defects and defects caused by incorrect user operations before the missed test analysis. In this way, you only need to analyze the valid missed test defects, only in this way can they provide information that needs to be improved during development and testing.

Missed Test Category

Next, we need to classify all missed test defects according to meaningful attributes. Of course, the prerequisite is that a missed test list contains detailed information about all missed test defects. Then you need to determine which attribute categories are useful for analysis. The following is an example of the property of missing test defects:

• Missed test activities (activities that are most likely to discover the missed test defects)
• Development stage (original Requirement Review, Concept Review, design review,CodeInspection, unit testing, module joint debugging, and information material development)
• Test phase (function testing, system testing, local language testing, device driver testing, installation testing, performance testing, and exception testing)
• Product module (Code module with missed test defects)
• Modules A, B, and C
• Impact of defects (impact on user use)
• System Crash, service suspension, data integrity, Command failure, installation failure, device/driver, performance, documentation, availability, etc.
• Introduce version (introduce code version with missing test defects)
• Platform
• Severity Level
• Missed versions found

Missed testing activities refer to the activities in which certain types of missed testing defects should be discovered during software development and testing. This category is set to facilitate further detailed analysis of missed tests.

The product module refers to the Code module where the vulnerability is missed.

Defect impact refers to the type of problems caused by missed defects during use.

Introducing a version is the version of the code when the missing test defect is introduced. It should be the first version of the Code to introduce the defect.

A platform is a platform or operating system that produces missed tests.

Severity refers to the measurement of the severity of a defect, such as fatal, severe, General, and prompt. If your defect tracking process does not contain this attribute, the missed analysis process should clearly indicate the severity level of each defect.

The version of the software that is detected and reported for the first time.

During the classification of missed tests, it is best to organize representatives of relevant departments in the development, testing, technical support and all other product lifecycles to discuss the recent missed tests, in particular, the technical support staff can provide a lot of detailed information about missed test defects, which is very helpful for missed test classification.

When the project team conducts the actual missed test and analysis activities, it may not need to be classified according to the suggested attributes, but other classification criteria need to be adopted, at this time, it is best to discuss in the project team to decide which categories are most suitable.

After the Vulnerability classification activity is completed, statistical analysis of the classification result data is required. For example, each missed test defect corresponds to a missed test activity, and further improvement can be considered.

Analysis Activity: tracking tool

If no defect tracking tool is available for missed test analysis, it is very difficult. Tools should be used to maintain missing test defect data of all different categories. The Lotus Notes database is a good tool that can easily split data in different ways, so that you can create various views for the same batch of data, in this way, statistical analysis can be performed from various perspectives.

Analysis Activity: Statistics

Statistical analysis is used to guide the improvement of the entire process. For statistical analysis, you must first determine the frequency of statistical analysis. Generally, it is appropriate to perform a statistical analysis once a quarter. During statistical analysis, you need to compare the data of each category item of a specific category with all other category item data of this category, and perform such operations on all categories. Further subdivided classification items with a relatively large number of categories for further statistical analysis.

Analysis activities: Process Improvement

During statistical analysis, the missed test analysis team should gather together to discuss the statistical analysis results. Various trend charts can be obtained based on the statistical analysis results. The analysis team can discuss the suggestions and solutions for improvement in the whole development process, and then make formal improvement suggestions for those areas requiring improvement, formulate the improvement implementation plan and, at subsequent meetings, the missed test analysis team will discuss the change implementation process. You can use the missed test analysis database or other tools to allocate and track tasks. Here, we can give two examples of improving the entire process based on the Defect Analysis: The first example is that if the system discovers many missed test defects during troubleshooting, when improving the entire development process, you can consider adding an exception Test Group to enhance the exception test. The second example is, if a user finds a large number of defects while using the software on a hardware platform, but the test group does not have the hardware platform, you need to improve the hardware acquisition process and increase the hardware platform for testing. The improvement of the whole process will have a huge impact on software enterprises, so we must obtain support and consent from the management.

Analysis Activity: Local Process Improvement

During the joint project team's missed test analysis, representatives (such as development activity representatives, test activity representatives, and document writing representatives) should be selected for each activity that has missed test ). For example, if a missed test defect is classified as a unit test for the property "missed test activity ", then, the development activity representatives should conduct further local Process Analysis on the vulnerability. All these defects are listed in the missed test analysis database. Representatives of each classification activity should list all missed test defects belonging to the activity, and then propose a local improvement plan for these activities. For example, the test activity representative should list all missed test activities as missed test defects, segment them, and assign them to test engineers for analysis; the test engineer will conduct a detailed analysis on the allocated missed test defects, find out the cause of the missed test, and propose a targeted improvement plan to prevent similar defects from being missed again. These improvement plans should be implemented after approval, and the whole improvement process should be tracked in the database. Each improvement plan should be able to correspond to the analysis results of single defect missing tests, test representatives should promote the completion, review and implementation of improvement plans. It should be emphasized that these improvement plans are not used to repair defects, because these missed test defects have been fixed, these improvement plans only re-determine the testing process (or development process) based on the Cause Analysis of a vulnerability missed test. They care about how to prevent such problems from happening again in the future, instead of worrying about whether the specific defect will appear again in the future (because it has been fixed ). For example, a local process improvement plan can supplement the use cases that were not previously taken into account, or add specific hardware in the test environment so that the test environment is closer to the user's use environment. Creativity should be encouraged when considering improvement plans.

? Measurement

The last step in the missed test and analysis process is to measure the phased implementation effect of the improvement process. This article will be discussed in more detail later.