Thoughts on missed bug tests [reprinted]

1. concept description

The so-called missed test means that the defects of software products are notTestIt is found in the process, but after the version is released, the customer service or user has defects during use. If the software product encounters problems during the customer's use, the consequences are extremely serious.

As we all know, the sooner a defect is discovered, the less costly it will be to discover and solve it. If a defect is found during testing, the cost will be much lower. Testing is one of the most important means to ensure the quality of software. Therefore, it is very meaningful to conduct missed testing and analysis, prevent missed tests, and prompt defects to be discovered as early as possible during the development process, it helps reduce software product costs and improve software product quality.

2. Cause Analysis

No one dared to pack the tickets and said that the items they tested by hand were okay, including the old bird, which would more or less cause defects to slide away from their own hands, no one can think about all the functional operations and application scenarios of the software, but I don't know the old birds like God. After all, "Ginger is still old and hot" is not a plug-in.

The main reasons for missing tests are as follows:

(1) The requirement specification is unclear, resulting inTest CasesRough writing

The requirement is still in a refined process, and the software product is already in the development process. The test case is written based on the requirement specification, and the specification is described in more detail, the more accurate the test cases can be written, the fewer defects may be missed. Therefore, in the initial stage, the test cases can only be rough and can only be further refined by specifications, supplemented test cases, in practiceWorkThe test cases are improved based on detailed specifications. Of course, if the test case is written too carefully, the maintenance work and cost in the later stage will be huge, and it is impossible for us to write the test case too detailed.

(2) Requirement Specification changes, and test cases are not updated in a timely manner

Once the specifications have changed, We need to update the test cases in a timely manner to avoid inconsistent test cases and specifications. However, in actual work, we have not developed the habit of updating test cases with specification changes. First, few of us are testing a software and a function module, during this period of time, the functional module may be tested, and a new functional module will be changed in the next phase, resulting in no energy or effort to update the test cases. Second, you cannot familiarize yourself with the style of the author who wrote the test case in a short period of time. You can add, delete, and fix the case at will, which may lead to other problems. Third, I don't know who will take the lead in updating test cases and how to update test cases.

(3) test cases are not fully covered and omissions occur in scenarios

Although testing is the most important and final level of software quality assurance, we also need to do our best to eliminate bugs before release, which is our responsibility. However, we cannot cite all the scenarios where the software is used at the customer's site. We can only perform tests in some major scenarios and conduct appropriate divergence tests during the testing process, discover bugs to the maximum extent within a limited period of time.

(4) the test case is not strictly followed during the test.

Executing tests based on test cases can avoid missing test points as much as possible. However, some of our students do not strictly follow the test cases to perform the test, so some bugs are missing. However, in other words, many of the bugs we found are not found by test case execution, but are randomly detected during the test process. These steps are not reflected in the test case, this is back to the description of cause (3). Our test cases cannot cover all scenarios.

(5) The test task is tight, leaving less time for the test. As a result, the test of the function is omitted during the test.

I think everyone in the company has experienced a shortage of test tasks. Due to project shortage, the software deadline cannot be postponed, and the development process may be due to various reasons, it takes a lot of test time and the test time is severely compressed. As a result, the original test plan has to be adjusted and some functional points cannot be tested.

(6) The tester has a private bug.

Private bug hiding: this is a private case where we do not report defects. No matter whether testers are out of the way or do not make bugs for developers, we only communicate with developers in private and hope that the other party can fix the defects; or because the release version is imminent, but some defects are exposed one by one, the tester has the idea of being burdened and does not report some defects.

If developers are responsible, they will soon fix the defects, but they will not be able to finish their work. After a while, they may forget the defects you have described, the terrible thing is that, in the busy work, you accidentally forgot the problem, and the defects were packaged to the customer's site in the case of knowledge.

(7) Restricted testing environment, leading to missing Defects

The customer's actual use environment may be complex and changeable. We can simulate and restore the customer's actual environment as much as possible, but after all, it is simulated and restored, rather than the real environment, due to environment differences, many unexpected problems may occur. These problems may only be exposed in the feature environment and specific operation steps, and cannot be restored in our test environment, the problem can only be solved based on the actual environment of the customer's site.

(8) new bugs introduced by developers

Some developers solve the problem only by describing the steps in the bug you submitted, and do not check all possible points involved in the problem. It may solve the problem, A new problem is introduced.

Second, there are a few developers who are not flattering in fixing bugs. I don't know if it is because they have used their minds to modify such a mentally retarded bug.

Furthermore, a developer who is unfamiliar with function modules can fix bugs. Due to unfamiliar business and Incomplete consideration, new bugs may be introduced unconsciously.

3. Corresponding Measures

How can we make up for the problem of missing defects, learn lessons, and take corresponding measures? Here, I will share some of my ideas for your reference for some of the above reasons:

(1) The requirement specification is not clear, leading to the rough coding of Test Cases

When the requirements and specifications are not clear, we cannot compile more detailed and accurate test cases. We can only write a rough framework first, so that the requirements and specifications are gradually clarified, further refine the test cases. In the test process, if the specification is not clear, we need to communicate with the requirement analysis to determine some of our doubts and complete the test. If the specifications are not defined, we can write test cases based on the communication results. After the specifications are clear, add, delete, and fix the test cases.

(2) Requirement Specification changes, and test cases are not updated in a timely manner

The requirement specification changes, causing the original test case to be inconsistent with the current specification. During the execution of test cases, if the test cases do not match the specifications, we need to record them and complete the test cases according to the new specifications, if you have any questions, you need to communicate with and confirm the specification design. You can define the Requirement Specification clearly and write new and modified test cases later, send to colleagues in the Group for review and update the reviewed use cases to the use case library.

(3) test cases are not fully covered and omissions occur in scenarios

It is inevitable that defects are missing due to the design of the test case scenario. It is impossible for the colleagues who compile the test case to think about all the scenarios comprehensively, writing Test cases in all scenarios is not realistic. The defect of external feedback is caused by incomplete Scenario Design. First, we analyze whether the problematic scenario is a required scenario or an accidental scenario. If this scenario is a customer's operation habit, we can communicate with the technical contact person to confirm some specific details of the scenario. In the process of improving the test case, we also need to consider some scenarios associated with the scenario, improve and review test cases in multiple scenarios and add them to the use case library.

(4) the test case is not strictly followed during the test.

We need to face the reality that the test cases cannot cover all the use cases. However, the test cases are compiled by experienced persons according to their specifications and analyzed, developed, tested, andOthersReview by relevant personnel to maximize the accuracy and comprehensiveness of use cases. Test cases may not necessarily cover all scenarios and functional points, but strictly follow the test cases to ensure the maximum quality of our software and avoid defects. In a word, we must strictly follow the test cases during the test process. Do not discard the test cases and perform random tests because of the complexity of the test cases. If the test is performed randomly during the testing process, leading to omissions, it is really not necessary.

(5) Insufficient test tasks, leaving less time for testing

To Avoid Omission of obvious defects when the test task is obviously tight, we can take some measures to ensure the omission of defects to the greatest extent:

First, test priorities are divided according to functional modules. The main functional modules have the highest priority. experienced persons are arranged for testing and new users are arranged to test some unimportant functional modules or rarely used functional modules, during the subsequent tests, experienced students will perform smoke tests on the modules tested by new students to check whether there are obvious bugs;

Second, overtime is adopted. It is estimated that no one here is disgusted with forced overtime. However, if the test time is insufficient, overtime is essential, still calm down to work overtime honestly, at least to show a good performance in front of the leadership;

Third, try to avoid wasting your time in case of unclear development. If the developer takes a long time to troubleshoot the problem, tell the test owner, the Test director should take appropriate measures to avoid the spread of similar problems through negotiation;

Fourth, add testing personnel;

(6) The tester has a private bug.

Once the problem is found, submit the bug library after determining it is the problem. You cannot show your mercy when you are in a private relationship with the development. We need to distinguish between the working relationship and the private relationship, do not affect your work because of your private relationship, and the number of bugs is also one of the manifestations of tester performance.

The release of the version is coming soon, but the defects are not effectively controlled. This isProject ManagementThe biggest thing. For testers, no matter when any defect is found, they must report it and submit it immediately, the module in charge of the test finds multiple defects and is afraid of it. Therefore, it is better to steal the bugs and cause them to slide out, discovery at home is much lower than that at the customer's site. If a fault is detected late and no report is reported, the tester must immediately report the fault, modify the fault, and suspend the fault, the project manager will make a global consideration. After all, most of the students have not yet reached the risk assessment level for defects. Reporting defects makes it better for project managers to think about private bugs than they are discovered by customers. Private bugs always make them unable to eat for a day.

(7) Restricted testing environment, leading to missing Defects

The combination of environments is infinite. We don't have enough time and cost to test the software in enough environments. We need to ensure thatOperating SystemIn the environment and main network environment, the functions and performance of our software can meet our expectation. For the operating system environment, we need to sort the current proportions. Here, we take a software product line as an example: for client software, Win XP and win 7 are the most used. In addition to win 7 and Win XP, we do not know whether the customer will use our software on win Vista and Win 2000, and all of them need to test win Vista and Win 2000.

For the network environment, we generally test in the Ethernet environment. For the customer's other network environments, we also need to perform regular tests when conditions and needs are met.

(8) new bugs introduced by developers

Will developers introduce new bugs because of fixing a bug? Developers generally say that there is no problem. What we need to do is to confirm and verify the bugs fixed by the developers and traverse the related functional points as much as possible, because if there is a new problem, the function associated with the Fixed bug is more likely to be submitted only after the current bug is modified and tested by the developer, however, the function points associated with it may not be tested.

It would be a compliment to a few developers to fix bugs, or because a developer who is not familiar with the module business fixes bugs, we have to spend more time, cover all functional points of regression testing as much as possible to avoid defects and omissions.

When a developer fixes known bugs and introduces new bugs, It is the scope of the Code process submitted by the development team, suchHuaweiThe code improvement is very rigorous, and the possibility of introducing new bugs is relatively small. However, for small and medium-sized companies, the code management is chaotic, and even the team leader does not review the code. The problem is inevitable. Therefore, standardizing the code submission process is a top priority.

4. Advantages of missed Test Analysis

No matter what causes the defect to flow to the customer's site, the problem has occurred. The first thing we need to do is to make up for the impact of the defect. The project team should evaluate the risks, losses, and correct the defects, provide a complete version for customers. After completing the previous work, we can, or even need to consciously think about and summarize, learn lessons, and add and complete the problems to the test cases, for some common situations, you also need to groupLearningTo avoid making the same mistake again in future work.

If we can do a better job, we can learn and make statistics to classify the missing bugs, the severity of the defects, the functional modules, and the causes of omission. During the classification of defects for missed tests, a dedicated person can organize the discussion, organize representatives of relevant departments in requirements, development, testing, technical support and all other product lifecycles to analyze and discuss recent missed tests, in particular, the technical support staff can provide a lot of detailed information about missed test defects, which is very helpful for missed test classification, accumulated experience and lessons learned.

The purpose of defect missing test analysis is to promote the continuous improvement of software quality and the development and testing process, so that we can consider the test process more comprehensively and make up for the deadlock in thinking. Specifically, it is to analyze the defects of missed tests during the testing process and take some corresponding preventive measures to avoid similar missed tests in the future. Continuous improvement in the testing process will improve the testing environment performance and test execution efficiency, reduce the number of defects left on the user and the defect resolution cost, and thus improve the software quality.

In actual work, the missed test and analysis process should focus on the common and serious problems to solve high costs. Specifically, the objective of the missed test analysis is:

● Classification of missed tests for further in-depth analysis

● Collect statistics on classified data

● Identify and change the entire process based on statistical analysis

● Identify and change some parts of the process based on the analysis of some special missed test items

● Use metric data to describe the effects of process changes

● How to conduct missed Test Analysis

5. Summary

Defect missed tests cannot be eliminated. After the missed tests take place, we need to learn to think about and learn lessons to minimize the measurement of defects.