Bug Analysis: lays the foundation for Bug prevention

Http://blog.csdn.net/KerryZhu/archive/2006/11/09/1375341.aspx

1. Introduction:

Companies that produce software have arranged for many people to test their software products. The purpose of the test is to discover bugs (defects, defect) to correct them. Normally, you can handle possible bugs as soon as possible to reduce the bug fixing cost. As we all know, the earlier a bug is detected and corrected, the less resources it consumes.

The problem is that, in many cases, the testing process has to be paused due to fixing bugs found.

So, isn't it better to promote the Organization's long-term quality goals with the same resources that are currently busy with software product testing? To achieve this, we should discover possible bugs in advance as soon as possible. As Philip Crosby said a few years ago, we should try to prevent bugs, not just fix them. This is the true quality.

2. Goal: To prevent bugs

Importance of prevention

As we know, Bugs should be discovered early in the development process. The cost of fixing bugs of products in the development stage is far lower than that of fixing bugs of products in the QC (quality control, quality control) stage, the relative cost of fixing bugs that have been released to customers is negligible. The reason is that when you fix a bug, it is equivalent to redo what you did before. Therefore, the later you fix a bug, the more you redo. If the bug fix is before the product test, only code implementation is required for redo. If the bug fix is in the test phase, the redo includes code implementation and testing. Another cause of increased costs is the dependent components and processes. As the project progresses, the components and processes that the product depends on will also increase.

Next, we will discuss this issue from another perspective. If the bug is detected and corrected earlier, the less the development cost,AvoidBugDoes the introduction cost less?If the bug can be completely prevented, there will be no repeated work during the development process. This is a highly recommended opinion by clusby and has been proven in many cases. However, not all organizations that produce software products try to avoid bugs. They spent most of their energy discovering and correcting bugs before the product was released to the customer. In some cases, software companies do not try to achieve this goal. After the product is released, enterprises can quickly fix bugs in the product to handle customers' complaints. This is because such enterprises are always in the "problem-solving mode", they do not try to find the root cause of the problem, but simply put out a local fire.

This mode not only leads to a direct increase in costs due to repetitive work, but also brings a long-term effect, which will affect the business of enterprises. First, releasing products with bugs will have an impact on the company's reputation and may have an impact on potential customers-they may decide whether to establish partnerships. In addition, because enterprises need resources to continuously solve problems in existing products, the resources for developing new products will inevitably decrease.

For many people, software products with zero defects seem impractical. We always hear software developers say, "software always has bugs ". It is not surprising that the product entered the QC stage because we "Expect" developers to create bugs. Unfortunately, releasing a product that contains many bugs is still surprising to customers. Even customers are not surprised.

In fact, every software enterprise can use some simple methods to prevent bugs without adding any additional resources. Bug prevention lies in a simple principle: the best method is to properly learn from our own experience.

 

Today's discovery is tomorrow's Prevention

To prevent bugs, we must first understand the sources of bugs. Software bugs can be divided into several categories (which may overlap with each other ). The first type of bugs may be random, usually due to temporary negligence. Although these bugs may be hard to prevent due to their randomness, appropriate analysis will help avoid these bugs.

Another type of bug comes from misunderstanding of requirements, errors in the development environment, or a lack of related technologies to solve the problem. A common feature of such bugs is that they all come from developers. Unless discovered, these bugs will always exist. For example, a development engineer who does not fully understand the requirements may not be able to find these problems during the unit test phase, only when the product is in another organization (such as the QC Group) during the test, the product implementation and requirements are inconsistent. This makes it more important to avoid similar problems in the early stage.

The good news is that software bugs tend to appear repeatedly, even a Random bug. The continuous emergence of software bugs is not only manifested in the work of the same developer, but also on the level of a project or even an enterprise. This does not mean that every developer in the company will make the same mistake. However, at least some of these errors are enough to become frequent problems. So why do we think repeated errors are good news? Because foreseeable bugs are easier to prevent. The fact is that we can find some common problems and take appropriate measures to prevent them (or at least reduce the number of times similar errors occur ).

A subset of human bugs? These bugs are more likely to be prevented. Domain bug? Domain bugs are closely related to product problem domains or solution domains. Such bugs have a greater opportunity to reproduce, because developers, project teams, and even enterprises are constantly working in this field.

The problem is how to prevent various bugs. Based on the purpose of this discussion, I suggest you set a more practical goal. Instead of completely preventing a bug, let me set the goal to prevent bugs that we already know are likely to produce. This means that we can promote future bug prevention through various bug discovery activities. When a bug is detected, we have a good opportunity to prevent similar problems.

Premise: Record bugs

The prerequisite is to keep track of detected bugs and correctly record them. If you leave this prerequisite, you cannot use bug discovery as a tool to prevent bugs. Whether you use the Bug Tracking System or write only one report to summarize the test results, it is important to save the data for later analysis.

When analyzing bugs, it is worth noting the bug record issues. The wider the bug definition, the more useful the data for Bug Analysis. Testers who report bugs should understand this point, not limited to bugs in a narrow sense. If possible, testers should use their experience to make initial assumptions about the causes of bugs. We will discuss this issue later in explaining the analysis process.

As important as logging bugs, it is the first step in bug analysis. Tracking bugs in the past alone cannot prevent the occurrence of bugs, because many different bugs may come from the same core issue. Different from information collection, proper analysis of the causes of bugs is very useful for Bug prevention.

3. Defect Analysis

Target

In the previous section, we explained the reasons for Bug Analysis. As mentioned above, the ultimate goal is to prevent bugs

Instead of correcting them. However, we can define an important sub-goal, which will continuously improve the skills and practical experience of the entire development team (including the QC Group. Of course, these two goals are closely related. Without continuous knowledge accumulation, bug prevention cannot be implemented. However, I think it is necessary to mention this subgoal to emphasize the process of continuity. Bug prevention is not an unrealistic goal. However, you cannot expect it to happen overnight. You should provide development teams with education and knowledge so that they can gradually improve their work.

Policy

The focus of this discussion-bug prevention strategies are very simple and easy to implement. The secret is to use the existing process elements in most development environments. We will not introduce any new expensive activities or introduce some new roles into the development process.

Our strategy is to discover bugs, find out the root cause of the bugs, and then find a method to fix similar bugs in the future. Because the QC process has been used to discover bugs in the current product, most of the work of this policy has actually been executed. What is missing in most development processes is the analysis of bugs found in the QC process. As you will see, although this part of the Strategy does not need to be expensive, it brings great extra value.

 

Analysis Process

(1) BugDISCOVERY AND PRELIMINARY ANALYSIS

As mentioned above, the first step in bug analysis is to discover bugs. However, QC engineers who have detected bugs (Note: test engineers) should not be satisfied with recording bugs on the surface. An important role of QC engineers is to find the root cause of bugs. When testing the product quality, the QC team should not regard the product as a black box. Instead, they should understand the product as well as the developer, including in-depth source code and product design and implementation. These capabilities are the basic requirements for the QC team to start bug analysis. Once familiar with the product code, QC engineers may deduce the root cause of the bug.

I want to emphasize the essence of the phrase below: What is the root cause of the bug? The root cause of the bug is not the source code that generates the bug, although the information may be closely related to the analysis process. However, discovering the root cause of a bug means finding the cause of these errors. The problem may be clearer through some examples.

Let's look at a common thread synchronization problem. Assume that a multi-threaded application needs to access a Data Structure synchronously. The QC engineer assigned to test the product found that, in some scenarios, the application stops responding even though there is no crash. The normal QC process is that this bug is recorded in the bug tracking system and describes the test scenario and the actual results of stopping the response. However, if the QA Engineer is familiar with the source code, preliminary analysis of the cause of the bug can be conducted. For example, the QC engineer may conclude that the cause of this bug is that the previous thread did not release mutex, resulting in a conflict. These analyses can be recorded in detailed descriptions of bugs as a basis for Bug Analysis.

(2) BugRevision and further analysis

As always, after discovering a bug, developers should handle it. However, if the bug discovery process involves a preliminary analysis of the root cause of the Bug, developers may have more information about how to solve the bug. Although this is not the purpose of the preliminary analysis of QC engineers bug, it may provide developers with more ideas.

In addition to fixing defects and recording implementation steps, developers should further analyze bugs. This analysis should focus on the development scenarios that cause bugs.

In the example of thread synchronization, developers should not simply add a call to release mutex (Note: Mutal exclusion = mutex lock, this ensures that shared data is not accessed by multiple threads at the same time, and only one thread is granted with exclusive access to shared resources ). On the contrary, developers should find out the reason for not releasing mutex. Assume that the cause of the analysis is: Because the method to be synchronized exceeds one return point, the developer forgets to clean up the code on some control paths.

This type of simple analysis brings great value. Different from the specific solution to record specific problems, we now have experience that can solve many situations. In some cases, it does not even involve thread synchronization and mutex release. However, the analysis process is not over. We need further analysis to convert all the collected data into practice, so as to help avoid similar bugs in the future.

(3) BugPreventive Analysis

The last step in the analysis is to find a way to prevent similar errors. This method involves not only development and QC engineers, but also senior developers who are not directly responsible for coding.

The results of this phase are some useful practical experience that developers can use to prevent bugs rather than correct them. These practices should not be a solution to a specific problem. In our thread synchronization example, we may obtain the following practice: Is there an audit scope mechanism to acquire and release resources? This practice (not necessarily suitable for all programming languages) can instruct developers to encapsulate resources with a class, so that constructor functions are easily allocated and destructor) function to release resources. If such an agreement is followed, resources (mutex obtained in the preceding example) are always released regardless of the control path when the program ends this method.

Bug prevention analysis is the core of the entire bug analysis process. The practice summarized at this stage can prevent potential defects in a wider scope. Due to the wide applicability of the analysis results, the investment in analyzing a specific problem will be easily withdrawn.

The preceding example shows a Random bug. The developer forgets to release resources due to negligence. During code implementation, such bugs are randomly generated, but the probability of producing similar bugs is very high. Therefore, although such bugs are random, they can still be foreseen and prevented.

(4)Release experience

The analyzed practical experience should be recorded and published, so that other developers can learn this experience to avoid similar errors. The best way to publish experiences is the knowledge base. This will allow new knowledge to flow within the Organization and be learned by relevant developers.

If the analysis results are not delivered to other persons in the organization, the analysis will not be achieved. The only way to avoid the next bug is to let developers know how to avoid it and encourage them to do so.

BugAnalyze instances

Let's look at another example to better understand the benefits of bug analysis. In this case, QC engineers perform the following operations: crash occurs when a long string is entered into the application ). This conclusion requires a certain degree of analysis, but this QC engineer is not satisfied with this analysis and further studies the relevant code, the cause of crash is that there is a problem with the processing of the input string. One step is to cache the entered characters in a fixed-size array, which sometimes seems too small.

Like the example of thread synchronization, the initial analysis by QC Engineers brings great value, so that developers can easily discover and correct this bug. In addition, recording the true cause of the defect rather than the appearance will help others avoid similar bugs.

Then, the developer began to fix the bug. During the correction, she not only recorded the solution, but also explained the cause of the defect. In this example, the cause of the bug is that when operating the unprocessed C/C ++ buffer, it is not often checked whether the buffer size is insufficient. However, this conclusion can even be further summarized as more widely used experience to help developers avoid similar defects in the future. Therefore, with the help of more experienced developers in the group at the final stage of analysis, developers have gained the following practical experience: Avoid using unprocessed C/C ++ buffers, use secure collections and strings whenever possible, such as available collections and strings in the standard template database. This completely avoids the previous bug.

 

Benefits

Bug Analysis brings many benefits. The first advantage is to help developers who generate errors sum up experience and prevent similar errors in the future. Sometimes, correcting a specific bug without analyzing the cause does not help to improve it in the future. In this case, developers can grow and improve their capabilities only through in-depth analysis and guidance from senior developers.

The wider advantage is that other developers can learn from their colleagues' mistakes. The practical experience summarized by analysis can prevent the generation of bugs. Such knowledge is shared among members of the organization. Bugs produced by a developer can help others in the Organization avoid similar bugs.

From a more general perspective, the release of best practices (such as bug analysis summary practices) promotes the learning and self-improvement of members in the Organization. In this way, the value of bug analysis is not only the prevention of defects.

Another benefit is that by recording bugs from a wider perspective, other QC engineers in the Organization will know how to identify similar errors. In addition to sharing testing knowledge and experience within the Organization, the bug analysis process can facilitate the development of better testing techniques and tools to help you discover similar bugs. Therefore, even if the defects are not completely prevented, they can be easily discovered.

As a result of all the above benefits, QC will have more time in a round of testing to test more complex scenarios and find more "Tricky" bugs. If similar bugs have been prevented and are not easy to generate, and QC has better technologies to find similar bugs, more time will be available for more advanced testing. Of course, the quality of the products produced by the Organization will also be improved.

Finally, I want to emphasize that bug analysis not only collects execution problems, but also summarizes practical experience from these problems. For example, the cause of a bug may be that the requirement is not clear enough. In this way, the experience gained from bug analysis provides a way to prevent unclear requirements. This experience may not work for developers in the Organization. Therefore, although QC engineers began to verify the implementation results of developers, they still need to improve the development process, such as requirement collection and design process.

4. Summary

The real quality is to produce products without bugs. Any other Goals enable Members in the organization to accept Software defects as part of a normal workflow. Therefore, the first step is to prevent the same bug from happening again. We can easily implement this goal. We can improve the practical experience of the entire Organization through a bug generated by a developer.

Through in-depth product analysis, we can understand the mechanism of this bug: Why? How to prevent it? How can we find it more easily next time? As long as we spend a little time understanding our bugs, rather than just fixing them as soon as possible, we can gain experience from them. In this way, the time wasted by a defect can also be converted into investment: ensure that similar errors will never happen again.