Java Theory and Practice: balancing test, part 1th: Do not write tests only, but also write bug detectors

For many teams, unit tests are now a major part of the development process; Frameworks such as JUnit can do non-destructive testing, although we do not like it and prefer to write some tests for some code. Unit tests are inefficient and can only test a single piece of code, and, generally, the reusability of test code is often low-the tests that were written for component A yesterday are not well used to test component B (except for sample code).

A typical unit test scenario

What is the first thing you do when you find a bug? You may just want to fix it, but that's not the most efficient way to run for a long time. In many development departments, the process of handling bugs is as follows:

Write test cases for bugs

Ensure that test cases fail to run when a bug is encountered

Fix bugs

Ensure that test cases are passed

Ensure that other test suites are still available

Check for corrective and test cases to form version control

Record the correction in the bug tracking system

Although this approach does a lot more work in the short term than just fixing bugs, but it offers a lot more value: get more confidence in fixing bugs because you've tested them, and get more confidence that bugs won't happen again, because test cases are part of a regression test suite. Between the version control system and the bug tracking system, you can also get a record that describes what the bug is and how to fix it-which is very useful information that others will benefit from.

If you are aggressive, you can think about how bugs appear and find the same error in other locations. If you find the same error elsewhere, you can test and fix the bugs. The main weakness of unit testing as a quality management tool is that each test case can only test one piece of code. Because test cases are designed for each component and for each potential error pattern, it can be time-consuming and costly to write enough unit tests to test a large number of products.

QA economy

Testing is a basic quality management tool, and we know that only a few sets of test cases are not enough to identify all bugs in complex software fragments. In fact, "Find all bugs" is an impossible goal for any good program. It is estimated that NASA has provided 20 test programs (much more than any business entity) to each developer for quality evaluation (QA)-but the software is still flawed. Therefore, the goal of quality evaluation should not be to find all bugs because it is impossible. Instead, the goal of quality evaluation should be to improve the confidence that code runs well, and thus maximize the availability of resources.

To efficiently run quality assessment (QA), you need to budget the resources available in the available QA methods to maximize confidence. A large coverage test suite can improve our confidence in the use of code because it carries out a thorough code review. Performing two times is better than performing one at a time, because each time you find another error that may be missed. Two times also follow the rule of diminishing returns, so the QA program with a test value of X dollars and a code review value of Y is better than any test or code review with value X+y.

Add static Analysis

Static analysis is the process of parsing a code without running it, which is very similar to what we did in the previous code review, or to what the IDE does when the suspect structure is tagged. Static analysis is an excellent technique for adding to QA Mix (QA Mix) because it is good at finding errors that might be missed by other methods, such as Tests and code reviews. Static analysis is relatively easy, and unlike unit tests you have to rewrite tests for each class that you want to test, you can run the static analysis tool on any code.

FindBugs is an open source static analysis tool that contains bug pattern detectors for many common bug patterns, and surprisingly, even in well tested software, FindBugs often finds "silent" bugs, but unit tests and professional code reviews may miss this Some bugs. FindBugs also allows you to write new bug pattern detectors and wrap them as plug-ins, so you can easily write your own detectors if a standard set of detectors doesn't perform as you want. This extensibility makes FindBugs a very powerful quality management tool, because when a new type of error is found, you can write the detector for the error and search for the error throughout the base of the code.

The primary role of static analysis is to analyze the output and determine whether the reported entry is a true bug or false alert. Some of the best analysis tools or bug pattern detectors will manage the false positives rate; The detector in the core FindBugs package has been tuned to make the false positives rate no more than 50, so there won't be too much annoyance when analyzing the output. (this threshold is compared to the Lint-like tool for C, which often emits many false alarms and is time-consuming to use.) ）