Black Swan series in software Testing (i) understanding black swans in software testing

1. The "Black Swan" in software testing

A few years ago, I led a Test team to omit a serious bug to the Internet, when the user feedback the bug, we analyze it deeply and reproduce it, and ultimately everyone agrees that it is a coincidence that this bug can happen because it requires multiple conditions to occur at the same time to trigger it, such as " XX algorithm switch must be opened, XX algorithm switch must be closed, XX parameters must take a certain value, the user's use of the environment must be XX scene, the hardware must be the use of XX interface board, the software must be XX version, XX's bandwidth happens to be not enough ... ", where the user has one of these conditions that is not satisfied, the bug does not occur.

Because the impact of this bug is more serious, but also user reports, the usual to submit a defect root cause analysis report. Among them, the report contains a "follow-up how to identify such defects, to ensure that there is no omission?" "We don't know how to avoid this defect happening again, and in fact, if we don't design a use case that satisfies all of these conditions from a forward design point of view, we're not going to do the test." However, we have to think very carefully about how the bug's occurrence is justified, understandable and therefore preventable to fill the void in the report.

Not long ago I read a book called "Black Swan". For thousands of of years, it was thought that there was only white Swan in the world, but after discovering the first Australian Black Swan, this unbreakable idea was overturned. The author points out that the Black Swan event satisfies three characteristics:

-rarity, or contingency, that is, it is usually out of expectation;

-Impact, that is, it will have an extreme impact;

-afterwards (rather than beforehand) predictability, human nature prompts us to weave reasons for its occurrence afterwards, and more or less to think it to be interpreted and predictable.

Are many of the important online bugs not the "black Swans" in the test? They happen beyond your expectations, have an extreme impact, and people always think these bugs can be avoided afterwards.

Think about it, do you have a "Black swan" in your project? January 31, 2013, Amazon's home page paralyzed for nearly 1 hours, do you anticipate in advance ("Black Swan" wrote, "from a symmetrical point of view, a highly unlikely thing to happen, and a highly probable event is not happening is the same")? 12306 website paralysis, high-speed rail accident, ATM withdrawal system failure ... , can you anticipate it in advance? In fact, you may have found that the "Black Swan" is not far away from us, there are always so many phenomena that should not have happened actually.

There are many possible reasons for the "Black Swan" phenomenon, and from a test point of view, how do we know the "Black Swan" in software testing – bugs that are always unexpected and have serious consequences?

2. Turning a blind eye to the "Black Swan"

People are used to turning a blind eye to the Black Swan. "Sociologists have been mistaken in thinking that their theories can measure uncertainty." It is believed that the use of trend analysis tools and complex mathematical formulae can help us to well predict the risks of the stock market, pick a good stock, the result of a few disappointed people; people believe that complex scientific instruments and advanced mathematics, physics, astronomy and other theories can help us well predict earthquakes, tsunamis, weather , the result is often an unexpected occurrence of natural disasters.

People are accustomed to using deterministic theories to speculate on things that are uncertain, and in an age of "variable" everywhere, the "status quo" approach is no longer appropriate. RBT (risk-based testing) is a good way for some people to use RBT in accordance with established routines: at some point in the project, invite stakeholders to collect risks, analyze risks, and then conduct the test activities according to the risk analysis results. But the risk is always changing, the risk is uncertain, you can not anticipate all the risks in advance, you have to learn to "dynamic braking", dynamically and continuously apply RBT technology. Even so, you still can't avoid all the risks, there will still be "Black Swan", because the process of software testing is full of randomness.

REQBT (demand-based testing) is a mathematically rigorous test technique that uses causal graphs to express every atomic logic rule in the business. REQBT's founder, Richard Bender, believes that as long as the REQBT method is used, there will be no missing from the black box perspective, functional, business logic-related, serious bugs. I am well aware of the beauty of this approach, but I still doubt the statement, for there is no reason why there is so much uncertainty in the test, that there is no certainty that a silver bullet can guarantee that there is not a certain kind of "black swan" happening.

But this does not mean that these forms or patterns determine that the technology of the class is of no use, as long as the use of nondeterministic techniques such as ET (exploratory testing), RST (Fast software testing) and so on. It should be said that these deterministic methods (RBT, REQBT, etc.) and the method of uncertainty (ET, RST, etc.) combined to use, "static and dynamic combination" will more effectively reduce the test "Black swan" occurrence.

3. Judge whether it is "Black Swan"

Want to know what important software flaws are "black swans" and which are not? Just ask, are these bugs as expected compared to what they were expecting before?

We will find that these important bugs can generally be divided into two categories: one that can be expected to occur, but does not take timely measures to circumvent it; This inspired us to do RCA (defect root cause analysis), the analysis of different types of bugs will bring different benefits.

Analyzing the serious flaws that can be expected can be a good way to help us improve the deficiencies in the identified class of methods or technologies that are already in place. For example, in the implementation of the RBT process, why did not identify a risk? Or why is there an insufficient estimate of the risk level? Or why not take risk aversion to known risks? For example, in the process of applying MBT (model-based test Technology), why is a factor missing during modeling? Or why is an important parameter ignored when generating test conditions or test cases based on the model? For example, in accordance with the Enterprise's established process management software testing process, why in the test strategy or test plan to ignore this risk? Or why not cover this risk point in the test analysis and test design? Or why did you not find this flaw in the test execution process? Wait a minute. With this RCA process, you can better help us improve the existing test methods or processes.

The analysis of serious defects that are not expected can provide us with more input and ideas for conducting uncertain test methods or techniques. For example, the example of an algorithm that is presented at the beginning of this article is unlikely to be possible by determining the test method of the class, forward design (that is, the expected) related test cases. Instead, we may have a greater chance of finding this defect in ET, such as adding a session to the complex scenario test that appropriately increases the combination of factors associated with the user's application environment.

"Black Swan" is a book about uncertainty. "There are two ways to recognize phenomena. The first is to exclude abnormal phenomena, only to focus on the normal phenomenon. Researchers ignore accidents and only study normal cases. The second approach argues that in order to understand this phenomenon, people need to first consider extreme phenomena, especially when they have extraordinary cumulative effects, such as the Black Swan phenomenon. "For testing, if you want to know about our shortcomings, these two types of defects (which can be expected defects and defects that cannot be expected) should be analyzed."

4. What we don't know is more meaningful

"The logic of the Black Swan is that what you don't know is more meaningful than what you know, because many black swan events happen and escalate because they are not expected." "The authors believe that the world is dominated by uncertainty, whether it is right or wrong, that the conclusion is very applicable to testing alone, and that software testing is dominated by uncertainty." Testing is the search for unknown processes, which include unknown flaws, the quality of unknown subjects, what the unknown real object is, and all the information that you think you already know, but that you don't actually know. These unknowns give testing a lot of meaning as well as challenges.

It can be said that all test cases are designed based on the knowledge of the testers, and that the execution of these use cases reveals the defects that can be expected, and the "Black Swan" in the test--how many of the important, unintended flaws are discovered by test cases?

Analyzing every "Black swan", it will help us to discover that what is known to us as "unknown", every "Black swan" omission, is due to some "unknown" factors, or unknown knowledge or information, or do not know the importance of certain information to be valued. In the next test, consciously try to broaden our known areas and reduce the number of "Black Swan" occurrences.

Exploratory testing is an essential part of broadening the "known domain". When we need to explore, when we do not know what to test next, when we want to get more ideas, we use exploratory testing, ET helps broaden our test depth and test width, extend the boundaries of known areas as far as possible and narrow our uncharted territory.

5. Conclusion

Do you see a flaw in a very serious consequence precisely because you think it should not happen? Isn't that weird? No matter what test method you use, how powerful your test team is, how much effort you put into the test, the Black Swan will always happen.

In the book Black Swan, the author points out a weakness of human nature: "Accustomed to learning accurate things, not the totality of things", and "as a result of focusing only on the pure and clearly defined ' form ' of the error" called ' Plato '. The platonic thought of the test will allow you to focus only on external forms, for example, whether the process of testing is complied with, whether the template being tested is applied, and whether the specific steps of the test method are implemented, and begin to overlook other less specific, less-than-clear and well-defined transactions, ignoring those that appear to be somewhat confusing and unpredictable, For example, in some people's eyes, exploratory testing is not easy to manage, difficult to explain to others, the steps are not so clear, there are too many unpredictable things in the inside.

Just as agile admits that change always exists and embraces change, we should also confront the existence of the "Black Swan" phenomenon in testing and try to minimize the chance of "Black Swan" (expanding known fields, reducing unknown fields) and trying to do the best possible There has been a chance to grasp the Black swan as best as possible (analyze those unknown points, better apply the uncertainty class method, seek the test improvement).