Some misunderstandings in exploratory testing

Exploratory testing (exploratory testing) is an important part of agile testing, and its value is different from general testing, such as user story testing or automated testing, which focuses on "unexpected" software flaws, and exploratory testing as a research, heuristic, and solemnity test method is an important complement to general testing. With the popularization of agile testing, exploratory testing has been paid attention to by everyone. This paper mainly discusses some misunderstandings of the test engineer in exploratory testing, and tries to correct these problems.

Myth 1: Exploratory testing is a test technique.

Exploratory testing is not a test technique in itself, but rather a way or an attitude that can be applied to a wide range of testing techniques. So unfortunately, test engineers may not find a software called "Exploratory testing Tools" in the real world to help them visually complete exploratory testing. Cem Kaner, author of test computer software, explicitly defines exploratory testing as "a test that emphasizes the freedom and responsibility of each tester, and each tester continuously optimizes their work value by interpreting learning, test design, test execution, and test results as a parallel activity across the project."

For example, in a Web application project, UI functional testing is essential. Exploratory testing as a thought can and should be used throughout the UI functional testing.

Specifically, the test engineer at some point in the preparation of a test case to start testing a site user registration of the page, step-by Go, input box, date selector ... Everything seems to be going well, but wait, exploratory testing is a heuristic, and we need to do something different at the same time when we are doing general functional testing in a disciplined way.

Watch the user Registration screen! The brain may start thinking like this: I see the cursor flashing, oh, I was the test before the mouse click, if the user only with the keyboard operation, the cursor in each input field traversal is sequential? Hands-on testing! Next, the mouse click on the Submit button, registration success! Oh, what would it be like to fail? Does the user need to re-fill all the data? Hands-on testing! Next, the user registration page is tested, but does it have helpful documentation or hints? As a test engineer we may already be familiar with the product, can you first visit the site and prepare the registration information for the user to understand the necessary content? Does the registered user know what permissions they have? Hands-on testing ...

All these are small examples of heuristic thinking, it can be seen that exploratory testing as a method, can be used in any general test, such as unit testing, functional testing, performance testing, system testing and so on, as long as there is exploratory thinking and implementation in practice, exploratory testing will play its important role, Find hazardous areas that are not covered by general testing.

Myth 2: Exploratory testing is a black-box test.

Test engineers and test managers are not unfamiliar with black-box testing. Black box test refers to the tester to test the product as a well-packaged box, can not see the implementation details, only through the product specifications, user feedback and other ways to design test cases and perform, analysis. Exploratory testing as a test method, in the Black-box test can actually get?? Well-applied, for example, in the UI functional tests mentioned above, exploratory testing can be a useful complement to discovering test points that conventional tests do not take into account. However, one of the principles advocated by exploratory testing is "an effort to gain a deeper understanding of the product under test." One of the important reasons that the general test case coverage is not comprehensive is that the understanding of the product is not deep enough, so it is difficult to excavate deep-seated problems. Exploratory testing approaches white-box testing, requiring test engineers to have a more thorough understanding of the product, creatively proposing and executing some "unexpected" tests.

For example, suppose we are testing a Web application that is deployed on a Java Virtual machine-based Web server (for example, WebSphere Application Server). According to the regular test case, we may test the situation of multiuser concurrent access, the situation of the cluster environment and so on, it looks like the black box test is OK.

Next, we try to start the exploratory test from the perspective of the white-box test. Look at the product use case and source code, OH, see the statement "synchronized (...)", "Lock.unlock ()", naturally, we realize that the product is using multithreaded programming, what does that mean? Multi-core! Oh, is the product in a single-core or multi-core environment in a conventional test? What about performance on a single core? Does the performance on multicore improve? How linear is the degree? Load Balancing? These issues become prey for exploratory testing, design test cases, and execute them, and then analyze the results.

Then look at the code, see a lot of "new ...", also see the product code pre-allocated some memory pool, it is natural to think of memory management, of course, Java memory management is controlled by the Java Virtual machine, what can test engineers do? Some! If you look at the mechanism of a Java virtual machine, you will find the initial heap size, the maximum heap size, whether the garbage collection log is output, the different algorithms for garbage collection, and so on, how does the Web app run by default? As the user increases, will the heap fail because it is too small to run? The heap size has changed, can the original garbage collection algorithm ensure that the response time of the Web application is as fast as before? These problems are heuristic clues to exploratory testing, so we can design the use cases and execute the tests accordingly.

With more and more understanding of the product, exploratory testing will gradually discover more hidden potential risks, usually in the white-box state of the exploratory testing is more valuable, because the results are based on a solid knowledge and understanding, and its point is more targeted.

Myth 3: Exploratory testing is random testing.

The relationship between exploratory testing and random testing is somewhat subtle. Random testing is generally referred to as?? In non-documented, unplanned software testing, it is often possible to quickly identify critical defects at the beginning of a test or to quickly perform regression testing at the end of a test, and, in some cases, to use random testing as part of exploratory testing. On the surface, there is indeed some connection between "random" and "exploration" two words, but to think about it, exploration does not mean random, although it is a heuristic thinking method, but it still follows a certain orderly process, although this process can not be recorded in words, and can not be applied to all test scenarios, But the process of thinking is still reasonable and orderly. Therefore, exploratory testing is a more advanced randomized test, which draws on the free and open nature of random testing, but incorporates the elements of heuristic thinking, more rigorous and formal.

Let's take a look at the user interface when Dr. James Whittaker discusses the software failure model (software Fault models). Several heuristic suggestions are proposed using exploratory testing:

1. Make sure that all types of input error messages are triggered once, trying to find invalid values that developers might ignore.
2. For all input domains, try the numeric value of the error type and the string that may be incorrectly treated and processed. Study the limitations and differences of operating systems and programming languages that are running the product, create some problematic strings, and perform tests.
3. In each input field, type the maximum allowable character.
4. Select some input fields that can cause the underlying data to be evaluated, and repeatedly enter the same values to see if the results are the same.
5. For each output field, consider the different results that appear in different situations, and then apply these to the test to see if the output field is the same as expected.
6. For each output domain, think about which results are not or should not, and then try to trigger these illegal results to see if they can be achieved.
7. An attempt to trigger a recursive function call, such as nesting itself in a document, a hyperlink that is nested within its own link, is similar to an error.
8. Try to trigger the storage overflow of the data and observe the resulting result.

Seeing the above suggestions, we would not consider this to be a random test. Exploratory testing of the user interface appears to be "methodical", although random testing may involve several of the recommendations mentioned above, but this is "random" and the test engineer has no control over the scope of the test coverage. Exploratory testing is different, it will have text records, will do coverage analysis, more orderly and controllable than random testing.

Myth 4: The exploratory testing phase is followed by a general test.

It is a common misconception that exploratory testing is a kind of post-test, and when general testing is over, move out of exploratory testing to find loopholes. Once again, exploratory testing is a test idea and method, not a test technique, which can be applied to a wide range of general tests. Because of this, in the agile world, exploratory testing and general testing are in the same position, with no pre-and post-existing points, so long as the test engineer needs exploratory testing, it can exist.

Taking an iterative cycle as an example, during the planning phase, the test engineer examines current test plans and processes by understanding the user's needs and the development and prioritization of user stories, and mining important test points that are not covered in an existing plan, which is part of exploratory testing. During the formal development phase, the test engineer communicates with the developer during the testing process to find potential product issues, and sometimes a developer's reminder can be an important clue to the test, and exploratory testing is the opportunity to highlight these discovery issues. During the closing phase, the test engineer summarizes the defect list and analysis for this round of iterations, and may find other issues during the review process to prepare for the next exploratory test.

If the exploratory test is at the end of the iteration, then the test engineer misses the opportunity to take part in the project in the planning, development, and other stages, or even in-depth participation in the project, but without the sense of exploratory testing running through it, the chances of discovering the problem tend to slip away, which is the most taboo thing for test engineers. So, if you need exploratory testing, apply this idea to every stage of your testing, maximizing its value as much as possible.

Myth 5: Exploratory testing requires a veteran to do.

Exploratory testing looks mysterious and cool, so you might be a little intimidating, think it's a very advanced thing, and wonder if you're suitable for exploratory testing, and the result is that a Test team might have to hand over the exploratory testing burden to experienced veterans. Is this the right approach? Let's start by analyzing the main features of the novice and veteran.

The novice lacks the test experience, does not have the systematic study and the thought training, likes to follow the instruction to do the thing, but accepts the new thing ability, occasionally may cause some "the small accident" because does not understand the situation, the veteran is the battle-hardened, has the thorough understanding to the test technique and the method, likes has the breakthrough, However, inertial thinking affects the awareness of accepting new things. The characteristic described here is the result of the author's observation in the daily work. Analyze the characteristics, and then look back to explore the test, it is a heuristic way of thinking, we need to innovate, need to jump out of the rules of conventional testing, we need to find the "accident" problem. Compared to the characteristics of novice and veteran, exploratory testing is more suitable for who? This is not a question of right and wrong, novice and veteran have their advantages: the Novice can quickly open their own innovative ideas, the veteran can quickly penetrate the internal mechanism of the product. So, whether you're a novice or an old hand, don't label yourself, exploratory testing engineers need some skill, with these qualities, you can perform exploratory testing brilliantly, and agile testing expert Lisa Crispin summarizes the necessary skills:

1. Careful observer: observe abnormal and undesirable results, and be careful about the assumptions of correctness, and be able to observe the software features or patterns in a subtle way.
2. Serious thinker: Checking tests in the run and changing them to unintended directions can explain the logic of finding defects and provide a clear test state.
3. The traitor of the system: the thought is rigorous, systematization, at the same time must have the diversification viewpoint.
4. Resource Digger: Exploration testers should explore more tools, techniques, test data, friends, and sources of information they can use.

Summarize

Exploratory testing is still in the groping and development stage, the understanding of it is still evolving, exploratory testing as a test method, it is worth testing engineers in a wide range of conventional testing, it needs to build on the product in-depth understanding of the basis, not black box test, although it is a heuristic thinking mode, But with a sense of rationality and controllability, exploratory testing needs to be run through the entire test cycle to realize its value. The right people for exploratory testing are not measured by the novice veteran, but the necessary skills are the factors to consider.

Reprint: http://www.uml.org.cn/Test/201107041.asp

Some misunderstandings in exploratory testing