Exploratory testing learning sharing

I. About testing and exploration

Software testing is about interacting with a software or system , observing that its true behavior is consistent with what you expect to compare.

The author thinks: Before the Test all is only speculation, should take the experimental result as the basis of action. Test the process of constantly exploring and doing experiments.

1. Two sides of the test:

The author believes that the test should include two aspects: a. Check whether the software meets expectations, B. Explore risks.

A. Check whether the software meets the expectations, is based on the traditional test strategy, test design and test execution process, the more people put into the network weaving more dense, the completeness of the test will do the better, but ultimately it is difficult to achieve "no missing" test. There will always be some slip through.

B. On the basis of a a number of exploratory testing, always find surprises, exploratory testing HN is a system based on the design of experiments, rapid and continuous implementation of the experiment, and the resulting conclusions for the next exploration, but also an iterative process.

2. Exploratory Testing Requirements:

1> Exploratory testing is a style:

He relies on the ability and experience of individual testers, emphasizing the sense that testers continue to improve their work value.

2> the difference between exploratory testing and other tests:

The regular st,ct will do the test design and test execution first. The second exploratory test (ET) emphasizes that the test design, test execution is carried out simultaneously, and in the process of continuous learning of the system being tested. Each iteration relies on the knowledge and experience gained from the previous iteration.

A. Continuous deepening of system learning, the depth of testing is also increasing.

B. Make sure that the good test ideas are cured with pride.

C. Constantly updated testing perspective.

D. With a small delay, time, discovery, application of a new perspective.

3. With regard to the conduct of ET activities:

1> testing is cost-and time-management, and how to address time management issues:

Adoption, SBTM (test management based on probing session)//This chapter is not currently open

2> How to prevent the randomness of testing?

Indeed et if the management of the bad easy to get out of control, into the St personnel only to hunt EC and adopt behavior, the starting point may be biased. How to determine the scope and direction of exploration? is a problem, this section mentions that ET needs to be well tested. In the process of continuous exploration need to record good test ideas, problems, risks, harvesting ... This question may not be fully answered.

The 4.ET test can be considered in several directions (previous experience summary):

A. Using similar products, testing experience of products in the same field.

B. Test with previous testing experience of the same product.

C. Testing with previous shortcomings.

D. Leverage the architecture diagram and use cases. (Analysis of deficiencies in existing test designs, based on product design)

E. Exploiting errors and exceptions.

F. Questions and checklists.

5. Other and thoughts and questions:

1>et Scope of application:

From the first chapter of the study and the previous experience of ET. Et feels more for product evaluation than for development-oriented process assistance. Seems to be more suitable for the traditional St team? How do I apply it in an Agile development team? (As a learning question, see if the following chapters are answered)

2>et coordination issues with other agile practices:

At present, the development team in the implementation of the "instantiation of demand" work. Personal feeling in the process of agile transformation, the implementation of the St team ET work must be considered together with agile practices such as the "instantiation requirements" of the development team. At present, when the "instantiation requirement description" is drawn from the user's business objectives, the output is limited to "key instances", and the content of quality attributes is seldom considered. And the focus of the ET test is in the quality attributes of this piece, the individual feel that in the relevant stakeholders to formulate the requirements of the case, you can also put quality attributes and ET content, strategy in the discussion, but also agreed.

Ii. Making charters for exploration

Explore unknown regions VS software exploratory testing

The same point: there is a need to make charters . Otherwise, the exploration of unknown areas is easy to lose direction, and et will appear very large randomness difficult to manage.

For ET, the Constitution is to be based on a certain goal, this goal is the software project stakeholders interested in information, to the project has value, is the project's most concerned about the risk.

1. Regulation Template:

A. Exploration (scope)

The scope of exploration, that is, where to explore? A requirement? A module? A feature?

B. Use

Use of tools, resources, data, technology .....

C. Information

Some aspect of system quality properties (performance, security, reliability, etc...) , agreement consistency, design consistency, etc...

(Do not confine yourself to the form of a template, but focus more on the intent of the probing session.) )

2. High-quality detection regulations:

It is a cue signal, it points out the source of inspiration, but also avoids the action or results made in detail. (Particle size appropriate)

Detection regulation ≠ test cases (excessive exposure details)

The detection charter cannot be too broad (unable to focus, difficult to discern whether the exploration is complete)

Each charter should focus on a single area, such as: exploring some type of security vulnerability

3. The creation of the Constitution:

Reach agreement:

In the agile practice, all close-ups emphasize stakeholder collaboration, such as "instantiation Requirements", the relevant stakeholders jointly identify key instances, acceptance, regression criteria. The same emphasis was placed on collaboration and agreement in the development of the ET Detection Charter, as:

1> discusses how much of his "vision" and "worry" fits with the concerns of other stakeholders to ensure that this "vision" is again worth exploring.

(Personal understanding is that you should avoid looking for EC for EC, and avoid those who expend a lot of energy but do not have much value for the whole software PRODUCT delivery.)

2> digging out these "imagined" messages without the other stakeholder's response is worthless.

3> reveals differences before they start exploring, easing your concerns, and alerting the team to improve overall awareness.

In the process of demand discussion, we get the idea of exploration:

1> the existence of any ambiguity, uncertainty and dependency in the course of discussion is worth exploring.

2> is also an opportunity to identify new needs during the discussion and review your recommendations with stakeholders.

(as the MFQ test design and its agile practices require, testing cannot be just a "back-end" behavior, testing related activities during the requirements phase to ensure that the right thing is done)

Hidden requirements:

Demand is divided into explicit demand and implicit demand (implicit demand is often overlooked) to discover the user a hidden expectation that should be explored, and use it as a probe charter.

Careful consideration of the design will be a good time to discover valuable exploration and make regulations:

(Get enough information for the feature, including design, implementation)

Existing artifacts:

Track code, historical flaws

In an uninterrupted quest, update the Charter:

1>et to continue, and constantly update the perspective of testing.

2> discovers New Idea, records, and explores in the subsequent ET in the process of exploration.

4. Nightmare headline game:

1> envisioned a major catastrophe in the software delivered (in the form of a news feed).

2> the relevant stakeholders to jointly envision what this news might include. (A wide range of perspectives to obtain the most relevant stakeholder risk).

3> chose to share the worst nightmare (risk) with a common belief (not the accumulation of voting method).

4> Brain Storm to find out why.

5> the reason for the detection regulation.

Third, observe the details

The author of this chapter is more like a teacher with rich experience in testing, very grounded gas tells you through the attention to detail to discover software defects cases and ideas:

Did 1> see the bear?

The "Moon Walk Bear" example illustrates the truth that the more people focus on something, the easier it is to ignore other obvious things.

It's like looking down at a cell phone, an acquaintance who walks by or even greets himself with a blind eye ....

As a software test, only focus on one dimension of the test, it is easy to miss the other dimensions of the surprise, such as in the background of the operation of the interface, you should also open log printing, to pay attention to some of the software when the work of some small changes, information, and explore, often can find some good problems.

2> asked a more in-depth question:

Do not be satisfied with the discovery of defects, submit EC single, to learn more about its principle, there is more testing idea

3> perspicacious:

With "Buzz Buzz", you can find great product defects. Look, smell, cut, to pay attention to the details, always find clues in the details.

4> can be tested and invisible to become visible.

This chapter highlights the attention to detail, but the attention to detail often requires some testable means, using everything you can think of, tools, and make your test details more observable.

(Operating System Monitor, Web test browser plugin, grab package tool ...) Also includes some test tools that you have written yourself)

Iv. Identifying meaningful changes

Don't be content with the test points you've analyzed and the idea. Excellent explorers are always able to discover some subtle, deep-hidden changes in the software through surface phenomena.

Variables are things that change:

The variable here refers to something that changes directly or indirectly during the operation of the software.

1. Obvious variables:

The obvious variable is the data input of the software under test, which should be covered in detail in the normal functional testing. Here the main use of the variable has the characteristics of the fractal (fractal), to construct some changes, from the shape of the variable to start.

For example, the tested software processing input binary value 011 is not a problem, you can consider input 110, see if there is a problem?

(But what is the rationale for exploring the corresponding computer's software or operating system?) )

2. Subtle variables:

For example, when interacting with the software, the speed entered, the number of times the software was executed, the number of files in the file system, the file length ..., can be subtle changes in certain contexts:

For example, once there is such a flaw, in the XX test system START process xx seconds--yy seconds between the device to restart the power again, then the system will be the probability of loss of configuration files, in this case, the time point is a very subtle variable.

How to identify these variables:

A. Things that can be counted:

Total number of accounts, number of logins, number of records, configuration number and so on. In particular, be careful of the following:

0,1, more, too much, too little.

B. Relative position and order:

Particular attention is paid to the order of the data, and the data and operations that focus on "start-middle-end" are somewhat similar to the boundary value test.

C. Format:

The format of the test data may have an impact on the test results.

For example, when testing IPv6 related features, I often write 2000:0000:0000:0000:0000:0000:0000:0001-like IP address input as 2000::1.

There are surprises.

D. Point-in-time, frequency and duration:

Time points are somewhat similar to the subtle variables described above, and point-in-time can also be seen as a subtle variable. Speaking of frequency and duration, testers often take a long time to perform some sort of operation to discover some of the software's reliability problems.

In addition, you can find and identify variables that exist in the software in terms of file and storage, relative location, size, depth, input and navigation, and so on.

V. Change Order and interaction

The real situation, the user will not be as you imagine the order, users often do not as you think, according to your given way of operation to operate the software ...

A few years ago, when I was working in Huawei, a banner on the wall, even an idea that I had been preaching, was so memorable that it was written like this:

"Users want to use how to use, users want to test how to test"

This is one of the best for a good user experience. Yes, users often do not use the software the way you think, and for a good user experience, we should not limit the user to the ability to operate the software in a specific way or step.

In order to satisfy the uncertainty of user's use of software, we should also be brave to break the self-inertia in testing and introduce the randomness into the software testing process:

1. Random combination of verbs and nouns of the subject being measured:

Nouns refer to the features/services that the system is capable of providing, while verbs are some of the operations supported by the feature. The random combination of nouns and verbs determines how to use the software, constructs more usage scenarios, and changes the way the system interacts. (This exploration is still focused on the risk of stakeholder concern, focusing on the value of testing)

2. Random Navigation:

This mainly refers to the GUI test, the need to focus on whether to achieve the same purpose there are many different ways of operation, to find out. Like what:

What are the ways to open/close a window? What are the ways to implement a configuration commit/modify/revoke? ........ You can implement a test flow with a random combination of these modes/operations.

3. Characters:

That is, a little bit of testing, the most like hanging in the mouth of the word: "Standing in the user's perspective test." Different knowledge backgrounds, expectations, personality of the same software can be the way the operation will differ. A person who does not understand computer technology, may be in a step of the GUI with the mouse click on the control to do software operations. A person familiar with software and operating systems may use a variety of shortcut keys for software operations, and a developer may write scripts or helper programs to manipulate the software.

Exploratory testing is to think like a user, to understand their personality traits and intentions. Test the software according to their habits, expectations, and even quirks.

VI. Exploratory Testing evaluation results

How do we assess the correctness of the results of exploratory testing when we start doing exploratory testing? For example, if you're exploring extreme scenarios, what if it's a surprise to evaluate a phenomenon or a flaw?

These can be relied upon in the following ways:

Identification and use of features "never", "always".

What are features of "never" and "always"?

Core features of 1> software:

For example, the core function of the base station product is to provide cell signal coverage, call ...., in no case should the base station be retired. When doing exploratory testing, even if your exploration is not explicitly stated in requirements and specifications, but it affects the core functionality of the software, it should be treated as a defect.

2> Quality properties of the software:

If it is found in exploratory testing that violates the quality attribute requirements of product accuracy, reliability, ease of use, accessibility, security, etc., it should be considered a defect.

The risks that 3> stakeholders are concerned with:

Each product should have a risk that should not always be touched in his area. For example, a financial software, any problem that could threaten the user's money is at risk.

How do I get and identify the "never", "Always" features?

1> as a tester, the domain knowledge that the software belongs to must be complete.

2> sometimes different people for the core function of the software, for "never", "always" should be included in the understanding of the content of inconsistent, different people have different knowledge background.

You can ask questions to learn more about other stakeholders, and bring together different ideas, such as:

A. Who, for what purpose, will use the software.

B. What are the alternatives and why others use the product instead of other products?

C. If other features of the software do not work, at least what features must be able to work.

(as in "nightmare headlines" game, use smart set-up questions to understand the risks that different stakeholders are worried about)

Two. Alternative resources:

1> Internal consistency:

For example, the base station transmission quality measurement related characteristics, in doing IPPD (based on ICMP or UDP transmission delay, packet loss rate ...). Inspection) The results of the test should be consistent with other transmission measurements at the same time as the base station, such as Twamp, SLA, or at least one of the features is defective.

2> Standard:

A tested software may not be able to obtain enough information before testing, but at least the protocol specification in its domain should be kept to a minimum. For example, the transmission domain has the RFC protocol family. Wireless domain has 3GPP and so on.

However, as testers should be actively involved in the pre-needs and development process, to ensure the consistency of understanding of the requirements, because the processing of certain reasons may be agreed and fully follow.

3> longitudinal and transverse contrast:

A. Comparison between different software versions of the same product:

The same features, different versions of the software release should be consistent.

B. The same characteristics, the comparison between different products:

For example, IP routing, base station products and digital products (routers, switches ...) And so on are supported, for the base station product software implementation can do a horizontal comparison.

C. There will always be some user habits or conventions in the application area where the SOFTWARE product belongs, which may not necessarily be reflected in the requirements, but our software implementations should conform to these habits and customs.

For example, may not be appropriate, in the field of data communications, may be the continuation of the user's operating habits more inclined to the command line, so the data communication products, command line configuration interface to do a very regular, the user is also accustomed to command line Operation Way. But in the base station product, the user's custom is uses the window UI the way configures, if one characteristic's configuration does not support the UI way, then does not satisfy the user usability quality attribute request.

Three. Features:

Some features are difficult to directly determine whether or not they are correct directly through the software's response and may need to be judged by characteristics.

1> Reference Range:

A. The reference range of the measured object is obtained according to the domain knowledge, and the results are obtained according to the comparison between the test results and the reference range:

C. It is sometimes possible to determine whether the test is correct based on the data characteristics that are present in multiple measurements:

For example, the Goose factory has such a test of the interview question: to provide a lottery big turntable Web application: Three-way prize winning ratio should be 2%,3%,5%, how should you test?

2> stands in the perspective of user experience to analyze:

For example, recently playing a game called "in the Super Wind" soccer Hand tour, games automatic game process (two teams are controlled by NPC) will appear the following phenomenon:

NPC will frequently use tackles, in order to compete for the right to play, exaggerated when the two teams will shovel back and forth 5-6 Times. As a non-soccer programmer, the implementation of the software may not be able to identify any problems, but the player who knows the ball may curse, this is playing football or rumble it?

3> result reversal:

Suppose the software implemented by the test is f (x), and the trusted software that implements its reverse function is g (x).

If B=f (a);  A ' =g (b); If a! = A ' indicates that f (x) may be defective.

For example, suppose the software is now implementing AES-encrypted modules, and the trusted AES decryption module already exists. If the decryption module does not correctly decrypt the key and the encrypted cipher, the original plaintext indicates that the newly implemented AES cryptographic module may be having problems.

Exploratory testing learning sharing