Combination test method: Pairing Test Practice

Introduction to Combinatorial Testing

A combination test (combinatorial test) is a test case generation method. It is the process of combining the values of the variables abstracted by the system and generating a set of test cases, which is abstracted from the test application to a system affected by multiple factors, each of which is discrete and finite. The multi-factor (N-WAY,N>2) combination test can cover all the combinations of all the values of any n factors, and in theory we can find the defects caused by the joint action of n factors.

Pairing Test (pairwise) It can automatically generate a balanced combination of test efficiency and test coverage based on the conditions you set, which is the main combination test method because of the good balance between the test case number and the error detection ability of the two-factor combination test.

pairwise algorithm

Pairwise is L. L. Thurstone (may1887–30 September 1955) was first proposed in 1927. He is a psychologist in the United States. Pairwise is also the result of the optimization of the traditional orthogonal analysis method based on mathematical statistics.

Pairwise is based on the following 2 assumptions:

(1) Each dimension is orthogonal, that is, each dimension has no intersection with each other.

(2) According to mathematical statistical analysis, 73% of the defects (single-factor is 35%, two-factor is 38%) is a single-factor or 2-factor interaction produced. 19% of the defects are caused by the interaction of 3 factors. As a result, pairwise is based on the highest cost performance of use case collections that cover all of the 2-factor interactions generated.

So the principle of choosing a better test mix is:

• The horizontal value of each factor can be tested;
• Each of the two factor combinations can be tested, which is called pairing test.

Many programs are generated around the pairwise algorithm, the most famous is Reducearray,smartdesgin and Microsoft's PICT tool, this article mainly introduces the use of PICT tools.

PICT (recommended) PICT tool Introduction

The full name of PICT is the pairwise independent combinatorial testing tool, which automatically gives the recommended test mix based on the input to improve test efficiency. is a free software: PICT. After installation, there is a detailed help file picthelp.htm.

PICT generate test Case Step 1, Confirm model file

In the TXT file, verify that all the impact factors and the configurable values of each factor are as follows:

Upgrade type: Shutdown, immediate- postpone device data: Save, do not save whether to force the upgrade: Yes, no client environment: shutdown, restart, power off, off-grid

Note: Before the colon is the factor name, followed by the desirable value, the colon and the comma use English punctuation

2. Run the PICT tool to generate use cases

The file generated by running "Pict.exe picta.txt > Pictb.xls" on the command line Pictb.xls is a two factor combination test case set with the following contents:

Upgrade type	Device data	Whether to force the upgrade	Client environment
Shutdown	Save	Whether	Restart
Shutdown	Do not save	Is	Power
Immediately	Do not save	Whether	Power
Immediately	Save	Is	Restart
Shutdown	Do not save	Whether	Off-Grid
Immediately	Save	Is	Shutdown
Delayed	Save	Is	Off-Grid
Shutdown	Do not save	Whether	Shutdown
Delayed	Do not save	Whether	Shutdown
Delayed	Save	Is	Power
Immediately	Do not save	Is	Off-Grid
Delayed	Do not save	Is	Restart

It can be seen that using PICT to generate a combination test case is very convenient, the tester is easy to understand and modify, the test tool can be easily parsed and run, in addition, you can also define the output file for other format files, such as: TXT file, Word file.

3, the actual situation of the use case for analysis and correction

Analysis according to the actual module

Other advanced applications of PICT 1, PICT parameter introduction

After you enter PICT in cmd, the parameter interface pops up as follows:

///a:c: Delimiter used to specify aliases (alias), Default: | /N:C: Indicates an invalid prefix, default ~///C: Specifies that the model calculation is case sensitive (default insensitive) (case-/s: Displays the statistics for the model (show mode statistics)

2. Define the constraint relationship of each factor in the model file

In the basic theory of combinatorial testing, the value of each factor is independent, that is, the value of factor A does not affect the value of factor B. However, most of the factors being tested apply have a constraint relationship.

Take the above upgrade test as an example, when the factor "upgrade type" value is "immediate", the factor "client environment" is not possible to value "shutdown", "restart", because the option to upgrade immediately after the upgrade, the user can not manually go to shutdown and restart operation

If you do not consider a constraint relationship, you can delete the invalid use case after you build the use case, but this will cause the final set of test cases not to implement two-factor or multi-factor combination overrides, as in the example above:

Upgrade type	Device data	Whether to force the upgrade	Client environment
Shutdown	Save	Whether	Restart
Shutdown	Do not save	Is	Power
Immediately	Do not save	Whether	Power
Immediately	Save	Is	Restart
Shutdown	Do not save	Whether	Off-Grid
Immediately	Save	Is	Shutdown
Delayed	Save	Is	Off-Grid
Shutdown	Do not save	Whether	Shutdown
Delayed	Do not save	Whether	Shutdown
Delayed	Save	Is	Power
Immediately	Do not save	Is	Off-Grid
Delayed	Do not save	Is	Restart

If the two impossible combinations are removed directly from the use case, it is unreasonable to shut down and save without overwriting, so it is best to filter by the tool when generating the test case.

In PICT's model file, the constraint relationship between the factors can be defined by adding the following constraint (contraint) statement.

" immediately " "  Restart "  "  immediately  " " Shut down ";

The modified model files are as follows:

  Upgrade type: Shutdown, immediate   -"Restart Now"" immediately   " shut down themachine";

Based on the modified model file, the resulting test cases are as follows:

Upgrade type	Device data	Whether to force the upgrade	Client environment
Delayed	Save	Whether	Off-Grid
Shutdown	Do not save	Is	Off-Grid
Immediately	Do not save	Whether	Power
Delayed	Do not save	Is	Shutdown
Immediately	Save	Is	Off-Grid
Shutdown	Save	Whether	Restart
Delayed	Do not save	Is	Restart
Shutdown	Save	Whether	Shutdown
Shutdown	Save	Is	Power
Delayed	Do not save	Whether	Power

You can see that the above shutdown and save does not cover the problem, in the new use case has been resolved.

3. Multi-factor combination test

The previous pairwise algorithm has been introduced: according to mathematical statistics analysis, 73% of the defect (single factor is 35%, the two-factor is 38%) is a single-factor or 2-factor interaction produced. 19% of the defects are caused by the interaction of 3 factors. In other words: According to the pairing test, 27% of the defects will be omitted.

And in the total coverage and efficiency between the choice, but also according to the measured object to judge, if the test is medical equipment, then more should ensure full coverage, rather than consider efficiency.

Next, you'll learn how to create a multi-factor combination use case in PICT, two ways

Method One: Command line add parameter

Use the parameter "/o:n" on the command line. For example, by executing "Pict.exe picta.txt/o:3" on the command line, you can generate a three-factor combination test case set.

Method Two: Define a sub-model

The following model file definitions are available in PICT's Help file.

PLATFORM:  x86, IA64, Amd64cpus: Single  , Dual, QUADRAM:128MB, 1GB, 4GB, 64GBHDD:SCSI, IDEOS:  4.0, 5.0, 5.5, 6.032

According to the model file, PICT will cover platform, CPUS, RAM, HDD implementation 3 factor combination, the specific generation strategy can participate. Based on the combination of two factors, PICT implements multi-factor combination coverage for some factors, which helps to achieve a more flexible testing strategy.

4, consider the introduction of random seeds in the regression test

If the same test case is used each time, the test case may simply repeat the same path, overwriting the same state space, and perhaps not discovering the hidden flaw.

If the test object and the test requirements are not changed, each time with a new test case, as the number of regression increases, the test execution can execute more paths, the probability of hidden defects will also increase, in PICT, the role of random seed is this.

In PICT, the parameter "/r[:n" can introduce random seeds for test case generation (N is an integer of random seed) to generate different test cases, as follows:

PICT picta.txt/r:6/r:8

The use cases generated by both commands can meet the requirements of two-factor coverage, and two use cases are different.

5, Guardian whistle statement

Many software use Guardian whistle statements to "filter" Invalid input. For example, in the following code, the IF statement "filters" out all a<=0 input.

int func (int A, int B, int C) {ifreturn  ERROR; ...}

If you do not read the code and do not carefully analyze the specification, the following model files may be developed. In this model, the value of a is-1, 0, 1.

A:-1, 0, 1-1, 0, 1-1, 0, 1

Using the above model, the set of test cases generated contains 9 test cases.

A       B      C  0       1      -1  1      -1       1-1       0      -1  1       1       -1      -1        0  0       0       1  1      -1     -1  -1         1      1 0-1        0  1       0       0

Of the 9 test cases, 6 of the tests were filtered out by the IF statement because of the a<=0. There are only 3 test cases that can perform subsequent logic, which means that only 1/3 of the combination of B and C is actually overwritten. This example shows that the combination test case set will not be able to achieve the goal of two-factor or multi-factor coverage if it ignores the outage of the Guardian whistle statement on the execution stream.

Method One: Artificially modify the Model file

In the face of such problems, testers should carefully read the specification or source code, and discover a "negative" (negative) value that would result in the execution of a flow break. The negative values are excluded from the model, and the values of the factors are placed in the range of the normal execution stream. For example, for the above-tested function Func, the model file is defined as:

A:1,1, 0, 1-1, 0, 1

After generating the test case set, add one more test case (a:0,b:0, c:0). The test cases generated by the original model can be "through" the Guardian whistle statement, covering factors A, B, C of the 22 value combinations; Additional test cases can override the "filtering" function of the Sentinel statement.

Method Two: Automatic filtering with parameters from PICT

In PICT's model, the illegal (invalid) value is marked with a special symbol "~" (the symbol can be customized, as described in PICT parameters). For example, in the following model, the value of parameter a of 0 is marked as illegal.

A: ~0, 1,1, 0, 1-1, 0, 1

PICT guarantees that the combination of all valid values will be overwritten, and that any combination of illegal and valid values is overwritten. The above model will generate the following set of test cases, a total of 12 articles.

A       B        1         1       -1 1         0        1-1       -1 1        -1        0       -       1-1        1        0                11 1 1 ~0 0-1 ~0-               1          ~0        1        1

If the implementation logic of the object being tested is clear, the first method can be used to generate a smaller set of test cases because it does not take into account the combination of illegal and valid values. If you just learned from the specification that a program might have a Guardian post, then you can generate a more "safe" set of test cases in the second way.

Graphical PICT tool (CVG) (recommended)

CVG (combinatorial Variation Generator) is an open source tool based on the WPF and Microsoft Pairwise Development package that defines the combination of dimensions, filtering conditions, and the daily use case writing is sufficient to recommend.

: CVG2.0, the specific interface is as follows, click Help to see the user guide, this is not covered

Allpair Tools Introduction

To make it easier for industry testers to use test cases of combination tests produced by the full-dual test method, James Bach developed a Allpairs tool using the Perl language: Allpairs

Allpair Generating a test case Step 1, confirming the model file

Input model file, the file is inconsistent with PICT format, can be entered into exce, and then select the file-Save as TXT file.

Upgrade type	Device data	Whether to force the upgrade	Client environment
Immediately	Save	Whether	Shutdown
Shutdown	Do not save	Is	Restart
Delayed			Power
			Off-Grid

2. Run the Allpairs tool to generate use cases

Download and extract the Allpairs tool directly to a path and run the command under a Dos window. Switch to the path where the tool is located, enter Allpairs.exe model.txt >testcase.txt or allpairs a.txt > TestCase.txt and enter, Where Model.txt is the model file, Testcase.xls is the generated test case and can be customized.

The output TestCase file consists of two parts:

The first part is a collection of test cases, the "pairrings" column describes the number of unique pairrings found by the full-dual method;

The second section lists all of the pair in full combination and shows the number of times the pair appears in the test case list, and the case number of the corresponding test case.

3, the actual situation of the use case for analysis and correction

Analysis according to the actual module

Resources

Official website: http://www.pairwise.org/

Common Pairwise toolset: http://www.pairwise.org/tools.asp

The pairwise algorithm of the layman: http://m.blog.csdn.net/blog/aassddff261/42776543

Full-dual test method in combination test method: http://www.infoq.com/cn/news/2011/08/combination-test/

Implementing Combinatorial Testing: http://www.cnblogs.com/liangshi/archive/2010/07/25/1784666.html

The webpage explains the seeds in detail (seeding): http://www.amibugshare.com/pict/help.html

and provides a torrent file download: http://www.amibugshare.com/pict/PICT_Seed_template.txt

What is a Guardian whistle statement?

In conditional expressions, where 2 branches are normal behavior, you should use conditional expressions like if.....else ...; if a condition is extremely rare, you should examine the condition separately and return it from the function immediately when the condition is true. Such a separate check is often referred to as the "guardian whistle".

Combination test method: Pairing Test Practice