Java implementation of pairwise testing

This article only uses jar packages hosted by others and does not involve specific algorithms

Software testing often takes the development resources of software development organization 30%~70%, and how to effectively improve the efficiency of software testing is one of the key issues that development organizations should consider. Because in many cases poor-lift testing is not feasible, testers must weigh the time and resource limitations with the thoroughness of test coverage. One effective way to solve this problem is to pairwise testing.

First, pairwise testing

Let me give you an example

In the test case design process, sometimes encountered a number of variables to be combined, the test activity of two or more variables together is a combination of tests, such as

Operating system: WIN98,WIN2K,WINXP printer: HP 4050, HP 4100 Duplex: Y,n

A full test needs to include an array of all these parameters:

Case OS Printer Duplex1Win98 HP4050Y2Win98 HP4100Y3Win2K HP4050Y4Win2K HP4100Y5WinXP HP4050Y6WinXP HP4100Y7Win98 HP4050N8Win98 HP4100N9Win2K HP4050NTenWin2K HP4100N OneWinXP HP4050N AWinXP HP4100N

With pairwise testing, test cases can be simplified into:

 case OS printer Duplex  1   Win98 HP 4050   Y  2  Win98 HP 4100   N  3  win2k HP 4050   N  4  win2k HP 4100   Y  5  WinXP HP 4050   Y  6  WinXP HP 4100  N 

A combination test (combinatorial test) is a test case generation method. It abstracts the test application into a system that is affected by multiple factors, each of which is discrete and limited in value. The two-factor (pairwise) combination test generates a set of test cases that can cover all combinations of values for any two factors, theoretically exposing all the defects caused by the combination of two factors. The multi-factor (N-WAY,N>2) combination test can generate a set of test cases to cover all combinations of values of any n factors, and in theory you can find the defects caused by the interaction of n factors. Because the two-factor combination test has a good balance between the number of test cases and the ability of error detection, it is the current mainstream combination test method.

Paired test (pairwise testing) is an effective test case generation technique that reduces the number of test cases by combining all the dimensions and values of the test variables to avoid exhaustive testing of all the values and combinations of all the dimensions.

The concept of paired testing was presented by Mandl in 1985 when testing the AAD compiler program. The "Sort" command in UNIX was tested using a pair of combination overlay testing techniques from Cohen and others. The test results show that coverage is over 90%. The visible paired composite overlay is a very effective test case design method.

In order to better understand the concept itself, the introduction of several related concepts is introduced first. Factors are the parameters or variables that affect the behavior of the software; the level is the possible value of a certain factor; intensity is the degree of the relationship between the factors, the value is two only consider the impact of the value of 22 factors on the software behavior, the value is three when the value of any three factors to consider the impact of software behavior; Is the number of test cases that are generated.

Because "most faults are caused by the interaction of up to two factors", pairwise testing is the case of a strength of two, which requires that the combination of all levels of any two factors be overwritten at least 1 times, and that the resulting test case can guarantee that each factor will appear at least once for each of the 22 combinations between levels. When the number of factors and levels is large, pairwise testing has a significant effect on the streamlining of test cases. Paired-test algorithms have been implemented by a number of tools, such as Tconfig, Microsoft's Pict, and so on.

Pairwise testing has an official website: http://www.pairwise.org/

These include a large number of pairwise testing tools: http://www.pairwise.org/tools.asp

Visual inspection of most of the links need scientific Internet, the most famous is PICT.

Second, the mathematical basis of pairwise testing

Pairwise testing has a rigorous mathematical basis-orthogonal table.

Do not use the orthogonal table, you want to use PICT.

Some articles describe the method of constructing combined test case sets using orthogonal tables. Although feasible, the orthogonal table is not the ideal combination test tool.

First, the nature of the orthogonal table does not apply to software testing. Orthogonal table is for the orthogonal experiment Service, it asks the combination of any two factors to implement "equal probability" coverage, so that the experimental samples are "evenly" distributed in the sample space. "Equal probability coverage" helps to determine the contribution of each factor to the experimental results in the orthogonal test, but it does not help to improve the error detection capability of the combined test. Constrained by the "equal probability overlay", the orthogonal table generates more test cases and increases the cost of testing than the "overwrite-can" combination test tool.

Second, it is not convenient to construct combined test cases using orthogonal table. Using orthogonal tables to construct test cases, the test case set can be obtained by finding the correct orthogonal table, tailoring it, and substituting parameters. If you use the Microsoft-provided combination test tool PICT, it is very easy to generate a test session.

Iii. PICT Tools
PICT (pairwise independent combinatorial testing) is a command-line tool used internally by Microsoft to automatically generate paired test cases, which is now available externally and can be downloaded from the Internet.

Pict Microsoft Related: http://blogs.msdn.com/b/testercenter/

Pict:http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi

PICT can effectively design the test case according to the principle of 22 test. When using PICT, you need to enter all parameters related to the test case to achieve full coverage effect
1. Model files
PICT receives a plain text file as input and then outputs a collection of test cases. Describe the model that is being tested in a text file, that is, what factors the software has, and what values each factor has. The text file is called a model file. The format is as follows:
<ParamName1>: <value1>, <value2>, <value3>,...
<ParamName2>: <value1>, <value2>, <value3>,...
For example: Save the text file as Test.txt and save it in E:\PICT\

Os:win8,win7,winxp,linux,macos Database: oracle,mysql,sqlserver,accessjdk: 1.8. 0,1.7. 0,1.6. 0,1.5. 0 Character set: UTF-8, GBK

2. Running the PICT program
Command line:

E:\pict>pict Test.txt

You can save the result output to a txt and xls file:

E:\pict>pict test.txt > Output.txt

E:\pict>pict test.txt > Outputfile.xls

3.PICT Options
/o:n number of combinations, the default value is 2, that is, each test data in a test case set that is generated by PICT will have two values different from the other test sets;
The delimiter between the/D:C value and the value, by default, is a comma (,), such as a parameter the operating system Winxp,win7,win8 is separated by commas;
/a:c delimiter between aliases, default is pipe (|), such as a parameter (user name is not case sensitive, but in order to ensure the credibility of the test, can be used in the case of rotation of the test) user name admin| ADMIN;
/N:C invalid value or is the prefix of the illegal value, the default value is (~), such as a parameter (can only take 1, 2, in order to test the input of illegal values, take 0, but do not want to 0 and other parameters of each value of the match once, in order to reduce the number of test sets, in front of 0 plus a ~)
/e:file defines a seed file, which can be used to specify a combination of methods, such as specifying a combination in the seed file Seed.txt, when executing 1.txt, the command: PICT 1.txt/e:seed.txt The resulting test set will contain the set of combinations specified in the seed (of course the specified set of combinations has certain conditions). (The current understanding of this parameter may be erroneous);
/R[:N] with the/R parameter, you can make each test set different;
/C plus this parameter, which indicates that the parameter values are fully case-sensitive
/s Display model statistics

More details on 4.PICT

For more complex usage of PICT, see the Help documentation for PICT, and refer to the

Liang Shi implements combination test http://www.cnblogs.com/liangshi/archive/2010/07/25/1784666.html

Wade Xu The factors of constraint relationship how to implement combination test http://www.cnblogs.com/wade-xu/p/4220496.html

Iv. Java implementation of pairwise testing

1.Pairwise Testing Service

Specific algorithms for pairwise testing on-line information less pathetic I only found a pairwise testing service project hosted on Google Code

Http://code.google.com/p/pairwisetestingservice/source/browse

This project introduces pairwise testing in great detail, including core packages and source code, and a construction orthogonal arrays.ppt and a pairwise testing-orthogonal Arrays Strategy.doc file, the mathematical basis and implementation method of pairwise testing are introduced in detail. In addition, the project was written by Chinese programmers, who expressed high respect for the author.

Google Code Close, I put pairwise testing Service part of the file in My network disk: http://pan.baidu.com/s/1dDo3t7r

2.Pairwise Testing Service algorithm

Pairwise testing is a large family, according to the theoretical background of different, can be divided into computer science and mathematical methods of two major categories. The methods of computer science are based on the heuristic algorithm, which can not get the optimal solution, the typical representative is Jenny, PICT and TVG. The typical representation of mathematical method is orthogonal table, although it can get the optimal solution on some specific occasions, but it is limited by the theoretical limitation of mathematical basis and can not be applied to all occasions.

pairwise testing Service is a pairwise testing based on orthogonal tables .

In other words, the pairwise testing service algorithm is different from other PICT software!

The heuristic algorithm (heuristic algorithm) is presented relative to the optimization algorithm. An optimal algorithm of a problem is obtained for the optimal solution of each instance of the problem. Heuristic algorithm can be defined as: An algorithm based on an intuitive or empirical structure, in the acceptable cost (referring to computational time and space) to solve the combinatorial optimization problem of each instance of a feasible solution, the feasible solution and the degree of deviation of the optimal solution is generally not expected. The two basic objectives of computer science are to find an algorithm which can prove that it performs well and can get the best solution or sub-optimal solution. The heuristic algorithm attempts to provide one or all of the targets at a time. For example, it can often find a good solution, but there is no way to prove that it does not get the worse solution, it can usually solve the answer in a reasonable time, but there is no way to know whether it can be solved at this speed every time. Sometimes people will find that in some special cases, heuristic algorithms can get very bad answers or inefficient, but the combination of data that creates those special cases may never happen in the real world. Therefore, the real-world heuristic algorithm is often used to solve the problem. Heuristic algorithms can often get good answers in a reasonable period of time when dealing with many practical problems.

3.Pairwise Testing Service Usage

We use the following example

Pairwisetesting.jar needs XOM and Google collections. first include the Pairwisetesting-core-1.1.jar,xom-1.1.jar,google-collect-snapshot-20080321.jar in the Java project

Create Pairwisetesting.java

Importpairwisetesting. Pairwisetestingtoolkit;Importpairwisetesting.coredomain.*;ImportPairwisetesting.engine.am.AMEngine;ImportPairwisetesting.testcasesgenerator.TXTTestCasesGenerator; Public classpairwisetesting { Public Static voidMain (string[] args)throwsException {Pairwisetestingtoolkit Toolkit=NewPairwisetestingtoolkit (); //set meta parameter providerToolkit.setmetaparameterprovider (NewImetaparameterprovider () { PublicMetaparameter Get () {//create factors and their level valuesFactor F1 =NewFactor ("User Star"); F1.addlevel ("One Star"); F1.addlevel ("Two stars"); F1.addlevel (Samsung); Factor F2=NewFactor ("User type"); F2.addlevel ("Normal User"); F2.addlevel (Students); F2.addlevel ("Internal staff"); Factor f3=NewFactor ("Packing method"); F3.addlevel ("General packaging"); F3.addlevel ("Reinforced packaging"); F3.addlevel ("Gift Wrapping"); //Strength of 2Metaparameter MP =NewMetaparameter (2);                Mp.addfactor (F1);                Mp.addfactor (F2);                Mp.addfactor (F3); returnMP;        }        }); //using AmengineToolkit.setengine (Newamengine ()); //test Case Builder with text outputToolkit.settestcasesgenerator (Newtxttestcasesgenerator ());    System.out.println (Toolkit.generatetestcases ()); }}

Get output after running

User-Star    user type packaging way one-star ordinary user packaging one-star       students       reinforcement packaging one-star       internal staff    gift packaging two-star       ordinary user    reinforcement packaging two-star       Student       Gift packaging two stars       internal staff    general packaging Samsung       Ordinary user    gift packaging Samsung       General       packaging Samsung       internal staff    reinforcement packaging

4.Pairwise Testing Service Detailed description

The following text is copied directly from pairwise testing-orthogonal Arrays Strategy.doc

In the comprehensive practice project, we apply the theory of orthogonal table to the development of pairwise testing Service, which is embodied in the design and implementation of Amengine (Algebra Method Engine). There are many other engine in the pairwise testing service, showing where Amengine is located.

Testdatatransformer is a translator that almost every engine has to customize, especially for Amengine, whose native test data is just numbers and needs to be converted into user-friendly information displays. Engine is an abstract class whose main interface is as follows:

+ string[][] Generatetestdata (Metaparameter MP)

# string[][] Generaterawtestdata (Metaparameter MP)

Generatetestdata is a public method for the upper call, which internally calls Generaterawtestdata to produce native test data and uses Testdatatransformer to transform the native test data. Metaparameter contains important input information such as factors, levels, and intensities. Each specific engine implements the abstract method of Generaterawtestdata to use specific strategies to produce native test data. The internal design of the Amengine is as follows:

Where the black Arrow refers to the delegate relationship, the Blue Line refers to the inheritance relationship. The green Ols_t2_oaprovider is the realization of the LT2 (tm) type orthogonal table (T is the prime number or the prime power), and the corresponding solution of the orthogonal table produced is optimal. H_2s_oaprovider is based on the direct product of the HA array to construct the two-level orthogonal table, the resulting orthogonal table of the corresponding solution is not optimal. Ols_tu_oaprovider is a solution that is introduced when the number of primes is greater than the level plus 1, and the solution of the orthogonal table produced is not optimal. Ols_oaprovider will delegate ols_provider to produce orthogonal Latin party complete groups, depending on whether the number of levels is prime or the power of prime, the decision to use based on the remainder of the class field of prime numbers Rp_ols_ Provider is also based on the poly_ols_provider of the remaining class domain modeled by irreducible polynomial. The implementation of the red Poly_ols_provider uses the Java implementation of the finite field written by Jan Struyf. Amengine after receiving a request from the upper system, the Oaproviderfactory is commissioned to construct the Oaprovider that can handle the request, and throws an exception if it cannot handle it. This separates the complex logic that constructs a particular oaprovider from the amengine, separating the responsibilities and making the structure clearer.

5.Pairwise Testing Service Availability

Amengine in when the level number of each factor is different, or the number of factors > Levels + 1 o'clock, the algorithm uses an imperfect implementation method, when the gap is too large to use

Another problem is the illegality of the combination of factor-specific values, such as the pairing of IE with the Solaris operating system, which is not a valid combination and should not appear in the test. However, because the orthogonal table is constructed all at once, rather than the heuristic algorithm is a single generation of test cases to gradually meet the requirements of pairwise testing test coverage. The difference of this tectonic process determines that the Amengine based on orthogonal tables is not as convenient as other heuristic-based engine, and the current Amengine implementation does not consider the processing of this situation.

6.Pairwise Testing Service Performance Comparison

On some specific occasions, in particular, when the test case and the orthogonal table can be perfectly combined, because the number of <= level and the level of each factor is exactly the same, the amengine based on the orthogonal table is far better in performance and quality than the jennyengine based on the heuristic algorithm, However, due to the limitations of the mathematical nature, it is relatively narrow and less widely used when the power of orthogonal table can be completely released.

V. Summary

Finally back to the theme of pairwise testing. The goal is to improve the efficiency of software testing, then pairwise testing as the 80/20 principle, just spend a small test effort to get a great test return? Industry applications have given some positive examples, such as the pairwise testing test e-mail software starmail, which shows that the test time is halved after applying the method, and 28% more defects are found than traditional test methods, Productivity is 2.6 times times the original.

The effectiveness of pairwise testing is precisely because it detects all single-mode and dual-mode errors and also detects many multimode errors, which are directly related to the fact that errors in software are often caused by the interplay of a few variables. But be wary of pairwise testing as a software-tested silver bullet. Picking the right factors requires a painstaking effort, with the concerted efforts of domain experts, developers, and testers, and determining the level values of each factor also needs to be combined with existing testing techniques such as equivalence class partitioning and boundary value analysis. A combination of testing techniques should be used in order to achieve better results in practical applications. And in the application of death and death pairwise testing test coverage is not up to the requirements, need to apply the strength of more than 2 combination test or other more stringent testing techniques.

In short, pairwise testing is not omnipotent, need to select the appropriate occasions to apply, to really effectively improve the efficiency of testing, improve the quality of software.

This article can be understood as Google Code on the pairwise testing Service Introduction, a large number of quoted content, if the author's interests caused damage, please contact me directly, I will unconditionally delete this article.

Java implementation of pairwise testing