Combination test Method PK Orthogonal Analysis method

During testing, we often encounter test scenarios that require multiple variation parameters, such as when we test a BS configuration Control client organization resource to remotely configure a device, perform a variety of combination tests for a device channel video parameter setting, as shown below:

In most cases, similar to this multi-combination test, the old staff is to rely on experience to carry out targeted testing, and our new staff will randomly random to see if the normal save success, to see if the corresponding video files meet the settings, fortunately, can find a few bugs. This method is not only unscientific, and the test coverage is also very narrow, here can be used in two common methods of test case analysis and Design: combination analysis method and orthogonal experiment design method. (All figures above are examples)

1. Use the combinatorial analysis method for test case design:

On the basis of the combinatorial analysis method, the test case set can be generated by the multi-factor combination test to cover all the combinations of the values of any n factors, and the defects caused by the interaction of n factors can be found theoretically. The simple understanding is that each value of each parameter needs to be paired with at least once for each parameter.

The above factors and the status of each factor are as follows:

Stream Category: main stream, sub-stream, event parameter

Stream type: Composite stream, Video stream

Resolution: Dcif,cif,qcif,2cif,4cif, Reserved, VGA,UXGA, hd720p

#这个设置要根据前段设备的型号, some devices do not support a larger resolution

Image quality: Best, better, second good, general, poor, poor

Code Rate Type: Fixed code rate, variable rate

Video bitrate: 32kbps,320kbps,2048kbps #对于这个设置可以随机选择几个进行测试

Video frame rate: Full frame rate, 1/16,1/4,1,10,20 #对于这个设置可以随机选择几个进行测试

Frame type: Single P-frame, BBP frame, BP frame

I frame interval: 0,200,400 #由于I帧间隔的范围0-400, we randomly select three

According to the traditional idea, the permutation of these parameters is: 3*2*9*6*2*3*6*3*3=104976, that is, we have to carry out 104,976 test execution, this is not possible in the reality test. We can use the combinatorial analysis method to cut the number of test cases to reach our acceptable range.

Here's a tool to use: PICT. The following are the steps:

1. Install the PICT work first

2. Build a model file, the file is a. txt format, such as: Modelfiles.txt. For the above image file, the contents are as follows:

Note: "," ":" in the text file are all English characters

3. DOS window into the PICT installation directory, run: PICT modelfiles.txt

Note: If the combination of test factors is saved to the installation directory and displayed in *.txt format, you can enter commands C:\Program files\pict>pict modelfiles.txt > Output.txt, or *.xls format display, can input commands C:\Program files\pict>pict modelfiles.txt > Output.xls

The combination of the test factors is as follows: (due to the. xls format, input 1/16 shows that January 16, or 1/16 cannot be displayed in. xls, here with 1\16 instead of 1/16, other similar)

With the exported. xls, we know that the same effect can be achieved 104,976 times with only 60 test cases. In the test, we can add some conditional statements in the model file to restrict some special cases, for example, when the screen rate selection is customized, the custom code rate input box cannot be empty, etc. For experienced personnel can be based on the above combination of appropriate to add some of the test cases prone to problems, increase the coverage of the test. Recently read the "Microsoft's software Testing", there is also a description of the combination test, the book suggested combinatorial analysis from the two-factor combination test, gradually improve the combination of dimensions, up to 6 factor combination test, because there are studies show that the 6-factor combination test can find the overwhelming majority of program defects. The description of the passage is not yet clear enough, and it remains to be further studied.

2. Design test cases with orthogonal analysis method

The so-called orthogonal test method, is to select from a large number of test points of the appropriate, representative points, reasonable arrangements for the test.

If we use the orthogonal test method directly in the above example, we may still not reach the scope we can accept, that is, the test case is still many. At this point, we can calculate the weights of each factor and state, and delete some of the lower weights, that is, the less important factor or the state, so that the resulting set of test cases is reduced to the extent that we can accept.

For the above example, we make the following modifications (this is just to point out how to apply the orthogonal test method, the selected modifications are simple, the actual can be truncated as needed):

Stream class: Main stream = 0, sub-stream =1

Bitstream Type: Composite stream = 0, video stream =1

Resolution: dcif=0,cif=1,qcif=2,4cif=3, reserved =4

Image quality: Best = 0, better = 1, second good = 2, general = 3, poor =4

Video frame rate: Full frame rate =0,1/16=1,1/4=2,1=3,10=4,20=5

Frame type: Single P frame =0,bbp frame =1,BP frame =2

Based on the above analysis, we select the matrix table with the smallest number of rows, based on the common orthogonal matrix table, the number of factors, the nearest but slightly larger than the actual value. We select the matrix table for 5^6:

	1	2	3	4	5
1	0	0	0	0	0	0
2	0	1	2	3	4	1
3	0	2	4	1	3	2
4	0	3	1	4	2	3
5	0	4	3	2	1	4
6	1	0	4	3	2	4
7	1	1	1	1	1	0
8	1	2	3	4	0	1
9	1	3	0	2	4	2
10	1	4	2	0	3	3
11	2	0	3	1	4	3
12	2	1	0	4	3	4
13	2	2	2	2	2	0
14	2	3	4	0	1	1
15	2	4	1	3	0	2
16	3	0	2	4	1	2
17	3	1	4	2	0	3
18	3	2	1	0	4	4
19	3	3	3	3	3	0
20	3	4	0	1	2	1
21st	4	0	1	2	3	1
22	4	1	3	0	2	2
23	4	2	0	3	1	3
24	4	3	2	1	0	4
25	4	4	4	4	4	0

For some of the factors that actually contain fewer than 5 states, you can use the determined number of States to remove the padding outside the range of States. Finally, the actual state of the factor is filled in the orthogonal test table according to the above rules, and the acceptable test cases can be obtained.

Here is a summary of the two methods:

1. Orthogonal analysis method is a combination of any number of factors to implement the "equal probability" coverage, so that we get the experimental sample evenly distributed in the sample space. "Equal probability coverage" helps to determine the contribution of each factor to the experimental results in the orthogonal test, but it does not help to improve the error detection capability of the combined test. Combinatorial analysis is randomly generated, and its principle is to overwrite, not to consider the problem of "equal probability" coverage.

2. We know from the above example that using orthogonal tables to construct test cases, the test case set can be obtained by finding the correct orthogonal table, tailoring it, and substituting the parameters. However, clipping it reduces the coverage of the test, and the number of test cases generated is more than we can accept without clipping. The combination analysis using PICT can easily obtain a set of test data, in the regression test, it is convenient to use another group of combination analysis data, improve test coverage.

The use of orthogonal test analysis needs to have a clear understanding of the factors and factor states to be tested, weighting them, which weights are heavier, which is lighter, and is suitable for testing niche use, can deepen the understanding of the test object, familiar with the business process. Combinatorial analysis facilitates the rapid generation of test cases without having a deep understanding of the test subjects. The specific use of which method, to see the specific test subjects, but also to see the tester's own preferences.

Written in 2011/10/29