Software testing no longer black box-threadingtest brings second generation of white box coverage technology

Software Test no longer black box-

Threadingtest brings the second generation of white box coverage technology

Threading test a significant innovation in the field of testing is that in the white box test theory appeared decades later, Shanghai 01 assembled Information Technology Co., Ltd. combined with the test theory of more than 10 years of painstaking research, the first to put forward the second generation coverage technology, this is not a slogan, It's Zoa's real understanding of white-box testing and a deep understanding of the standard third-party testing services after years of basic research and more than 2 years of research and development to achieve the commercial standards of technology. Now let's review the classic test theory:

1. Testing Methodology

Black box Functional test method

Black box function test, is to test the software as a "black box", regardless of its internal structure and what algorithm to achieve the required functionality, but in accordance with the requirements of the functional requirements, the design of the corresponding test cases (including the test input data and conditions set and the expected software run output), By manually or automatically comparing the output (including the output of the characters and the output of the image) to the expected results, the software is tested to verify that the software is able to give the correct results, and to determine whether the software satisfies the requirements and is consistent with the specifications and user manuals of the software system.

The advantages of this method are:

(A) The most intuitive and direct reflection of whether the software is designed to meet demand;

(B) can be completed by manual testing, even without the support of any test tools;

Its shortcomings are:

(A) This test method is difficult to identify certain special types of errors. For example, when the software that corresponds to a set of inputs does not provide any output information – it may just change a certain working state, if the source code processing part of the error, it is difficult to find out;

(B) It is not possible to determine which test cases are valid or invalid (so-called invalid, not that you cannot receive any test results when using a single test case alone, but rather that it does not contribute when used with the previously used test case, but duplicates the tests already completed by the previous test cases);

(C) Avoidable blindness: When the software is modified, because it is not known which test cases can be tested to a module that has been directly modified or affected by a modified module, it is very time-consuming and laborious to run all the test cases again from the beginning and to run dynamically.

White box structure test method

White box structure test law and black box function test method The opposite: it regardless of the software tested to meet the requirements, whether the design of the function, and only focus on the internal structure of the software, in order to design enough test cases, so that hundreds or as many of the program components can be tested to at least once, So as to expose as many software errors as possible.

Advantages of the White box structure test method:

(A) The ability to identify many software errors that cannot be found using functional testing methods;

(B) Each unit can be tested separately before the entire software system has been completed;

(C) test cases can be minimized by the validity analysis of test cases, so as to greatly shorten the time and cost of the software modified reply testing;

(D) Memory leak analysis can be performed simultaneously;

(E) The frequency of branch execution can be analyzed simultaneously;

(F) Software complexity analysis can be performed at the same time;

(G) Data and variable analysis can be carried out simultaneously;

(H) performance analysis can be performed simultaneously;

(I) dynamic operation error location and execution path traceability can be performed simultaneously.

Disadvantages of the White box structure test method:

(1) must be carried out through a special test tool, the user's software should be copied on the insertion of piles (insert record point) to record whether the branches/conditions have been executed or executed the number of times the information;

(2) The running speed of the software being tested will be slowed down;

(3) It is necessary to increase the resource overhead when the software under test is running.

On the misunderstanding of software quality

There are many software development organizations and application software development Department managers mistakenly believe that they have developed the software has done a full functional test (also known as "black Box Test"), that "Our software quality is no problem!" "-but, after analyzing a lot of" well-tested "software, the experts found that about half of the programs in the software were never executed!

Why is that? Originally, the functional description of the software is relatively easy, very simple, but also very coarse, can not be detailed to the implementation of the software within the specific logical structure to illustrate, and to achieve the same function, the software can have a lot of equivalent implementation methods, in particular, the implementation of software functions, and the language used to write programs, The operating system environment that is running, the database used, and some third-party software are all related. In fact, a lot of program branches in a software are not directly related to the function of the software itself, but are used to deal with various possible running situations. For example, how to protect a document that has been opened when the software being developed is suddenly terminated (a system outage or user interruption), how to warn before the system resources are exhausted, how to deal with some files that are accidentally deleted, and so on. There are also possible errors in writing these program branches, which must be added to the test. This usually needs to be done through the program's structural test (also known as "white Box Test"), and the white box structure test must be carried out with the aid of software testing tools.

Threadingtest on the quality of the appeal in the course of the test, for a set of inputs, determine whether their output, if any, is consistent with the expected value, and whether the execution path matches the expected value. This way, even if the test output is consistent with the expected value, there may be errors being found-if the expected execution path is inconsistent with the actual execution path. For example, when testing a calculator program, if the input is a three-to-three, the measured result is 4, it can also be an error-if its execution path is inconsistent with the expected value: Its final result may be the output of the 2x2 path. Since TT can test an input without output (at which point the execution path is only tested to match the desired path), it can be used in any development phase to achieve a veritable full-process test drive.

2, the second generation of white box coverage Technology

Coverage Technology is one of the basic technical means of software testing, but although there have been many theoretical methods and commercial products in the past few decades, it has not been popularized in the mainstream application field of testing industry, mainly due to the following points:

(1) Usually the coverage results must be re-accumulated after the release of the version, for a large program equivalent to the history of testing all zeroed.

(2) The common scenario of software testing is the need to use test tools to analyze the code, and software testing tools, especially the white box testing tools that can meet the commercial standards have been monopolized by several foreign software testing tools, high prices, and for aerospace, military-level testing requirements for information security reliability is poor.

(3) White box test operation is difficult, testers difficult to understand, in the Test team is difficult to promote.

(4) White box testing tools are stand-alone version, it is difficult to promote the use of large-scale Test team.

(5) Coverage is not related to test cases, and typically coverage is a mixed result of performing a series of actions, whereas testers and developers need to be aware of the code coverage of a feature when locating a problem. These traditional white-box test tools are not supported.

(6) With the increasing proportion of mobile applications in the consumer and enterprise market, some of the old testing tools for mobile environments (Android, IOS) significantly support fatigue and even do not provide support.

The above reasons make the first generation of coverage technology difficult to really get promoted. Threadingtest for the first generation of coverage technology, a new second-generation coverage technology is proposed, and based on the coverage method, a new application function is designed:

(1) No need to monitor the test scenario: coverage statistics can be run by the background program is collected, the tester to achieve transparency, testers only need to run the post-insertion program, open the program's automatic collection function, can not be supervised routine testing, TT will automatically collect the test execution of the program, analysis, In the database, with TT you can easily see the real-time coverage of the program.

Figure real-time monitoring interface automatically collects the test program execution and statistics

(2) Bidirectional traceability is an important tool for the TT to reach 100%, and through the two-way traceability function testers can discover all the non-tested branches and develop a confirmation of how to overwrite and supplement the use case. Up to 100% of the critical modules covered by the test. For more difficult to cover program logic, developers and testers can focus on code walk-through and joint use case design. Around coverage results, development and testers can interact fully, and before threading test tools appear, because there is no coverage for this shared data, it is difficult for developers and testers to fully interact and collaborate because developers do not know exactly what program execution logic corresponds to the test case. Testers, however, are not sure how to complete a full test.

Graph bidirectional Traceability Interface test cases and code before interacting by surrounding coverage

 

(3) Support Android mobile app testing based on Java language development.

Figure on the phone to operate on the computer connected to the TT Real-time collection of test information

(4) Cumulative coverage technology: If there are multiple versions of the program coverage results, TT can be implemented to consolidate multiple versions of coverage, and in a view to show

Figure main Interface callgraph chart to select multiple versions of the cumulative coverage Show

(5) Support in the program structure diagram, control flow chart and other graphics display coverage, testing and developers can clearly see the program coverage from multiple perspectives, you can see the overall coverage, you can also see the coverage of a single function, or even to see the coverage of a branch of execution.

Figure coverage Show

Figure coverage Show

Figure coverage Show

Figure coverage Show

(6) Support distributed testing, multiple testers test the resulting coverage, can be displayed in the unified view.

(7) To achieve the U.S. military standard do-178b MC/DC white box structure testing technology, achieve 100% coverage, visual complex conditions combination, so that product quality greatly improved.

With the second generation coverage technology, the entire test can be run in a fully quantifiable environment, and the entire development and testing team can see in real time how much coverage each use case has contributed to the overall test. Dynamically adjusts the entire test process based on metrics such as coverage growth, and can lead to supplementary use cases for key modules with low cumulative coverage. We hope that the domestic professional-grade white box test tool TT, can really make the white box testing technology to do the system upgrade, and for testers to grasp and preferences, and then the Chinese software testing to a new realm.

Software testing no longer black box-threadingtest brings second generation of white box coverage technology