Coverage test of embedded software

1 Overview

Software testing is a broad concept. From the whole process of software life cycle, testing can be divided into modules testing, integration testing, system testing and other stages. Tests can also be classified into static checks and dynamic run tests. In dynamic running tests, white-box tests (or coverage tests) based on program structures and functional-based black-box tests can also be performed. Testing not only focuses on the functions of the program, but also includes testing and strength testing.

To achieve better testing results, in addition to comprehensive testing plans, controllable testing processes, and rich experience of testers, some effective auxiliary tools are also required, especially when the scale of today's software is growing and the testing workload is increasing exponentially. According to the preceding test classification, the test tool can be divided: supports static analysis tools for static rule check and quality evaluation of program source code, tools for dynamic coverage testing of program units, and tools for testing the overall running performance of software systems. In addition, there are some special or dedicated tools, such as protocol tester and memory testing tools. These tools have mature commercial products and can also be obtained through self-developed methods.

This article discusses the coverage testing of a special type of system software, embedded real-time operating system. The content involves the research on the characteristics of such software, the difficulties and characteristics of testing, the adaptability transformation of existing testing tools and the description of testing instances.

2 software coverage testing

Coverage is a white-box testing method. testers must have the specification and list of programs and design test cases based on the internal structure of the program. The basic principle is to use test cases to overwrite the internal logic structure of the program as much as possible and discover errors and problems. Therefore, coverage testing is generally applied in the early stages of software testing, namely, the unit testing phase.

The methods or policies covered are listed in Table 1.

Table 1 Typical coverage policies

Overwrite Policy	Definition
Statement Overwrite	When developing a test case, each statement in the program must be executed at least once. The disadvantage is that some logic errors cannot be found.
Overwrite Determination	Execute enough test cases so that each decision in the program gets a "true" value and a "false" value, or make each branch pass at least one test case.
Conditional coverage	Execute sufficient test cases to obtain various possible results for each condition in the determination.
Decision/condition override	Execute sufficient test cases to obtain various possible values for each condition in the decision and to obtain various possible results for each decision.
Condition combination coverage	Execute enough test cases to make each combination of conditions in each decision appear at least once. It is characterized by sufficient coverage. The test cases that meet the combined coverage of conditions must also meet the requirements of decision coverage, condition coverage, and decision/condition coverage.

From the above brief introduction, we can see that the strict procedures of these coverage policies have the following trends:

Other coverage policies include: modified condition/judgment overwrite (usually referred to as mcdc) path coverage, function coverage, call coverage, linear code sequence and jump coverage, data stream coverage, target code branch coverage, loop coverage, relational operator coverage, etc. As the scale of the software grows, the number of test cases required for comprehensive coverage is also growing. Therefore, it is very important to select an appropriate coverage policy based on the characteristics of the software objects to be tested. At the same time, to determine a reasonable test target and reach 100% of coverage, it is often costly. It should be combined with formal review and other methods to discover more software faults.

3 coverage test tool

Some tools and software are required to achieve better coverage testing results. These tool software generally has the following functional features, which can make up for the defects of human testing:

① Analyze the internal structure of the software to help develop coverage policies and design test cases;

② Works with appropriate compilers to implement automatic plug-in for the software under test, so as to generate coverage information and collect such information during its operation;

③ Calculate the coverage rate based on the collected Coverage Information, and help testers find the software parts that are not covered to improve the coverage rate of test cases.

When using tools for dynamic coverage testing, three elements are required: test cases, inserted tested code, and the tool itself that collects and analyzes covered information. The Code plug-in is automatically completed by the tool. By executing test cases, the tool collects and analyzes the coverage information, you can see the coverage rate indicator. Figure 1 shows the basic process of coverage testing.

4 embedded software coverage testing principles

The major difference between embedded software development and general software is that cross-development is required: Development Tools run on host machines with rich hardware and software configurations, embedded applications run on the target machine with relatively insufficient hardware and software resources. The test tool runs on the host machine, and the information required for the test is generated on the target machine, it is also transmitted to the shrinking host through a certain physical/logical connection, which is received by the test tool. Therefore, an important issue in embedded software testing is to establish physical/logical connections between the host machine and the target machine to solve the problem of data information transmission.

The basic principle of coverage testing for Embedded Software 2 is shown.

On the target machine side, the encapsulated application sends the overwrite information to the Message Queue. A dedicated task is responsible for sending the information to the host machine as appropriate. The contraction host has a dedicated module to receive coverage information. Analysis tools are provided to analyze and dynamically display the growth of coverage rate online.

Tools supporting coverage testing of embedded software should solve the following two key issues:

* Integration with Embedded Operating Systems

It covers the combination of testing tools and embedded operating systems in three aspects. First, on the target machine, the application task and the task specifically responsible for collecting/uploading overwrite information transmit data through the message queue, this message queue can be implemented using the corresponding mechanism of the embedded operating system. Secondly, this specialized task can be considered as a special application task and must be supported by an embedded operating system, because task management is one of the basic functions of the latter. Finally, the communication between the target machine and the host machine can adopt serial port or Ethernet mode, and the corresponding program components of the embedded operating system can be used for the serial drive or network protocol.

* Relationship with other embedded cross-development tools

The development of embedded applications usually adopts the cross-development method. Almost all development tools need to solve three problems: functions of the host machine, functions of the target machine, and connection between the host machine and the target machine. The connection between the host machine and the target machine is a bottleneck. If different tools need to use the same physical line for data transmission, the physical line (or hardware port) should be resolved). For example, in the environment shown in figure 3, various tools of the host machine access the communication line through the unified interface-target server, the debugging agent of the target machine is the core of collecting and transmitting various information (such as debugging information, Coverage Information, time information, and object information.

5 Application of logiscore in deltacore test of Embedded Operating System

Logiscope is a case product provided by the company of OpenGL. It provides various services for coding, testing, and maintenance of software and supports coverage testing of embedded software.

5.1 preparations before testing

Preparations before the test are to support the porting of the deltacore test.

Currently, logiscope has provided support for some mature commercial embedded operating systems, such as PSOs. Deltacore is an embedded strong real-time operating system kernel developed by China. To use logiscript to test the deltacore application and deltacore itself, we mainly solve the 4th key problems described in section 1st.

To support testing of embedded programs, logiscope provides the program code (or the support library of the target machine) that runs on the target machine, including:

* A Main Loop thread used to collect and send overwrite information. This thread is a special task in embedded applications;

* Specific data transmission functions, including serial port or network drivers, will be called by the preceding threads;

* The implementation of plug-in functions. These functions are called by the tested code and placed into the buffer to overwrite the Message Block;

* Manage the buffer information queue;

* Initialization code.

For example, when the tested program runs into an IF (......) The entire process is shown in step 4.

To support deltacore testing, Code related to these mechanisms is transplanted, including the following:

* The main cyclic thread that collects and sends overwrite information as a special task in the application running on the target machine;

* The serial driver is replaced by the serial driver in the lambdatool BSP (board-level support package). The network driver is implemented using the driver in deltanet, a supporting component of deltacore;

* Use the deltacore mailbox mechanism to create and manage message queues, and insert code to send overwrite message blocks to these mailboxes;

* Call the logis initialization function in the root task of the detacore application to create a special task mailbox.

When developing the deltacore application, we used its supporting development tool lambdatool. Because the tool version used does not implement the target server debugging method, the use of physical ports is exclusive, that is, the debugging tool cannot share the same port with other tools. We can use the network to upload and start the target application, and transmit the coverage information through the serial port.

5.2 coverage test process and result of deltacore

Logiscope supports the following coverage policies for functions:

* Command block IBS (Instruction blocks)

* Determine the path DDPs (demo--to-demo-paths)

* Mcdc (modified condition/demo)

The following coverage policies are supported at the project level:

* Process-to-process path PPP (procedure-to-procedure path)

In the deltacore test, we adopted a more common coverage policy-determining the path to the judgment, which means that DDP is a sequence of commands, it starts with a function or judgment (if, while ,......) Its exit is the exit of the next function or the exit point of the judgment. No further judgment can be made between them. For example, figure 5 contains five DDPs:

The test process is as follows:

① Use the plug-in analyzer to plug in the source code of deltacore and generate the plug-in information file.

② Compile the target logiscope program of the transplanted logiscope and the plug-in kernel source code into a library to replace the original kernel library and use the program for use.

③ Compile test cases, use deltacore's various system function calls from the perspective of application implementation, and strive to traverse all the judgment branches of kernel functions, and compile these cases into executable programs.

④ Start the covering information collection and analysis program on the host, and use the lambdatool debugger to download and start the application. The deltacore overwrite information is transmitted to the host. The analysis program dynamically displays the growth of coverage and records the information in a file.

⑤ After the application is executed, start the logis post-event analysis tool to compare the overwrite information record file with the plug-in information file (inserted in the source code in the generated ancillary file, it helps testers clearly understand the path coverage of each function to help improve the test case.

6. The tester modifies the test case and re-performs the test process. The test results can be superimposed and the coverage rate increases.

After more than two months, we tested the coverage of a total of 1.1 functions in 79 files of deltacore 115 with a coverage rate of 70.55%. 89 test cases have been written, and the main 60 API functions have been well covered, with a coverage rate of 100% or about 51.3%.

6 Summary

With the help of the logiscope tool, we have tested the coverage of the embedded real-time operating system deltacore to achieve better coverage. We have found and handled some defects and improved the quality and reliability of the software, however, there are also shortcomings:

① The test should be well planned, including the selection of test sequence, design of test cases, and management of test documents.

② Because the testing method depends on the relevant mechanisms of the operating system, and the tested object is the operating system itself, some code related to these mechanisms are not inserted and tested, otherwise errors will occur. For example, the initialization function OS _init of the operating system has not yet been established before this function is completed. Therefore, inserting it will cause problems, the overwriting information cannot be obtained correctly. For example, for efficiency, some functions related to the system clock are not inserted, because the clock is the most frequently generated external event during the program running, A large amount of overwrite information is generated, which puts pressure on Information caching, transmission, collection and processing. In addition, the tools used do not support plug-in and testing of assembler functions. Based on the above reasons, the overall coverage rate of deltacore 1.1 is still relatively low, and other methods are needed to improve it. For testing non-operating system components and applications, it is expected to reach a high coverage rate because there are no operating system problems.

③ This method cannot be used for time performance testing. Therefore, it is a software-only test method. The generation, transmission and collection of a large amount of data has a great disturbance to the tested program. It can only be used for white-box functional structure verification. For performance tests, some software and hardware tools, such as codetest, should be used.

Testing of embedded software products involves many aspects. Besides coverage testing, it also includes time performance testing and memory usage testing and analysis. This is also an important topic of our research.

From: http://www.uml.org.cn/embeded/qiantest.htm