Top 10 tips for Embedded Software Testing

In the embedded software development process, generally, the time spent on testing and coding is (in fact, more ). This ratio keeps decreasing as your programming and testing level increases, but software testing is very important to the average person. Many years ago, a developer asked Oracle the following question in order to have a deeper understanding of embedded systems: How can I know what my system is doing? Oracle was a little surprised at this problem, because no one asked this question at the time, most of the questions asked by contemporary embedded developers are superficial questions such as "how can I make the program run faster" and "What compiler is best. Therefore, in the face of this unusual but almost mature question, Oracle is delighted and seriously replied to him: your question is very deep and mature, because it is possible to continuously improve the level of understanding. To encourage the persistent programmer, Oracle told him 10 tips about embedded software development and testing:

 

1. Understand how to use tools

 

2. Detect memory problems as soon as possible

 

3. in-depth understanding of code optimization

 

4. Do not make yourself haystack

 

5. Reproduce and isolate problems

 

6. Move back

 

7. Determine test integrity

 

8. Improving code quality means saving time

 

9. discover it, analyze it, and solve it.

 

10. Use the beginner's thinking

 

These 10 secrets have been widely spread in the industry, benefiting many people. This article focuses on these 10 tips.

 

1. Understand how to use tools

 

Generally, embedded systems have high requirements on reliability. The failure of security of embedded systems may lead to disastrous consequences. Even if it is a non-secure system, mass production may cause serious economic losses. This requires strict testing, validation, and verification of embedded systems, including embedded software. As more and more fields use software and microprocessor to control various embedded devices, it is increasingly important to quickly and effectively test complicated embedded software.

 

Just like the tools needed for car repair, good programmers should be able to use various software tools skillfully. Different tools have different application scopes and functions. With these tools, you can see what your system is doing, what resources it occupies, And what external things it is dealing. A tool may help you easily solve problems that have been depressing for several days. Unfortunately, you just don't know. So why do so many people always think of using test tools after they get stuck? There are many reasons. There are two main reasons. One is fear, and the other is inertia. Fear is that new errors may be introduced when skills are required to add test tools or test modules to code, so they always prefer to eliminate bugs by constantly modifying the re-compilation code, the results do not help. Lazy because they are used to simple testing methods such as printf. The following describes some common embedded testing tools.

 

. Source Code-level debugger [Source-level debugger]

 

This type of Debugger generally provides single-or multi-step debugging, breakpoint setting, memory detection, variable viewing, and other functions. It is the most fundamental and effective debugging method for Embedded debugging. For example, GDB provided by VxWorks tornadoii belongs to this type.

 

. Simple and Practical printing and display tool [printf]

 

Printf or other similar printing and display tools are estimated to be the most flexible and simple debugging tools. Printing various variables during code execution allows you to know the code execution status. However, printf imposes a lot of interference on normal code execution (generally, printf occupies CPU for a long time) and needs to be used with caution. It is best to set the print switch to control printing.

 

. Ice or JTAG debugger [in-circuit emulator]

 

Ice is a device used to simulate the CPU core. It can detect the internal operation of the CPU in real time without interfering with the normal operation of the timer. Similar to the features provided by the desktop debugging software: complex conditional breakpoints, advanced real-time tracking, performance analysis, and port analysis. Ice generally has a special CPU, called a bond-out CPU. This is a type of encapsulated CPU that can be accessed through special connections to the internal signal of the CPU. When the CPU is encapsulated, you cannot "see. When used together with powerful debugging software on the workstation, ice provides the most comprehensive debugging function you can find. However, ICE also has some disadvantages: expensive; cannot work at full speed; likewise, not all CPUs can be used as external CPUs. from another perspective, these external CPUs are unlikely to be promptly replaced by new CPUs. Although JTAG (joint test action group) was initially developed to monitor IC and circuit connections, this serial interface extends its use, including support for debugging. The visual DSP ++ designed by ad for Blackfin supports high-speed JTAG debugging.

 

. Rom monitor [rom monitor]

 

A rom monitor is a small program that resides in the embedded system ROM and communicates with debugging software running on the workstation through a serial or network connection. This is a cheap method, and of course the lowest-end technology. In addition to a communication port and a small amount of memory space, it does not require any dedicated hardware. It also provides the following functions: Download Code, Run Control, breakpoint, one-step, and observe and modify registers and memory. Because the ROM monitor is part of the operating software, it only works when your application is running. If you want to check the CPU and application status, you must stop the application and go to the ROM monitor again.

 

. Data Monitor [Data Monitor]

 

When the CPU is not stopped, the monitor not only displays the specified variable content, but also collects and graphically displays the variation process of each variable.

 

. OS monitor [operating system monitor]

 

The OS monitor displays events such as task switching, semaphore sending and receiving, and interruption. On the one hand, these monitors can present the relationship and time relationship between events. On the other hand, they can also provide a diagnosis for problems such as semaphore priority reversal, deadlock, and interruption latency.

 

. Performance analysis tool [profiler]

 

It can be used to test the CPU consumption. The profiler tool lets you know where the system bottleneck is, CPU usage, and where optimization is needed.

 

. Memory Test Tool [Memory teseter]

 

You can find out the memory usage problems, such as memory leakage, memory fragmentation, and memory crash. If you find that the system has some unpredictable or intermittent problems, you should use the memory testing tool for testing.

 

. Run the tracker [execution tracer]

 

It can display the functions executed by the CPU, who is calling, what are the parameters, and when to call them. This tool is mainly used to test the code logic and detect exceptions in a large number of events.

 

. Overwrite tool [Coverage tester]

 

It mainly shows the code executed by the CPU and lets you know that the Code branches are not executed. This helps improve code quality and eliminate useless code.

 

. Gui testing tool [GUI tester]

 

Many embedded applications have some form of graphical user interfaces for interaction. Some system performance tests are performed based on the user input response time. Gui testing tools can be used as script tools to run test cases in the development environment, its functions include recording and playback of operations, capturing screen display for future analysis and comparison, setting and managing the test process (the rational robot and Mercury LoadRunner tools are outstanding representatives ). Many embedded devices do not have a GUI, but they can often insert embedded devices to run GUI test scripts. Although this method may require changes to the tested code, however, it saves time for functional testing and regression testing.

 

. Homemade tool [home-made tester]

 

In embedded applications, sometimes some tools need to be compiled for specific purposes to achieve certain testing purposes. The video stream recording and display tool I have compiled has helped the company find several hidden bugs by helping to test the stream direction and changes of video conferences.

 

2. Detect memory problems as soon as possible

 

Memory problems are very harmful and difficult to troubleshoot. There are three main types: Memory leakage, memory fragmentation, and memory crash. The attitude towards memory problems must be clear, that is, early detection and early treatment ". In software design, memory leakage is the most famous, mainly because the continuously allocated memory cannot be released in a timely manner. Over time, the system's memory is exhausted. Even careful Programmers sometimes encounter memory leaks. A friend who has tested the memory leak is expected to have a profound experience, that is, the memory leak problem is generally hidden deep, and it is difficult to find out through code reading. Some memory leaks may even occur in the database. This may be a bug in the library, or it may be caused by errors caused by the programmer's failure to properly understand their interface instructions.

 

In many cases, most Memory leakage problems cannot be detected, but may result in random faults. Programmers often blame this phenomenon for hardware problems. If the user is not very stable about the system, the restart of the system is not a problem. However, if the user is very stable about the system, such a failure may cause the user to lose confidence in the product, it also means that your project is a failed project. Due to the huge damage caused by memory leakage, there are already many tools to solve this problem. These tools discover memory leaks by looking for unreferenced or reused code blocks, junk memory collection, library tracking, and other technologies. Each tool has advantages and disadvantages, but in general, it is better to use it than to use it. In short, the responsible developers should test the memory leakage problem to prevent it from happening.

 

Memory fragments are more hidden than memory leaks. As the memory is constantly allocated and released, the large block of memory is continuously broken down into small blocks to form fragments. Over time, when a large block of memory needs to be applied, it may fail. If the system memory is large enough, it will take a long time to stick to it, but it will not be able to escape the bad luck of allocation failure. In systems with dynamic allocation, memory fragments often occur. Currently, the most effective way to solve this problem is to use a tool to show the memory usage in the system to find out who is causing the memory fragmentation and then improve the corresponding part.

 

Due to various problems in dynamic memory management, many companies simply disable malloc/free in embedded applications.

 

Memory crash is the most serious result of memory usage, mainly due to the out-of-bounds array access, memory write released, pointer calculation errors, and access stack address out-of-bounds. This memory crash causes system faults to be random and difficult to find. Currently, few tools are provided for troubleshooting.

 

In short, if you want to use memory management units, you must be careful and strictly abide by their usage rules, such as who assigns and who releases them.

 

3. in-depth understanding of code optimization

 

When talking about system stability, people will think about real-time and speed more, because code efficiency is too important for embedded systems. Knowing how to optimize code is a skill that every embedded software developer must possess. Just like a girl who loses weight, she knows where she needs to be reduced most to buy diet pills or equipment to lose it. It can be seen that the premise of code optimization is to find the place that really needs optimization, and then take the right remedy to optimize the corresponding part of the code. Profile (a performance analysis tool with some fully functional ides providing this built-in tool) it can record various situations, such as the CPU usage of each task, whether the priority of each task is properly assigned, how many times a data is copied, how many times a disk is accessed, and whether a program sent and received by the network is called, test whether the code has been disabled.

 

However, the profile tool is insufficient to analyze the real-time system performance. On the one hand, the profile tool is often used after a system problem occurs, that is, the CPU is exhausted, and the profile tool itself occupies a large amount of CPU, so profile may not work in this case. According to the Heisenberg effect, any testing method will change the system operation more or less. This is also applicable to profiler!

 

In short, the premise of Improving the running efficiency is that you must know what the CPU is doing.

 

4. Do not make yourself haystack

 

Haystack is just a vivid metaphor for debugging.

 

People in the group often say shit to the code they are debugging! It can be understood that because the code is not written by him, he has enough reason to go to the shit bug code, as long as he does not write this code, otherwise, the code written by others in the same group may be shit one day. Why is there a needle in a haystack? Someone must have dropped the needle into the sea. Why did the needle fall into the sea? Someone must be careless or hasty. So when you are complaining about the difficulty of finding a needle, have you ever thought about losing it by yourself. Similarly, when debugging is half-dead, have you ever wondered whether you should reflect on the problems that may not strictly abide by the good coding design specifications, but not check the correctness of some hypothetical conditions or algorithms in order to find shortcuts? mark the code? For details about how to write high-quality data, see Lin Rui's "high quality c ++/C Programming Guide" or "0x8 books about C".

 

If you have dropped your needle in the ocean, you must do some preventive work, such as wearing safety gloves, to prevent yourself from being stabbed before finding it. Similarly, to fully expose and capture the root cause of a problem, we can design a comprehensive error tracking code. How can this problem be solved? Process every function call failure as much as possible, and check whether the input and output of each parameter are valid, including pointers and whether a process is called too many or too few. The Error Tracking will let you know where you probably dropped the needle.

 

5. Reproduce and isolate problems

 

If you don't drop the needle into the sea, but in the straw, you should do it well. Because at least we can divide the straw heap into many blocks and find them one by one. For large-scale projects with independent modules, isolation is often the final method to deal with hidden bugs. If the problem occurs intermittently, we need to try to reproduce it and record the entire process for it to reproduce the problem for the next time using these conditions. If you are sure you can use the recorded conditions to reproduce the problem, we can start to isolate the problem. How to isolate it? We can use # ifdef to close some code that may have nothing to do with the problem and minimize the system to the point where the problem can still be reproduced. If the problem persists, open the toolbox. You can try to use ice or data monitor to view changes to a suspicious variable. You can use tracking tools to obtain function calls, including passing parameters, checking for memory crashes and stack overflow issues.

 

6. Move back

 

In order not to make him lost in the forest, the hunter will often shed some marks on the trees for him to find his way out in the future. Tracking of past code modifications is helpful for debugging in the future. If one day, your last modified program suddenly died after it ran for a long time, then your first reflection was what I actually changed, because it was good before the last modification. So how can we check the last modification? Yes, the code control system SCS, or the version control system VCs (concurrent version control, CVS is the evolutionary version of VCs ). Check the previous version and compare it with the current test version. Compared tools can be SCS, VCs, and CVS diff tools or other comparison tools with better functions, such as beyondcompare and examdiff. Record all changed codes by comparison and analyze all suspicious codes that may cause problems.

 

7. Determine test integrity

 

How do you know how comprehensive your test is? Coverage testing can answer this question. The overwrite test tool can tell you which code is executed by the CPU. A good coverage tool usually tells you that there is no problem with code between 20% and 40%, while other bugs may exist. The coverage tool has different test levels. You can select a level based on your needs. Even if you are confident that your unit test is comprehensive and does not have the dead code, the overwrite tool can still identify some potential problems for you. refer to the following code:

 

If (I> = 0 & (almostalwayszero = 0 | (last = I )))

 

If almostalwayszero is not 0, the last = "I" value assignment statement is skipped, which may not be what you expected. This problem can be easily discovered through the conditional test function of the coverage tool.

 

In short, coverage testing is very helpful for improving the code quality.

 

8. Improving code quality means saving time

 

Studies show that more than 80% of software development time is used in the following aspects:

 

. Debug your code (unit test)

 

. Debug yourself and other related code (test between modules)

 

. Debug the entire system (system test)

 

Worse, it may take 10-times to find a bug, which can be easily found at the beginning. A small bug may cause you to pay a huge price. Even if the bug has little impact on the performance of the entire system, it may affect what you can see. Therefore, we must develop good coding and testing methods to improve code quality and shorten debugging code.

 

9. discover it, analyze it, and solve it.

 

There is no panacea in this world. If the profile is strong, it cannot be found. If the memory monitor is good, it cannot be found. If the coverage tool is good, it cannot be covered. Even if you use all the tools, you may not be able to find the root cause of some hidden problems, what we can do at this time is to discover the rules or exceptions through the external phenomena or data output presented by these problems. Once any exception is found, you must thoroughly understand and trace its root cause until it is resolved.

 

10. Use the beginner's thinking

 

Someone said this: "Some things may have various situations in the minds of beginners, but they may be very simple in the minds of experts ". Sometimes, some simple problems may be complicated. Some simple systems may not be designed very complicated because of your "expert thinking ". When you are in trouble, turn off your computer, go out and talk about your problem with your friends or even your puppies. Maybe they can give you unexpected inspiration.

 

Conclusion: Embedded debugging is also an art. Just like other arts, if you want to succeed, you must have the wisdom, experience, and knowledge to use tools. As long as we understand the ten secrets of Oracle, I believe we can succeed in embedded testing.