The art of creating a small test program to eliminate software faults

The art of creating a small test program to eliminate software faults

By chip Camden
Translation: purpleendurer, version 1st

When your program fails, chip Camden recommends that you create a small test program because it will help suppliers and developers to help you. The information about creating such a program is as follows.

Again! No matter how carefully you test each performance of the language, function library, and architecture you use, no matter how carefully you create unit tests for each component, what you get when you finally include it into the application is an incomprehensible fault.

Try every Debugging Technique you know, rewrite and simplify the most suspicious code segment, and kill or remove the entire component. This may help you narrow down the fault to a specific area, but you still have no clue about the error or the cause. If you know the source of the language or function library, you may get further information, but you may still have insufficient understanding of the fault to solve the problem due to lack of knowledge or documentation.

You need help. Submit a question on the forum or contact the author/supplier directly, but they cannot reproduce the question. (Somehow, you know this happens .) They requested that the instance be reproduced. If you send the entire application directly to them, the problem will never be solved because it is too troublesome. Finally, it is gone.

Well, we don't like the ending. How can we rewrite this ending? With paid support, we can throw, yell, and upgrade to force suppliers to spend time on this issue. However, if the entire application program runs and debugging is proved too difficult, then they can still say "unreproducible )". Vendors can only do so much. Even if they leave this question for further research, it may take a long time to answer it. Fortunately, when we encounter such a problem, we can do something to help our suppliers: This is what I call the small Test Program (small test program, STPs ).

"Wow! Wait! When we try to debug, we have deleted all the extra things !" I heard you cry.

You may be telling the truth, but our goal is to eliminate other reasons. Turn the target into a narrow scope of the test program, so that you almost always have more things to do. These two goals seem almost the same, and they overlap a lot, but their coverage is not exactly the same. In the first case, we tried to do everything possible to solve the problem. In the second case, we should do our best to help developers solve the problem. This means we need to take the following steps:

▲Remove the dependency on specific configurations.Hao has no doubt that you have customized your own development environment and saved time through various shortcuts and practices. However, for those who are not familiar with it, it takes time to take any of them. You can either remove these dependencies, create a more common environment, or provide them with a fast installation program that will not be intruded. For example, if you need to set certain environment variables, you can provide a script to complete these settings and then start the application. It is best to completely eliminate the dependency on environment variables. If you set the dependency in more than one place or cannot export the dependency correctly, this dependency will increase uncertainty.

Do your best to remove all custom or third-party components. You have already done this, but it becomes more important when submitting a fault. External components are always blamed-indeed, because they often cause unpredictable problems. Exclude them. In addition, if the external components need to be installed and set, this will delay the developer's viewing time. Developers often encounter troubles when making these components work on their systems. If they do not really need these components at the beginning, it is a waste of time.

▲Reduce the steps that require user operations.If you think you will find problems by running the test program once or twice, you are a blind and optimistic person (Pollyanna ). If they want to run the test program for one thousand times, the fee per minute during the execution time is two working days. In fact, this is because developers are also people-every time a developer has to restart a lengthy and difficult setup step, they need to pause from time to time and sigh and wonder why their lives are so bitter.

▲Clearly record the operation steps.I don't know how many times I received something that claimed to be a reproduction of the problem, with the content "running the application ." Unless the application is very simple and requires no installation or interaction, and the failure is obvious to the [typical incompetent person], this command cannot be reproduced. No matter how obvious it looks, it involves every step-each installation command, the command to start the application, and everything that needs to be entered. If you follow the above steps, this should not be too much.

▲Minimize the number of lines of code executed.The entire program may run in two seconds, but if it executes 30000 lines of code, developers may want to eliminate at least 30000 possible reasons. In addition, this makes debugging complex. If you can compress the entire program to "one, two, get it done !" Then you can learn it.

▲Contains clear fault identification.Don't take it for granted that the developer will immediately tell you that you are 10 pixels too small-please tell them in the step description. Ideally, the application should yell, "This is where I went wrong !". Use assertions, or at least use printf or message boxes for output.

▲Contains a clear success mark.Many times, I have solved a problem in the test program and then encountered another fault. Did I solve a problem they did not report before, but now I can see the fault they proposed? They usually know the second fault, but they don't bother to stop it because they have reproduced the first fault. This is rude. Ideally, you want the test program to be tailored to a specific issue so that the same issue will not appear in another test program. To achieve this goal, it must have a mark of success. This will leave us no doubt about whether the fault has been successfully ruled out.

▲Change roles to test the program.Imagine yourself as the developer responsible for the job and run the test program again to ensure that you have no omissions. Do not run it on your own development system, because your environment settings may be different from those of developers. Use common configurations in a virtual machine to run the test program and make sure that it accurately presents the fault as you plan. This can reduce the number of e-mails for you and avoid the impression that "you are confused.

 

　　Why do you need to create a small test program?

 

Why do you need to invest extra effort to create a small test program? After all, this is their mistake. Let them find it and solve it.

Most of my customers are software developers, so I am looking at this problem from two perspectives. I have to submit them to many software vendors as a person who has hundreds (perhaps thousands) of failures to be resolved over the past 20 years. From my personal experience, I can tell you one thing: the only most influential factor that determines how quickly a developer can solve a problem is whether you pay for the vendor that provides product support or how much you have paid, it's not like shouting out of your hysterical anger, or your compliment, or the credibility that they can respond to on time-but how to clearly explain the fault.

So the next time you need to submit a problem report, you should remember the immortal saying of Steve Martin: "Let's do a little bit ". (End)
Http://www.techrepublic.com/blog/programming-and-development/the-art-of-the-small-test-program/3787)