Automation is not a panacea

Chatting with colleagues one day, he tried to implement a function to enable a program to automatically start after the system restarts and load the specified configuration.

I analyzed it and found that implementing this function is complicated, mainly involving modifications to multiple configuration files. Looking back, I found that this system rarely restarts, maybe four or five times a year. So I suggest him give up on this idea. Because maintenance of the configuration file will become a burden, considering the frequency of system restart, This is not worth the candle.

But it seems that he does not agree. He said his goal is to configure these configurations.WorkNo one is involved. "It is best to move one finger at home. All the configuration work is done automatically for me ". I don't want to disturb him when I see him with confidence. But I believe that even if one day he truly achieves this goal, maintaining this automated system will make him difficult.

In this pessimistic view, I think I certainly did not have it two or three years ago. At that time, I had infinite good expectations for "Automation" like him, as longTestAutomated,Software TestingWork becomes as easy as civil servant now. Just touch your finger and all testcase will run automatically.

But with moreAutomated TestingProject, but also read more success and failure cases. The more we know about the features of automation, the less we expect it.

I think this change is more and more close to the real "Automated Testing ".

　
To make full use of a tool, you must understand its features. Nowadays, in the industry, automated software testing is like a kind of social networking. Automated Testing seems to have been integrated with "high technology" and "high test efficiency ",
"Mature test team "... These comments are linked. In this one-sided atmosphere, the shortcomings and shortcomings of the automation program and the price that must be paid are concealed.

Knowing the lack of automation, I think the first idea to change is "automated testing is not used to discover bugs ". The reason is simple. Automated Testing is based on testcase, but only about 1/4 of all bugs can be detected only by testcase testing,OthersThe bug comes from the smart "person" experience, analysis, divergent thinking, and unclear "a flash of flexibility". These features are completely powerless for automation programs. Therefore, it is completely impossible to implement all the bugs that have been discovered by automated programs.

Another feature of automation is that automation itself is a program and requires a large amount of manpower. The more copies a software buys, the greater the value it has. For automated testing programs, the more times they run, the more valuable they are. Some of the following causes may reduce the number of automated test programs, or even lead to failure of the automation project.

1) frequent changes in product functions or interfaces

2) Automated cases do not require frequent tests, such as the installation process or some non-core functions.

3) The automation program is unstable and easy to run.

I have seen some automated testing programs suffer a lot of discounts for these reasons, and my popularity of "automation everything" has gradually declined.

　
The last feature of the automated testing program is that the automated testing program is not a commercial product and its users are professionals. Therefore, it does not require the high scalability, flexibility, and configurability of commercial products.
Friendly and interactive. Because these features come at a high price, that is, the complexity of manpower investment and software is greatly increased. For designers
You must make the choice simple.

1) handle errors simply and directly, instead of being flexible and intelligent. Print the error and exit directly is the best choice, do not try to guess the root cause of the error and try to recover from the error. This includes user input and environment errors.

2) The Environment Settings must be simple and not compatible with environment changes. In general, testing environments are maintained by testers. Trying to be compatible with the environment may greatly increase the code.

3) For automated testing code, do not use too elegant design skills and complex design patterns. Because the code for automated testing is better understood by the testers who use it, and can even make simple modifications and troubleshooting, this can increase the number of automated testing times.

I don't expect too much automation and do not want to design automation programs perfectly. It is my most profound experience in the design and use of automation testing programs.