Automated Testing: is it a silver bullet?

Content: Introduction Isn't automated testing a silver bullet? Is there enough time to test everything? How does automated testing help us? Create automated testing guidelines What can automation do for me? References Thank you About the author Comments on this article Subscription: Developerworks Newsletters Developerworks subscription (Subscribe to Cd and download) Dawn Haynes Technical Specialist, IBM Rational January 2005 There is no pure technological or management improvement that can independently commit to significantly improving the productivity, reliability, and simplicity of the software within 10 years. Brooks encourages us to regard technology and methods as an evolutionary means, not a revolution. When I introduce automation technology into testing, I tend to support the same view. Introduction Frederick p. brooks, Jr. I wrote an article entitled "no silver bullet: fundamental and secondary issues of software engineering" in 1986 (no silver bullet-essence and accidents of software engineering ). This article lists some expectations of the development of software engineering technology and compares them with the reality. His arguments are summarized as follows: There is no pure technological or management improvement that can independently promise to significantly improve the productivity, reliability, and simplicity of the software within 10 years Brooks encourages us to regard technology and methods as an evolutionary means, not a revolution. When I introduce automation technology into testing, I tend to support the same view. I have been dealing with potential customers of automated testing products and solutions for five years, and have encountered many "Silver Bullet" ideas. They always come up with the following ideas: All tests can be automated! Since automated testing can significantly increase productivity, we can complete all the tests with fewer people (fewer people ). Automated Testing is so simple that we don't need any training. The automated method reduces the overall test workload. We do not need to develop any testing solutions. With automated testing, won't testers become "outdated" or "redundant? That time-consuming test design is no longer necessary. Although I don't want to break people's fantasies, I always feel the responsibility to help them understand the difference between implementing automated testing and obtaining the powerful weapon that I dreamed. This usually means explaining the true meaning of automated testing and the actual functionality of automated testing tools and solutions. Isn't automated testing a silver bullet? That's exactly what it means. Automated testing, or the implementation of automated testing strategies and tools, is only a tool in the toolbox of testers. Note that I emphasize that it is a tool located in the toolbox. I intentionally did not equate automated testing with the tester, nor could it replace the tester. Even so, automated testing is still without a doubt powerful, and it can benefit us a lot in terms of test efficiency and thoroughness. The key is to determine the best time and method to play its role. Let's discuss another question in detail. Is there enough time to test everything? I think people will say "No! ". There are always more things to test, or try again on another platform or with other configurations. However, with the approaching deadline and product delivery date, the time allocated to each test cycle is shortened. So how should the software development project manager and test team handle this situation? Generally, they cut down the amount of testing in each test cycle prior to software release. Have you ever experienced this situation? Ideally, some risk-based analysis is required to determine which risks are queued. More often, however, the test team only focuses the entire test cycle on verifying bugs that have been fixed. What's more, even the reduced test plan does not have enough time to complete. How many products are delivered after the test? I do not know much about this situation. Development teams often decide whether to deliver software based on other factors: Is it time? Has the budget exceeded? Have resources been exhausted? Are there any more pizzas and beer? Unfortunately, the development team is unable to fully understand the overall quality of the product due to the arbitrary reduction of the test work, and they are faced with the risk of serious problems with the software delivered. Can we get rid of this dilemma with the power of automated testing? We will discuss it later. How does automated testing help us? Before you plan to implement automated testing, you need to understand the definition of automated testing. In other words, what does it mean to you? Here are some of the other people I have heard about automated testing: Completely unattended tests. Test script. Test tool. Not clear. Sometimes people understand the concept of automated testing too narrowly and focus only on the test scripts generated by tools or programming. In fact, the word automation contains a broader meaning. Let's take a look at the definition of automated testing by a quality engineering team when building a set of automated testing guidelines: In our environment, "Automation" refers to the use of policies, tools, and artifacts, it increases or reduces the need for manual or manual participation or intervention in non-technical, repetitive, or lengthy work. In addition to this definition, the guidelines also provide the team with examples of application automation methods. Table 1 lists some of them. Does this small example allow you to look at automation from another perspective? Now, defining automation is crucial for you and your team. Then you can use this definition to build a set of automation guidelines, so that everyone in the team can use the same method to quickly assess whether a task is suitable for application automation. Create automated testing guidelines The following lists some strategies and considerations you can consider when defining automation and guidelines: Determine the "application" of Automated Testing" Take specific parts of all work as candidates for application automation. Consider highly redundant tasks or scenarios. Automate tedious and error-prone work. First, focus on mature development and thorough understanding of use cases or scenarios. Select the relatively stable part of the application, rather than the variable part. Improve Automation by using data-driven testing technologies (increasing the depth and breadth of test coverage ). Assign several experts to be responsible for automation and do not let everyone in the test team do this. Remember not to pursue 100% automation, manual testing is still crucial. More tests are planned Automate repeated tests to win more time for testing other methods. Added a test. Add a configuration test. Build more automated tests. Perform more manual tests, especially for high-risk features. Careful planning: the division of labor for manual testing and automatic testing cannot be fully automated. All tests and documents should be designed for each design. If an automated test cannot be run, make sure it can be completed manually. Regard automation as an investment Train users to make full use of automated tools. Build a reusable code library. Test modularization is maintained and the size is controlled within a certain range, which is easy to maintain. Documented test scripts (CODE) for verification and reuse. Strengthen the backup process. Use source code control. Recognizing that automation is a software development task usually requires code generation. Step-by-Step Automated Testing Do not automate all tests within one day. Accumulated experience, step by step. From a small part of the entire test plan, it is gradually added to the automated test set. (That is, increasing in an actual and controlled way) What can automation do for me? Although automated testing requires a lot of investment in planning and training in the early stage, it can indeed benefit from several major aspects. It brings you the following benefits: Higher quality software-because you can spend less time and resources for more tests. More comprehensive test coverage potential. More time is invested in other testing activities, including: Detailed plan. Carefully designed tests. Build more complex tests (data-driven, add code for conditional branches and special reports) More manual tests, not less! Automated Testing also provides you with intangible value, which can bring to testers: Opportunities for acquiring new skills (opportunities for building and learning skills ). Learn more about the system in the test, because automation can reveal the internal status of the system, such as object attributes and data. (More understanding of the system makes better testers) Now you know what automated testing is and what it can do. I hope you can use this knowledge to perform more and better tests for your products. Although automated testing is not a silver bullet, it is still an excellent tool. If it can be applied to a suitable job, it will bring you great benefits. References For more information, see the original article on the developerworks global site. For more information about the topics in this article, see the following articles on the CEM Kaner website. Web: http://www.kaner.com/articles.html 1. "ubuntures of test automation" 2. "Improving the maintainability of Automatic Test suites" 3. "avoiding shelfware: a manager's View of automated GUI testing" Thank you Thanks to Cem Kaner for providing the reference link. I would also like to thank TED squire of IBM Rational and James Bach of satisfice and Inc. For carefully reviewing this article and providing help in writing. For more information about satisfice and its widely acclaimed test lectures, visit www.satisfice.com. About the author Dawn Haynes, Technical Specialist, IBM Rational

Http://www-900.ibm.com/developerWorks/cn/rational/r-testing-automation/index.shtml