When to automate System Testing

There are many additional factors that determine when to automate. The following analysis is based on the practical experience of multiple projects in multiple companies since 1996.

2.2.1 automation time is always the first factor

Creating an automated test project takes more time than creating a manual test project. There are no shortcuts to automation at the beginning. You must perform the same steps for all manual tests. Automation should be completed after all test cases are defined based on well-developed requirements (this is an optimistic idea) and after a build version of aut can be delivered.

In addition, application features that are automatically tested must be valid. If the requirement or application feature cannot be 100% effective, you cannot create an effective and complete automated test based on it. The basic purpose of automated testing is to verify that the subsequent build version or the modified version of the aut meets the requirements once the requirement is confirmed. Moreover, automated testing reduces the time required to perform regression testing, which is a benefit of its acceptance.

If you create a project with tight schedule and there is a tight delivery schedule in project management, you should not consider automation unless you allocate sufficient time for it.

2.2.2 An extreme example

Let's take a look at the number and complexity of the aut build version to be delivered. If you only want to get a build version of a project of normal scale and complexity, testing automation may be of no benefit. In fact, if we only reuse it once during the maintenance period (which is rare in this case), testing automation is not reasonable at all. Even if the project itself is relatively complex, if the number of re-build versions and the deliverables for testing are limited to patching, it is not worth the time to automate.

If the project will be delivered for testing repeatedly, and the new feature set will be delivered at multiple test intervals, and the feature set is complex, automated testing may be of great benefit. It is widely accepted that automated testing takes three days to perform manual testing ~ 4 times the time. If you can determine in advance that there will be more than 3 ~ Four important testable build versions are delivered to you, so your project can choose automated testing.

There is no miraculous combination or precise formula to determine when and when automation should be performed. The following are considerations.

• Is the feature set in aut simple or complex?
• Will the feature set in the aut greatly change in the development process and phase?
• Is the feature set in the aut described as needed?
• Does the feature set in the aut require a large amount of data combinations to confirm all business rules?
• Does the tool used for automated testing interact with all necessary attributes of the features of the purpose of testing? (For example, can it interact like a user? Can we obtain all necessary data from the GUI and sub-objects ?)

As you can see, these problems are complex and difficult to judge. In general, there are some practical guidelines that can be used to determine whether to perform automated testing.

1. If the aut is not complex and not big, do not automate it.
2. If you only receive (3 or fewer) Build versions, do not automate them.
3. If a feature is not 100% effective, do not perform automated tests on it, regardless of the size or complexity of the Aut (you can plan for it, but do not create real automated test scripts, unless this feature can be completed and is 100% valid ).
4. If the development cycle has a tight schedule and each delivery interval is short, you will not have time to automate it.
5. If a feature cannot pass automated testing to 100% accuracy, do not automate it unless it saves a lot of manual testing time. This does not mean that the feature must undergo a 100% test. Note that software testing can hardly cover every feature of Aut. Do not try to achieve this goal. Your tests should not be subjective. The results should be predictable and indicate conditions for passing or failure.

2.2.3 quantitative example

An aut feature set takes six hours for manual testing. If it takes six hours to execute these tests, it takes at least six hours to use automated testing tools to record the tests (probably more, because you will switch between manual test and test data/test Script Creation ). Generally, it takes another 12 hours to organize and establish automated test scripts. In this case, the automated testing of this feature set will take 18 hours in the early stage, three times the time for manual testing. Generally, only 1/10 of the manual test execution time is required to re-run a test script. The test run speed is close to the machine speed, and is usually limited by the response time of the application and the latency of any inserted test script (such as the simulated user thinking time.

It is said that the automated testing takes about 36 minutes. Therefore, when testing the feature set for 4th times, the automated testing starts to save about 5.5 hours of testing time compared with manual testing, in addition, each regression test on this aut feature set will save the same time. Automation takes a lot of time in the early stage, but it can return quickly every regression test execution. This not only saves time, but also increases the accuracy of test execution in addition to human interaction.

Excerpt from"Software test automation" (-just enough software test automation) (US) Mosley (Mosley, D. J.); translated by Deng Bo-Beijing: Machinery Industry Press, 2003.10

Personal Purpose: Many testers do not know why they want to perform automated testing. So I excerpted a section in "software test automation" to explain this problem, and hoped to achieve the goal of making everything happen.