[Reprinted] ROI Analysis for automated testing (I)

1. Introduction

Many leaders willAutomated TestingIt is regarded as a silver bullet. They believe that automated testing can solve many problems such as testing planning, testing costs, and defect reporting. Automated Testing will bring positive results in many aspects, and there are already many successful cases that enable people to think that automated testing can save costs and solve some testing problems. However, there are also many stories of terror. Disappointment is more painful than expectation and process, and even appears in some cases that have gained benefits. I have encountered many cases where the automated test project fails. These projects have invested a lot and eventually abandoned the automated testing results developed over several years.

The purpose of this article is to enable people to understand and perform automated testing based on practical guidance.WorkThe required investment and possible returns. It describes the business, organization, management, and testing impact that will be encountered during the construction of automated testing.

When planning automated testing, you should consider multiple aspects. For example, automated testing will change the complexity of testing and change the testing organization and management methods from test design to test run. It usually has a wide impact on organizational management, such as task execution, testing methods, and even product features.

Considering the benefits and capabilities of automated testing, we can divide the impact factors into two types: tangible and intangible.

Before and after automated testing, you can use existing measurement technologies (such as code coverage analysis) to evaluate and calculate the testing results. Automated testing can be very effective, increase code coverage, and provide a new perspective to observe the tested software. At the same time, automated testing provides us with a solution that manual testing cannot achieve certain tests. Automated Testing can generate countless commands and combinations, only limited by the capabilities of computers and the time they can be used to run tests. These tests can be used to discover defects on the basis of 100% of the Code. The automated probe program can view the internal state of the program, such as the intermediate processing result, data in the memory, and the status of the internal program, so as to determine whether the tested software can complete the desired function.

2. Management Perspective

We need to set management expectations in multiple aspects: Intangible costs and benefits, unrealistic expectations, common factors for manual testing and automated testing, and organizational impact. We also need to pay attention to the measurement and calculation methods.

The intangible cost is very difficult to calculate reasonably. At the points where they can be measured, there are huge variables when we determine their financial value. It is difficult to calculate the actual value when measuring how much change the automated test can bring. In general, some intangible costs are absolute, sometimes relative, but most of them cannot be differentiated. This depends on a person's opinion and solution. Based on this understanding, we recommend that you try to save these intangible costs from the calculation of the Input-return ratio in most cases.

Examples of Intangible costs:

1) test without user intervention. Although human costs are easy to calculate, it is difficult to quantify the cost of additional computer-controlled behaviors.

2) improved methods for testing institutions. This pipeline often increases productivity, but with the new rules and tasks required for automated testing.

3) The observed sudden decrease in productivity of the testing organization. This observation is generally based on the testing work after the start of the staff began to gradually increase when the phenomenon of stagnation, the installation of test tools and the delay in creating automated test scripts.

4) Not all test groups expect a change. Automated Testing forces individual habits to change a lot, and even some testers still have to perform automated testing when they still need to perform manual testing.

5) number of software product test cycles before release. Automated Testing can quickly confirm product build and inspire people to use it multiple times. However, although the reciprocating cycle can improve productivity and quality, it may also lead to people's laziness, decreased focus, and gradually reduced quality.

6) test coverage rate. It can increase the test coverage rate, or vice versa, mainly depends on the efficiency of manual testing, automated testing tools, and automated testing.

A) some tests can only be implemented through automated tests.

B) It is difficult to measure the value of the change in test coverage.

C) Good Exploratory tests may better find some unusual situations than general automated tests

D) manual testing may make it difficult to automate certain situations or environments

The expectations of automated test management are often publicized by media, conferences, manufacturers, and related books on the advantages of automation. Some information is accurate and applicable, but most of the information appears in some specific environments, applicable to some specific projects, and is overly emphasized. Automated testing is not a silver bullet. It cannot solve all test problems and requires careful planning. Incorrect expectations will eventually lead to a failure case for an automated test with benefits.

For example:

1) All tests should be automated. This is impractical and unpredictable.

2) Get immediate returns from automated testing. Some automated tests may be able to see immediate results, such as build tests, but in general, the return is always visible after a period of investment. It takes a lot of time and effort to create most of the automated test content, and the results are always obtained after testing and running over and over again.

3) Zero start time. Automation of testing takes time. You need to select test tools, build and install them, and planning and implementing automated testing takes several times as much as manual testing.

4) automate all test plans. Automated testing tools cannot do everything.

5) use recording/playback for regression testing. This situation is only applicable to the software under test which is very stable and will change with very few test cases. This is rare.

6) Automatic defect report (no user intervention required ). This usually brings a lot of problems to the testing organization or development. This includes checking whether there are duplicates with existing defects, detecting the cause of failure of errors, failure of Multiple tests caused by one error, and errors that cannot be reproduced.

Organizational Management impact includes the skills required to design automated tests and to execute automated tests, automated testing tools, and automated testing environments. There is a big difference between automated testing and manual testing for development and maintenance. During the construction of automated testing, the work skills and testing methods have changed, and even the testing itself has changed. Automated Testing also has potential impact on the tested products, development processes, and release processes. We have to carefully consider and analyze the positive and negative factors in these influences.

To succeed in automated testing, you must set reasonable expectations on management and correctly identify the benefits to be gained from automated testing. The key is to keep in mind that the goal of automated testing is to make testing better in some aspects. Automated Testing is just a means to complete our task-test a software product. Cost/benefit analysis provides us with useful information in terms of managing testing and investing in testing.

We also need to see that different automated testing practices will bring benefits and problems. For example, automated testing reduces the human resources required for testing and saves manpower consumption during the testing process. However, automated testing may also produce a variety of results, requiring more manpower for analysis, resulting in more labor costs than manual testing. Generally, after obtaining the results of automated testing, it takes longer to analyze and isolate the discovered defects.

Reposted fromWww.51testing.com, Reprinted only for favorites and learning, copyright and related rights are still owned by the author