Automated testing of software testing

Benefits of automated Testing

Can greatly improve the efficiency of the test, the tester can quickly deploy the test script on the specified platform and test the corresponding function.

"Weakening" the impact of individual differences in software testers on test results.

Improve the skill level of the entire Test team.

Defects in automated testing

The flaw in automated testing is that it always goes down according to the established process and doesn't act like a human. Once the functionality changes, the test scripts need to be re-maintained.

Key to automating scripts

To develop a set of high-quality test scripts, not simple recording/playback, but need to meet the following characteristics:

Ability to identify product defects effectively
Good readability and error logs to help testers quickly locate the problem
Able to operate stably, repeatedly, independently and undergo a rigorous review process
After a full script acceptance process

When developing a test script, it is always necessary to remember that the purpose of the script is to expose the problem, and any exception that is thrown when running the script is likely to be a product problem, so you need to avoid hiding the problem in your code.

The developer of a good automation script must first be a good tester who is very familiar with the products that need to be tested to develop a truly effective test script.

How can I improve the maintainability of test scripts? This requires the script to have detailed error logs and readability.

How to improve the stability of test scripts? This requires test scripts to be run independent and repeatable.

When a script fails to run, there are several possible causes:

Script execution failed due to defect in the product itself
False positives due to flaws in the test script itself
Failure due to problems with test environment setup

Unfortunately, in a project, the percentage of script execution failures that really result from product defects is not high, and testers often spend a lot of time troubleshooting script defects and test environments.

Therefore, when developing a test script, you need to be aware of:

After the environment is set up and the data is loaded, it is necessary to have a clear verification step, if the data load failure, timely interrupt the script run and prompt the cause of the error
For each verification point, you need to output the actual values and expectations in the log, if the verification fails, in the log detailed description
Try not to catch a possible exception in the program, the exception should be exposed to the user, so that the tester can clearly know where the exception occurred

How can I improve the readability of my scripts effectively?

Common Code Programming specifications
Take advantage of comments in test code
Separating the test description from the test code

If you do not guarantee the independence of test case execution, you may have the following problems:

Because of the close relationship between all use cases, the failure of one use case execution causes the execution of subsequent series of use cases to fail
Increased the difficulty for testers to solve script problems, use cases fail, testers are difficult to quickly locate the cause of the problem
Testers cannot select some of the test cases from which they run separately

Automated Test framework

A well-designed, automated testing framework that can easily help testers develop high-quality automated test cases.

Before you develop an automation framework, you need to first consider what requirements the framework needs to meet.

The following are the main tasks of a set of automation scripts:

Test environment Configuration
Execute the use case for testing
Record of test results
Test environment cleanup
Test report generation

These processes also form the most primitive functional requirements for designing an automation framework.

A well-designed automated testing framework should have:

Improve the efficiency of test case scripting development-how to make it easy for testers to develop test cases and to be able to separate data and execution processes
The support of the perfect environment construction--Let the testers pay more attention to the development of the test business logic part code
Comprehensive test results reporting capabilities-enable testers to better test business logic

We need to implement the environment-ready work at the level of the test automation framework, specifically to implement the following features:

Setup of test environment and elimination of test data
Execution invocation of the test case script
Perform a result report generation

When designing such an automated test execution engine, the first thing to consider is platform independence.

Factors to consider when developing an automated development point in time

Do not develop automation when the product is not stable: more initial defects, manual testing can also be familiar with the product, the original test plan to find deficiencies and lack of coverage.

Test scripts are used in regression testing: The functions are relatively stable and the plan is optimized.

Start by developing common tasks: pre-emphasis

Test automation scripts should be based on a relatively stable test environment and developed in accordance with a proven test plan. If the initial development of a large number of automated scripts, often resulting in a large number of late script rework, but reduced efficiency.

Selection of automated Tests

Automated testing is divided into two categories: UI testing and API testing, and API testing is a higher-level test method, which is closer to the user's actions than unit tests.

GUI testing often uses the recording/playback method to maximize the simulation of user operations, to stand in the user's perspective to identify problems, and the end user's behavior is closely linked. However, it is often more difficult than expected because UI design changes increase the complexity of automated testing, so GUI testing is useful when the UI tends to be stable.

API testing verifies that the return is expected by directly invoking the external interface of the SOFTWARE product, but does not overwrite the display correctness of the interface UI. API interfaces often do not change frequently and can reduce the amount of post-test script maintenance.

Depending on the characteristics of the product, the API test can be implemented in different ways, either directly invoking the API exposed by the product or invoking the server-side service by impersonating the user's HTTP request.

Acceptance of test scripts

When script development is complete, an efficient script review process and acceptance process are required to ensure high quality delivery of the script.

The product code clearly describes a feature point and is able to determine whether the function is completed through an intuitive check.
The automation script clearly describes the test flow, the function points that need to be checked, and the expected results.

A set of scripts that can run forever without errors is not necessarily a high-quality script, because it is also possible that the script does not have a problem with product discovery.

Automated scripting reviews are divided into two areas: Code review and feature point review.

Code review:

Code conforms to code specification
How extensibility of the code
Read if the code clearly knows the test steps for each test case
Is it good to expose problems that are found when the script runs

Function Point review:

Follow whether the code covers all of the functional verification points, and whether the test steps and verification points are consistent with the test plan

The test script is accepted to ensure that the script user is able to run the script successfully.

If the review is to ensure that the quality and functionality of the code are covered, the acceptance process ensures the operational stability and repeatability of the automation scripts.

Automated Script Acceptance:

Should be implemented by a non-scripting developer with a medium-skilled tester
Products that support cross-platform should be covered to different platforms
Focus not only on whether the script is running smoothly, but also on whether the log is detailed and helpful in locating the problem
Any issues found in the acceptance should be addressed by the scripting developer

Stability of automated tests

The stability of the test automation script directly determines the efficiency of the test.

Factors that affect the stability of automated test scripts:

Hard coding of some input parameters in the script: This is the most important factor that affects the stability of the script.
Script Wait Time hardcoded: The wait time in the development script is not necessarily the run time in the future test environment
Cross-platform issues: different operating systems or databases may be different, so testing must be done on multiple platforms.

How to weigh the relationship between manual testing and automated scripting development

For a more stable test project, consider synchronizing scripts when writing a test plan, the author of the test plan and the developer of the test script, which will greatly improve the efficiency of automated development, provided that each tester has the ability to automate the development of the script.

There will be no frequent changes in environment building and data preparation, so you can consider automating this part of the project in the early stages.

Grasp the degree of test automation.

The necessity of automatic script development with the formula Pinggu

Script Development Execution Cost = Script development effort + (average Debug script workload + Average execution script effort) * Number of executions per product cycle

Manual Execution Cost = Average manual effort * Number of executions per product cycle

roi= script Development execution cost/manual execution cost

If this ROI ratio is within 5, then it will take 5 times times the effort to develop the automation script. In other words, if the script executes 5 times in the future, we'll make the cost back, and we'll be profitable once every time we run it. And if a function point manual test takes half a day, and we need to spend 1 months to develop automation scripts, this ratio is more than 60, that is, to run more than 60 times in order to recover costs.

Automated testing of software testing