Automated software testing process and our automated testing

Summary

Every software tester who is interested in software testing or professional will have a strong interest and passion at the beginning of software automation testing. It can be understood that after automation is completed, the repetitive workload of testers will be reduced, comprehensive test data coverage can improve the software quality, a wide range of logs and functions can improve delivery efficiency and ease of problem analysis. advantages, software automation testers are crazy. However, automated testing often brings us frustration and disappointment, because automation will introduce more problems while solving problems for us, the Research and Implementation of many automation technologies will stop here. Therefore, before carrying out automated testing, we should develop an automated testing plan. At present, it is basically from improving the software testing process, defining requirements, supporting product testability, and having a sustainable design, planned deployment.

Improve software testing process

In the process of software testing, in addition to ensuring the quality of software, we are more concerned with how to improve the test efficiency. Therefore, we must first define a specific process to improve the efficiency, then, we should adopt a simpler and lower-cost approach while investing manpower and material resources. In this process, we can introduce automated testing.

At present, more and more project teams are introducing agility, frequent iterations, and frequent regression and execution, which make the project teams with few testers unable to afford; therefore, the idea of automation and agility is also quietly spread.

At present, many project teams are introducing automated tests in Smoke Testing and regression testing. The smoke test requires testing the main process at the beginning of the test. For example, each e-commerce website of an airline has a small requirement (that is, each change is relatively small ), however, the main process such as "Reservation"> "invoice"> "refund" needs to be tested during each test, in such a main test process, a large amount of manpower will be invested to ensure the main process functions. If this part is replaced by automated testing, the tester's main focus will be on the functional points of this change. Regression tests need to be executed frequently to check that the effective test cases that have been executed have not failed due to software changes. Regression tests need to be executed repeatedly and are monotonous. In this case, the test owner needs to accurately analyze which functional points may be affected by the changes. These functional points are made into a list to form a regression test document and then checked against the list and document. The introduction of automated testing can greatly reduce workload and random errors. However, this work also brings many problems, because some module features make it very difficult to achieve automation. In this case, you need to analyze the mentioned defects and write a test execution operation for each defect to detect the problem, thus forming the need for automated testing. With such a good design document, designing a test script is a step-by-step process.

Test requirements

Automated Testing needs are determined by the addition of testability analysis based on test requirements, coupled with the expected results of automated testing. Automated Testing usually has the advantages of solving the following problems: repeated simple logic testing, a wide range of testing, speeding up the test progress and improving the release progress.

When designing test requirements, we should understand that automated testing is designed not only to automate, but to discover more defects, not to improve the skills of automated testers, it serves software quality and software release.

When developing test requirements, you should determine the criteria for successful automated testing. Do not add the automated testing function points during the testing process. Do not use Automation in every part of the test to achieve automated coverage. We need to find the part that can bring the greatest return, and the part that uses automation is the best method, instead of simply looking for challenges, we should adopt automation in every step. We should not simply pursue complexity and functional integrity, so that automation can completely replace manual refinement of each functional point to every detail, the more complicated it is, the more difficult it will be. Therefore, in the process of automated testing, we need to weigh a ratio of return to pay, this is a critical point that requires the experience of our automated testers to control.

Defining requirements for automated test projects requires us to fully and clearly consider various situations, and then give the weighed requirements, in addition, the project personnel can reasonably put forward their expectations for automated testing.

Support product Testability

The testability analysis of products is an essential part of automated testing, and is also the most important part I think from the perspective of technical personnel. In particular, the GUI is automatically tested. During the entire development process, the GUI is often modified or completely re-designed. If the GUI solution is constantly changing, there will be no progress in conducting automated GUI tests. You can only spend a lot of time modifying the test script to adapt to the changes in the graphic interface.

How can we avoid the above problems? At this time, we need to conduct testability analysis on the tested products. More accurately, we do not want to avoid problems but avoid difficulties, because we cannot oppose the improvement of the graphic interface by developers. We all know that software products may use three interfaces: command line interface, application interface (API), and graphical user interface (GUI ). In essence, API interfaces and command line interfaces are easier to automate than gui interfaces. Therefore, you can start with these two interfaces that are easy to automate. If the product does not have similar interfaces, developers must be encouraged to provide corresponding interfaces to support product testability.

Often, our initial automation begins with direct UI operations, because at the beginning of automation, we always hope that machines can simulate manual operations on pages, however, UI page automation testing is the most complex, difficult to maintain, and unstable. Therefore, to do a good job of Web UI automation testing, we must first do a good job of testability analysis. The most common is to regard the page stable process, main function points, and highly repetitive function points as the testability standard for Web UI automated testing.

Whether you need to support graphic interface, command line interface, API interface, or direct operations on the UI for automated testing, if you propose the product testability design requirements at the product design stage as early as possible, it will increase your confidence in automated testing and continue on this road.

Continuous Design

There may be some conflicts with the test requirements mentioned above, but the continuity design here does not allow us to constantly expand the requirements in the same version to meet the needs of automated testing, instead, the related documents and scripts should be completed in a planned and phased manner. Initially we will focus on how to automate the operation, resulting in testing automation failing to achieve the expected results. Automated testing is a long-term process. To be consistent with the features of the new version of the product and other related modifications, automated testing requires constant maintenance and expansion. In the automation test design, it is critical to consider the scalability of automation in the future. However, the integrity of automation test is also very important. If the automated testing program reports that the test case is successfully executed, the tester should trust the results and the actual results of the test execution should also pass. In fact, there are many problematic test cases that are successfully executed on the surface, but in fact the execution fails, and no error logs are recorded. This is the automation of failure. This failed automation will have disastrous consequences for the entire project, and when the testing automation built by the tester adopts a bad design scheme or due to subsequent modifications introducing errors, will lead to this failure of automated testing. Failed automated tests are usually caused by the lack of focus on the performance of automated tests or the lack of adequate automated design.

The simplicity, readability, maintainability, independence, and repeatability of the script must be paid attention to for the continuity design. The test database design of the automation framework, data-driven testing, and log results can be analyzed. Script attention is inseparable from our usual coding habits. This is a long-term refining process and requires the technical accumulation of automated testers; the automation framework and log results require us to select a good framework to support our large automated testing system.

Planned deployment

After the preparations and design work for automated testing are completed, the deployment issue is particularly important. Whether the deployment is successful will directly affect the test execution.

There are currently many tools for process deployment and execution, such as the open-source tool Jenkins. Only simple installation and use documents can be used for implementation and deployment.

Use of autosky framework

Here we will not describe the autosky framework too much. We have detailed user manual documents and demo files, in addition, the framework provides the Test Library design, data-driven testing, and log Result Analysis in the continuity design mentioned above.

Test Library Design: the Framework contains selenium encapsulation, System File Reading, eterm PNR reservation and K-seat, HTTP request operations, various Date and Time Acquisition, XML and Excel Data Source Operations, and compression. file Processing, screenshot processing, etc, it contains almost all test libraries for UI and interface testing.

Data-driven testing: the Framework supports data-driven testing for XML, Excel, and SQL. The data combination form of the Excel table is shown here.

Log result:

Module execution information:

Use Case execution information:

Step info:

 

 

Flysky Log Management Platform

After using the autosky framework, we all know that the framework will form a complete log report locally after execution, and flysky also sends log information to the log management platform after execution of the automation project, the log display results of each implementation project can be displayed, so that the automation owner or test Manager can monitor the project execution and facilitate data analysis. The platform can also accept interface calls from other automation frameworks.

 

Automated software testing process and our automated testing