The author of the robot framework suggests how to select an automated testing framework.

Original article: http://www.infoq.com/cn/news/2012/06/robot-author-suggest-autotest

Automated software testing, as an alternative to manual testing, has received increasing attention. Pekka KL ärck, as the creator and core developer of the robot framework, describes the differences between several automated testing methods based on the system level.

I. record playback is popular in commercial tools and can be quickly used without programming skills. However, this method is relatively fragile, and testing will be affected once the UI changes, scattered scripts cannot be reused and difficult to maintain, and the system must be available before testing (that is, the A-TDD method cannot be used ). Therefore, this method is not suitable for large-scale automated testing.

2. Linear scripts allow you to write unstructured scripts in various languages. The scripts directly interact with the tested system. Quick Start and high flexibility. However, writing scripts requires programming skills. A change in the system will affect all scripts and it is difficult to maintain a large number of scripts without modularization or reuse. Therefore, this method is suitable for simple tasks and not for large-scale automation.

3. the modular script consists of two parts: the driver script executes the test, and the test library function interacts with the tested system. The driver script is very easy to write, so that new tests can be created more quickly and easy to maintain. However, it takes time and programming skills to build a test database and embed test data into the script. To create a new test, a new test script is required. Therefore, as long as you have programming skills, this method is suitable for large projects, but not for non-programmers.

4. data-driven method: separates data from test scripts. Based on a modular Test Library, a single driver script can execute multiple similar tests, which makes it easy to create new tests. Maintenance work can be separated, testers are responsible for data, and programmers are responsible for writing test libraries. However, new driver scripts are still required for different types of tests, and it takes effort to initially build a data parser and reuse components. This method is suitable for large projects and requires less programming skills.

5. keyword-driven, combining data with keywords to describe how to use data for testing. This method has the advantage of data-driven, while non-programmers can also create new types of tests. All tests are executed by the same framework without different driver scripts. However, the initial cost is very high, but open-source solutions can be used! Therefore, it is very suitable for large projects.

Pekka's introduction to the above five methods is also an introduction to the development history of automated testing, and also reflects the design ideas behind the robotframework.

In addition to the selection of the testing framework, you must pay attention to other aspects in order to do a good job of automated testing.

Automated Testing requires attention to testability. The most difficult part of automation is interaction with the tested system, especially the GUI layer. Ensure that the system is easy to be tested, such as adding identifiers to Gui elements, outputting easy-to-Parse text, and providing automated interfaces.

Generally, the system can be divided into the GUI layer and the business layer under the GUI. Gui-layer tests require calls of the same interface as normal users. However, some gui technologies lack good tool support, which may make the tests vulnerable and perform relatively slowly. Testing from the business layer is relatively easy and execution is fast. However, the GUI layer still needs to be tested to ensure that the GUI is correctly connected to the business layer, and sometimes the GUI layer also has business functions. Pekka recommends that you fully test the business layer and partially perform end-to-end tests on the GUI layer. Not all systems have GUI layers, but may have APIs, databases, servers, and command lines. The automated testing framework can call different drivers for testing. These non-Gui layers are relatively easy to test, as long as the test case is regarded as another client.

So what stage should automated testing be conducted? If the development is automated independently after completion, this is a typical waterfall process. communication barriers exist between different teams, the feedback cycle is slow, and the product is hard to obtain testability in the future, this leads to complex and vulnerable testing solutions. On the contrary, programmers and testers collaborate to automate typical agile processes. Taking automation as part of team development, testability is no longer a problem. When the team makes technical decisions, it can consider testability and tool selection. programmers can join in advance to provide testability hook features.

Automated Testing requires version control and continuous integration. Put the test and code together and manage the test script like managing the code. There are so many available tools, such as SVN, git, and mercurial. No reason is needed. Continuous integration is the key to comprehensive automation. When the test or code changes, the test is executed immediately. If the test runs for a long time, you can run it regularly. You can use Jenkins, Hudson, cruise control, and buildbot to write a scheduled script or cron job on your own.

Choose commercial automation tools or open-source tools? Good stuff is definitely expensive, but expensive is not necessarily good, and cheap licenses will also prevent the entire team from cooperating. In addition, commercial tools are difficult to integrate and customize with other automated tools (especially those from other vendors) or version control and continuous integration. In addition, product termination or Company shutdown are potential risks. There is a lot of room to choose from open-source tools. Open-source tools are usually easy to integrate with other tools. The key is that they are free of charge and can be used and customized at will, and will never disappear. As for free software, there are fewer and fewer free software. Many free software are open-source. Free software cannot be customized, and there is a risk of suspension.

What skills are required for automation? In general, including python, Ruby, Perl, JavaScript, regular expressions, XPath and CSS positioning, SQL statements, version control, and so on.

With automation, is manual testing required? Of course !! However, to avoid manually executing a script test, we recommend that you fully automate it. Testers can focus more on exploratory testing. Remember, machines are good at regression testing and humans are good at finding bugs.