Automated Testing Framework Design reference principles

Automated Testing Framework Design reference principles
Guidelines for Automation framework design

Original Author: Tarun Lalwani
Translator: Jenvee, Jeff, Wally

Http://knowledgeinbox.com/articl... n-framework-design/

Date:

[E test the works of the Chinese translation team]

The translation is as follows :]

Introduction
The testing framework is a complete set of guidelines defined in all different test automation phases: Demand analysis phase, script design phase, execution phase, reporting and maintenance phase. The framework is a kind of packaging for the complex internal architecture. Such packaging can make end users easy to use. The framework also has the enforcement of process standards.
Problem description
So far, there is no guideline on how to develop a testing framework and what factors need to be considered during the development process. There are many white papers that record a wide variety of test frameworks and how they work, but none of these white papers yet record the factors that need to be taken into account in the design of the test framework. Based on the requirements of the testing framework, this article covers all aspects of the testing framework and some necessary basic elements.
1. Types of automated testing frameworks-currently, the following frameworks are common:
O data-driven framework-this testing framework can be used when the testing object process remains unchanged (only data changes. Test data is provided externally, such as Excel tables and XML files.
O keyword-driven framework-this automated testing framework provides some common keywords that apply to various types of systems. It also provides abstraction for automated testing tools and tested systems. For example, it can use the same test cases to test similar Web and Windows systems.
O Hybrid Framework-hybrid automated testing framework has the advantages of both data-driven and keyword-driven frameworks. This testing framework not only has common keywords, but also keywords based on the business logic of the tested system. For example, "login" and "exit" are keywords that can be used only for a system.
2. Do not make too many changes-the automated testing framework should try its best to make the automated testing tool use its own powerful functions, rather than redefining the entire language by implementing new keywords. Developing a keyword-driven automated testing framework is costly and time-consuming. The cost of developing a hybrid automated testing framework is relatively small and the development cycle is short.
3. reusability-the testing framework should do its utmost to improve reusability. Combining individual actions into business logic can provide reusability. For example, you can combine actions similar to "enter user name", "Enter Password", and "click Login" into a reusable module: "login"
4. Support different versions of the system-the automated testing framework should allow repeated use of baseline scripts to ensure that the script can be used to test multiple versions of the tested system. There are two ways to support different systems:
O copy and modify-This method contains a copy of the new baseline script and modifications to the project to test the specific version.
O reuse and upgrade-This method includes reusing the baseline script, providing an upgrade and Optimization for this script to test a specific version of the project. This can maximize the reusability, which is also recommended.
5. Support for script version-test scripts should be stored in version control tools such as CVS and Microsoft VSS. In this way, the disaster can be recovered.
6. Separate the development and release environments-automation should be treated like other development projects. The test script should be created and debugged in a test environment. Once the test script passes the test, you must deploy it to the release environment. This method is also applicable to urgent releases.
7. External configurability-configuration items of the script should be saved in an external document. System URLs, versions, paths, and so on can be considered as configurable items in external files. This allows you to execute test scripts in different environments. Note that the path of the external configuration file should not be written to death. If it is written to death, the script can still be run in any environment, but it can only be run in one environment at a time. You can solve this problem by using the relative path of the configuration file.
8. Self-configurability-the ideal test framework should be self-configurable. Once deployed to the system, no manual configuration is required. The script should be automatically configured to complete some necessary settings.
9. changes caused by any object changes should be minimal-the most common problem in the automation process is changes to object recognition. The test framework should support making these modifications easily. This can be achieved by storing all object recognition settings in a shared file. This file can be an XML file, an Excel file, a database, or an automation-specific format. There are two possible ways to set object recognition:
O static method-in this method, all object definitions are initially loaded into the memory during the test. Any object definition change can only be achieved by stopping and re-running the test.
O dynamic method-object definition is driven by requirements. This method is relatively slow compared with the static method. However, for very large scripts, and object recognition needs to be corrected at runtime, dynamic methods are applicable.
10. test execution-the automated test framework may need to meet the following requirements (based on actual needs)
O Perform separate test cases;
O Perform test cases in batches (a group of test cases );
O only test cases that fail to be executed;
O you can execute the next (Group) test case based on the execution results of the previous (Group) test case;
There are many other situations based on actual needs. A framework may not be able to meet all of these needs, but it should be flexible enough to meet such needs in the future.
11. Status Monitoring-a framework should allow real-time monitoring of the execution status and trigger an alarm upon failure. In this way, feedback can be quickly provided after failure occurs.
12. Report-different systems have different requirements for reports. Sometimes a group of test reports is required, and sometimes only a test execution report at the test case level is required. A good testing framework should be flexible enough to support different reports as needed.
13. Minimal dependencies on the test tool when changes occur-some test script changes may only be made in the test tool after the test tool is opened, and then saved. The test script can be changed without a test tool. This can reduce the purchase of license and save money for the company. This implementation also allows all users who want to modify the script to easily modify the script without installing the test tool.
14. convenient debugging-debugging takes a lot of time in the automation process, so special attention is needed in the debugging process. Keyword-Driven Testing Framework is difficult to debug because external data sources (such as Excel Data Tables) are used to read keywords and test processes in the script.
15. log generation is an important part of execution. It is very important to generate debugging information at different points in a test case. This information helps you quickly locate the scope of the problem and shorten the modification time.
16. ease of use-the Framework should be easy to learn and use. Training of framework personnel is time-consuming and expensive. A framework with good documentation is easier to understand and use.
17. Flexibility-the framework should be flexible enough to adapt to any improvement without affecting existing test cases.
18. performance impact-the framework should also consider the impact on execution performance. A complex framework increases the loading or execution time of scripts, which is definitely not what we expected. Like caching technology, when executed, compile all the code to a single library medium... as long as it should be used for performance improvement.
19. Framework Support Tools-develop some external tools to complete the task, which will be helpful for the framework design. Here are some examples:
O upload scripts from local folders to QC
O combine the library files to the currently opened scripts
O synchronize local and QC script files
20. encoding standards-encoding standards should ensure consistency, readability and ease of maintenance of scripts. The encoding standard should include the following:
O naming conventions (variables, subprograms, functions, file names, script names, etc.). For example, I _VarName is an integer variable, and fn_ I _FuncName is a function whose return value is an integer;
Comments to the headers of libraries, subprograms, and functions. This should include, such as version history, creator, last modifier, last revised date, description, parameters, and examples;
O object naming rules. For example, txt_FieldName is a text box;

Summary
We should regard automated testing as a development project, not just recording and playing back events. There is a good framework first, and then automated testing can ensure a low maintenance cost. When developing a framework to analyze the requirements of the framework, you can refer to the principles mentioned in this article.

References
1. Definition http://en.wikipedia.org/wiki/Framework of "Framework"
2. select automated testing framework http://www.ibm.com/developerworks/rational/library/591.html
3. Automated Testing Framework http://safsdev.sourceforge.net/F... ationFrameworks.htm

[Original]

Guidelines for Automation framework design
Original URL: http://knowledgeinbox.com/articl... n-framework-design/
Author: Tarun Lalwani

Introduction
A Framework defines a set of guidelines for all phases of test automation: Requirement Analysis, Script Design, Execution, Reporting and maintenance. A framework can be a wrapper around some complex internal architecture which makes it easy to use for the end user. it also enforces a set of standards for implementation

Problem Statement
There is no standard set of guidelines available on developing a framework and what all considerations need to be taken during the development of the same. there are different white papers which go over types of framework and how they work. but none of them defines what all factors go in to the design of the same
Design guidelines
This paper covers different aspect of a framework and key features it needs to have based on the requirements.
1. Selection of a framework-Different types of frameworks that exist are:
O data driven framework-used when flow of the application remains constant, only the data changes. The data is provided by external medium e.g.-Excel sheet, XML etc...
O Keyword Driven framework-this framework provides generic keywords that can be used with any type of application. it also provides procedure action from the type of automation tool used and type of being application tested, e.g. -It can test a similar web and Windows application with the same test case
O Hybrid Framework-a hybrid framework is the one which takes advantages from both data driven and Keyword Driven frameworks. these frameworks do not implement generic keywords but implement business logic keywords Based on the application being tested. for ex-login, logout cocould be application specific keyword that can be used.
2. don't reinvent the wheel-A framework shocould try and use the power of the automation tool rather than re-defining the whole language by implementing new keywords. developing a keyword driven framework is time consuming and costly. A Hybrid framework can be developed in a shorter time period and with less cost.
3. reusability-The framework shocould allow highest possible reusability. combining individual actions into business logic provides re-usability. e. g. -Combing actions like "Enter username", "Enter password" and "Click Login" into one re-usable component "Login"
4. support of different application versions-A framework shocould allow re-use of baselines scripts in case different versions/flavors of an applications are to be tested. there are two different ways to support different applications
O Copy and Modify-This method involves creating copies of the baseline scripts and modifying them for a specific application version
O Re-use and Upgrade-This method involves re-using baseline script and providing a upgrade code for specific version of application. This ensures maximum re-usability and shocould be preferred.
5. Support of script versioning-Scripts shoshould be stored in a version control system like CVS, Microsoft VSS etc... This ensures recovery from any disaster.
6. different environment for development and production-Automation shocould be considered as any other development project. test scripts shoshould be created and debugged in Test environment. once tested then only shocould be deployed to the production environment. this holds true for any emergency releases also
7. Externally retriable-retriable items of a script shocould be kept in an external file. This wocould contain configuration like Application URL, version, path etc... This allows running the same script against different environment. ensure that location of the configuration file is not hard coded. hard coded files wocould allow running on any environment but only one at a time. keeping the configuration relative to current test path allows overcoming this limitation
8. Self configurable-Ideally a framework shocould be self configurable. Once deployed to a system, no manual configuration changes shocould be required and scripts shocould automatically configure the required settings
9. minimal changes required for any object changes-most common issues faced during automation are object identification changes. framework shocould be able to patch such changes easily. this can be achieved by storing all object identification settings at a shared location. this cocould be an external XML file, Excel file, database or automation proprietary format. there are two possible way to load this object identification Configuration
O static-in this all the object definitions are loaded into the memory at the start of the test. Any changes made to object definition can only be loaded by stopping and re-running the test
O dynamic-object definition is pulled as per request. this approach is a bit slow as compared to the static one. but in case of huge scripts where the fix needs to be made at run-time this is suitable.
10. Execution-framework might need to cater to below requirements (on need bases)
O Execution of a individual test case
O Execution of a test batch (combination of tests)
O Re-execution of only failed test cases
O Execution of a test case/test batch based on result of another test case/test batch
There cocould be your other needs based on the project requirement. A framework might not implement all of them, but shocould be flexible enough to accommodate such requirements in future
11. Status monitoring-A framework shocould allow monitoring the execution status in real time and shocould be capable of sending alerts in case of failure. This ensures quick turound time in event of a failure
12. reporting-Different applications have different reporting needs. some require combined results for a test batch and some require individual level test report for each test case in test batch. the framework shoshould be flexible enough to generate required reports
13. minimum dependency on Automation tool for changes-Some fixes can only be made by opening the script in the automation tool and then saving it. scripts shoshould be developed in such a way that modification is possible even without the unavailability of the automation tool. this deflates company cost by matching the number of licenses required. it also allows anyone to make changes to the script without having the need to setup the tool
14. easy debugging-Debugging takes a lot of time during automation and hence special care needs to be taken for this part. keyword driven frameworks which use external data source (like a excel spread sheet) to read scripts keywords and process the same are difficult to debug.
15. logging-Log generation is important part of execution. it is very important to generate debug information at various points in a test case. this information can help find problem area quickly and reduce the time to make a fix at the same time
16. Easy to Use-The framework shocould be easy to learn and use. It is time consuming and costly to train a resource on a framework. A well known ented framework is easier to understand and implement
17. Flexible-Framework shocould be flexible enough to accommodate any enhancements without impacting existing test cases
18. performance impacts-A framework shoshould also consider the performance impacts of the implementation. A complex framework which increases the load time or execution time of scripts is never desirable. techniques like caching, compiling all code into single library while execution etc... Shocould be used to improve performance whenever possible
19. Framework Support Tools-External Tools can be developed to perform tasks that help in framework design. Some example tasks wocould be
O Uploading scripts from local folder to HP Quality Center
O Associating library files to currently open scripts
O Synchronizing local files with HP Quality Center.
20. Coding Standards-Coding standards ensures scripts that are consistent, readable and easily maintainable. Coding standard shocould define all the below listed things
O Naming convention for variables, subs, functions, file names, script names etc... Ex-I _VarName for interger, fn_ I _FuncName for function returning interger
O Library, subs, functions comment header. This shocould include information like version history, created by, last modified by, last modified date, description, parameters, example
O Object naming conventions. Ex-txt_FieldName for a text box
Summary
Automation shocould be considered as a development project and not just record and playback of events. starting automated testing with a good framework ensures low maintenance. guidelines discussed in this paper can be used as input for developing requirements for a framework.

References
1. Framework, http://en.wikipedia.org/wiki/Framework
2. Choosing a test automation framework, www.ibm.com/?works/rational/library/591.html
3. Test Automation frameworks, http://safsdev.sourceforge.net/F... ationFrameworks.htm

Copyright: This article is an original work of the E-test translation team. E testing of China (www.5etesting.com) and related content providers
5etesting.com All Rights Reserved. No one or organization may copy or,
Reprinted or mirrored; otherwise, legal liability will be held.

This article from the CSDN blog, reproduced please indicate the source: http://blog.csdn.net/etesting/archive/2009/10/28/4735910.aspx