Test automation and continuous integration using Staf/stax

About regression and incremental testing

Regression testing is often used to ensure that software changes do not introduce new problems or failures into the software. Alternatively, you can use it to ensure that you do not reintroduce errors that were previously fixed. If software has many dependencies on Third-party components and libraries, such as XML processing APIs or database extraction APIs such as JPA, the introduction of new problems is very common. In these cases, a small change in the database schema can result in unpredictable application behavior changes. If the software itself is complex, a small change to common code snippets can incur unpredictable side effects.

This increases the probability of introducing problems during maintenance or development when there is a need to provide continuous support for previous versions of a middleware software or for different platforms. A patch or feature that works in a database middleware version can have disastrous consequences in different versions. And when adding functionality, only APIs that are available for newer versions of the middleware may cause damage to clients that use previous versions. When supporting different platforms, perform a full test of the functionality that depends on the command-line interface or script. Otherwise, different behaviors may appear in different operating systems, such as Windows and Linux.

Regression testing is for the overall software and considers the interdependencies of the software with other components or applications. During development, focus on the specific modules or features that you are developing. You can test them incrementally when implementing modules or functionality, and sometimes you need to use stubs to reference other features that have not yet been implemented. If incremental tests are well structured and self-contained enough that you can repeat them in a given environment (such as the necessary middleware software, machines, or configurations), you can reuse them in future regression tests. This structure and its independence reduces rework and improves test quality because each piece of code is reviewed and tested in detail at development time.

When you test development functionality on a set of schemas, operating systems, and middleware versions, you can consider using automated incremental testing. Take an example of a command-line interface that is supported by Linux, AIX, Windows 7, and Windows XP. During development, it may take a lot of resources and time to perform tests on each operating system in these operating systems. In some cases, the time required to test the new feature exceeds the time it takes to develop it. This situation becomes even more cumbersome, given that many features are developed by different developers at the same time in a project. To complicate this pattern, the number of machines used for testing is usually limited, even with virtual machines. Testing typically requires a pre-configured environment, including database servers, application servers, and so on. The more machines dedicated to testing, the more time it takes to update and configure them. Because automated incremental testing is run by a tool in a set of environments or a test bed, you can save more resources by saving the time it takes to run tests manually in each environment.

Automated regression testing can use the same test bed as the incremental test. For example, after you set the regression test to run once a day, you do not need to consume resources that would normally be used for development and incremental testing. You can set up a lengthy regression test to run at a longer interval, such as running on a weekend.

Staf/stax Framework

The Software Test automation framework (STAF) is an open source framework that contains a set of built-in services specifically designed to build automated solutions. These services are reusable components that provide all the functionality in STAF. Each STAF service provides a specific set of features, such as:

Log records

File system operations

Security and process calls

Definition of a set of requests

Other services, such as the e-mail service, the FTP service, and the timer service, are not built in, but can be easily inserted into the framework.

The STAX service is a special service that is implemented in a Java environment or supports an execution engine that invokes other STAF services through XML. It also provides the following support:

Parallel execution

User-defined granularity of execution control

Support for nested test cases

Ability to control the length of execution

Ability to import modules at run time

Support for existing Python and Java modules and packages