Cross-platform automated test deployment

 

This article from: http://www.ibm.com/developerworks/cn/opensource/os-cn-stafxml/

 

At present, our testing requires more and more platforms, and the testing scale is as large as dozens of machines. To improve the test efficiency and quality, we designed and implemented a test framework based on staf (software test automation framework) and XML. Use XML as the configuration file to configure all platform information; Use staf as the engine to implement cross-platform automated testing; Separate the test framework from the test cases to implement plug-and-play of test cases. This article is highly practical. You can refer to the design of automated testing across multiple platforms.

Preface

At present, our testing needs to cover more and more platforms and the testing scale of dozens of machines. To improve the test efficiency and quality, we designed and implemented an Automatic Test Tool Based on staf and XML, which mainly solved the following problems:

1. Apply staf internal services and xml configuration files to automate testing on dozens of machines;
2. Calls staf internal services directly in Perl scripts to provide more flexible and powerful functions;
3. Separate the test framework from the test cases. Each time we generate a new test case, we only need to configure it in the XML file to implement plug-and-play of the test case.
4. Generate log files in HTML format to facilitate query by testers.

Our testing work includes the following three components: Automated Testing Framework, configuration files, and test cases. Shows the structure.

Figure 1: Structure of the automated testing framework

Back to Top

Automated Testing Framework

Improvements to the staf framework

The automated testing framework is implemented based on staf and XML. Staf/Stax is an automated testing environment developed by IBM. staf is an open-source, cross-platform, and multi-language automated testing framework. It establishes the foundation for automated testing and accelerates the process of automated testing. Stax is an execution engine based on staf. Based on staf, it helps you to distribute, arrange, execute, and analyze test cases. In the testing work, more and more testers are using staf with Stax. However, Stax itself is an XML document and must be embedded in the Python language, which features complicated procedures and inconvenient operations. To achieve comprehensive automated testing across multiple platforms and improve the efficiency of automated testing, we decided to directly call the staf service in the Perl script to implement the key remote process call and file transfer functions, configure the XML file management test platform and test cases.

Staf services involved in the testing framework

Staf builds an automation framework based on reusable components. These reusable components are services. The services used in this test framework are as follows:

• Program call service: an internal service that can be used to remotely call an external program. In this framework, the staf process service is used to remotely execute Perl scripts. For example, the syntax for calling on Windows is:

STAF “hostname” Process start command “cmd /C perl xxxxxxx.pl”

 

• File System Service: an internal service that allows you to operate a file system, such as copying, deleting, and viewing a file system. In this framework, the staf File System Service is used to deploy test cases to all test platforms. After the execution, the log files are returned to specific machines for inspection. For example, the calling syntax on Windows is:

STAF local FS COPY DIRECTORY $fromDir TODIRECTORY $toDir TOMACHINE $hostname EXT $ext

 

$ Fromdir refers to the local folder, $ todir refers to the remote folder, $ hostname refers to the remote test platform, and $ ext refers to the file extension to be copied, which is "PL" here ".

 

Back to Top

Configuration File

Script Execution commands and environment variables are different on different platforms and vary greatly. To avoid hard coding these differences to automated scripts and make them transparent, we use XML-based configuration files to cover the differences among multiple platforms. For example, the configuration file in this framework is as follows.

Code 1: configuration file content example

<Monitor name="hostname" exeCmd="b:/Scripts/executeCommand.pl" / log="b:/Scripts/ logfile/ " configure="b:/Scripts/configure.xml">

 

In the configuration file, specify the script to be executed, the location where the log file is stored, the location where the configuration file is stored, the operating system type of the test machine, and the specific path of the script to be deployed. When the script is executed, it first reads important information from the XML file, such as the platform of the testing machine, so as to use the corresponding execution command and query the required environment. The sample code is as follows.

Code 2: Read XML sample code

use XML::Parser;use XML::SimpleObject;$file="configure.xml";$parser = XML::Parser->new(ErrorContext=>2,Style=>"Tree");$xso = XML::SimpleObject->new($parser->parsefile($file));$monitor = $xso->child('Monitor');$exeCmdFile = $monitor->attribute('exeCmd');$log = $monitor->attribute('log');$configureFile = $monitor->attribute('configure');chomp($exeCmdFile);chomp($log);chomp($configureFile);

 

Traverse all test platforms and query related information. If the platform is a Windows platform, the staf variable is queried, the system drive value is obtained, and the path under the test platform is constructed, this path is the storage location of the test case to be executed. The sample code is as follows.

Code 3: Sample Code for traversing all test platforms

foreach $host($monitor->children('host')){ my $exeCmd; my $hostname = $host->attribute('name'); my $os = $host->attribute('os'); my $dir = $host->attribute('dir'); chomp($os); chomp($hostname); chomp($dir); if($os =~ /win/i){ my @systemDrive = `staf $hostname VAR GET var STAF/Env/SystemDrive`; chomp($systemDrive[2]); $dir = "$systemDrive[2]//automation//"; }}

Back to Top

Deploy and execute test cases

In this automated test tool, we separate the test framework from the test cases. Every time we generate a new test case, we only need to add it to a specific location and perform simple configuration in the configuration file. When the test starts, the test tool deploys the test case to the test platform based on the information in the configuration file, sets the required environment variables, and starts the execution engine on the test machine, the execution engine executes the use case and returns the log file.

Deploy Test Cases

First, use the file system service provided by staf to deploy the test cases and configuration files to all test platforms. Each test case is a script written in Perl. The reason for deploying the configuration file to the test machine is that the execution engine on the test machine executes the test case according to the test case information in the configuration file. The sample code is as follows.

Code 4: Sample Code for deploying Test Cases

my $ext = "/"pl/"";$toDir =~ s/[////]$//;$fromDir =~ s/[////]/w+/./w+$//;my $cmd = "staf local fs copy DIRECTORY $fromDir TODIRECTORY / $toDir TOMACHINE $hostname EXT $ext";open (EXEC, “$cmd | ”) or die “can not deploy test cases onto test beds”;while(<EXEC>){ if($_ =~ /error/i){ print “execute command error/n”; #todo }}close EXEC;print “successfully deploy test cases onto test beds, next step, / execute them on test beds automatically/n”;

 

Execute test cases on the test platform

After successfully deploying the test case to the test platform, you need to execute the test case. Each test platform has an execution engine. We use staf's remote call service to call the execution engine in the test framework. The execution engine is responsible for executing test cases on each test platform. The execution engine is also a Perl script and is responsible for the following functions:

1. Initialize and obtain the name of the test case;
2. Execute test cases in sequence;
3. Generate log files in HTML format to facilitate query by testers.

Code 5: Initialize and obtain the test case name

sub initialize{while($option = shift @ARGV){if($option =~ /.html/){$logFileName = $option;}elsif($option =~ /[////]/){$dir = $option;}else{$hostname = $option;} }print "dir is $dir/n"; print "hostname is $hostname/n";print "logFileName is $logFileName/n";$logFileExt = ".html";$dateAndTime = &getCurrentDateAndTime();#Analyze the xml configure file, register all test cases to be executed$file = "$dir"."hostInfo_$hostname.xml";$confFile = "$dir"."configure.xml";$parser = XML::Parser->new(ErrorContext=>2,Style=>"Tree");$xso = XML::SimpleObject->new($parser->parsefile($file));$confxso = XML::SimpleObject->new($parser->parsefile($confFile));$count = 0;foreach $tmp_host($confxso->child('Monitor')->children('host')){if($tmp_host->attribute('name') =~ /$hostname/i){foreach $testcase($tmp_host->children('testcase')){$testcases[$count++] = $testcase->attribute('value');} }}}

Code 6: Sample Code for executing test cases in sequence

sub executeCommand{ $count = 0;foreach $testcase(@testcases){ $testcase = $testcases[$count++];$testcase =~ s//s+/_/g; $testcase .=".pl $hostname $dir"."function.pl $logFileName";$testcase = $dir.$testcase; print $testcase; `perl $testcase`;}}

 

Generate logs

Generate HTML logs based on the test case execution. The sample code is as follows.

Code 7: Generate logs in HTML Format

#initial html headsub initLog{my ($logFileName,$hostname) = @_;my $sdq = "/"";&appendLogScriptInfo($logFileName, / "<HTML><HEAD><TITLE>Automation Results</TITLE></HEAD>");&appendLogScriptInfo($logFileName,"<ALIGN=left");&appendLogScriptInfo($logFileName,"<FONT FACE=$sdq"."arial$sdq>");&appendLogScriptInfo($logFileName, / "<B>Automation Results for host: $hostname ".&getCurrentDateAndTime."</B><BR>");&appendLogScriptInfo($logFileName,"</FONT></ALIGN>");&appendLogScriptInfo($logFileName,"<TABLE BORDER");&appendLogScriptInfo($logFileName,"<BR><BR>");}#end html after testsub endLog{my ($logFileName) = @_;&appendLogScriptInfo($logFileName,"<BR><BR>");&appendLogScriptInfo($logFileName,"</TABLE BORDER>");&appendLogScriptInfo($logFileName,"<BR><BR><BR><BR>");&appendLogScriptInfo($logFileName,"<BR><BR><BR><BR>");&appendLogScriptInfo($logFileName,"</HTML>");}#append log contents into html filesub appendLogScriptInfo{my ($logFile, $content) = @_;unless(open(LOGFILE,">>$logFile")){die("can not open log file $logFile");}print LOGFILE ("$content/n");close(LOGFILE);}

Back to Top

Log query

After the test, the staf File System Service will return the log files generated on the test machine to the monitoring machine and deploy them to the Web container, the tester can view the execution of the script through Web browsers such as IE. The following figure shows an example.

Figure 3: each link corresponds to a test platform

Figure 4: each link points to the list of test cases executed by the Platform

Figure 5: each test case points to the specific execution status