Analysis on official PHP automated testing methods

The following describes the official PHP automated testing method. The following describes how to analyze the source code of PHP's latest official version 5.2.8 on the LINUX system platform. First, let's take a look at PHP's automated test script PHPT script. Automated TEST script PHPT example php test script is suffixed with ". phpt", including TEST and FILE

The following describes the official PHP automated testing method. The following describes how to analyze the source code of the latest official PHP version 5.2.8 on the LINUX system platform. First, let's take a look at PHP's automated test script PHPT script. Automated TEST script PHPT example php test script is suffixed with ". phpt", including TEST and FILE

PHP Official Automation Test MethodBrief Introduction

The following describes the latest version of PHP.OfficialThe source code of version 5.2.8 is analyzed on the LINUX system platform as an example. First, let's take a look at the PHPAutomationTestScript PHPT script.

Automation TestScript PHPT example

PHPTestThe script is a FILE with the suffix ". phpt" and contains multiple paragraphs, such as TEST, FILE, and keep CT. It is short for PHPT. In each section, TEST, FILE, and CT are the basic sections.TestThe script must contain at least three paragraphs. The TEST segment can be used to enterTestThe name of the use case. The FILE segment is implemented by a PHP script.TestUse case; the CT segment isTestThe expected value of the use case.TestWhen the use case is running, PHP will be usedTestPHP executable object to runTestUse Cases to compare with actual resultsTestThe expected values listed in the reverse CT segment in the use case. If the actual results are consistent with the expected valuesTestIf they are inconsistentTestFailed.

Table 1 lists common section names and descriptions of their corresponding filling content.

Table 1. PHP TestSection description in the script

Paragraph name	Filling content	Remarks
TEST	TestCase name	Required Section
ARGS	Input parameters of the FILE segment	Optional paragraphs
SKIPIF	Skip thisTestConditions	Optional paragraphs
POST	InputTestScript POST variable	Optional paragraphs. If the POST segment is used, we recommend that you use the SKIPIF segment, for example: -- SKIPIF --
GET	InputTestGET variable of the script	Optional paragraphs. If the POST segment is used, we recommend that you use the SKIPIF segment, for example: -- SKIPIF --
INI	ApplyTestScript ini settings	Optional paragraphs. For example, foo = bar. The value can be obtained through the ini_get (string name_entry) function.
FILE	TestScript statement	Required section. A script statement written in PHP. The execution result will be compared with the expected value of the CT * segment.
Secondary CT	TestScript wait Value	Required Section
EXPECTF	TestThe expected value of the script. The expected value can be expressed in the format of the sscanf () function.	Variant of the CT segment
EXPECTREGEX	TestThe expected value of the script. The expected value can be expressed in a regular expression.	Variant of the CT segment

ToOfficialBuilt-inTestThe script "001. phpt" is used as an example (see Listing 1). From the content of the TEST section, this is a verification of the PHP version.TestUse Cases. The content of the SKIPIF segment is written in the "skipip. inc" file. In the FILE segment,TestThe use case prints out the PHP version set in the environment variable TEST_PHP_EXECUTABLE. This result is in the same format as the string in EXPECTF.

Listing 1. PHPT TestExample of script "001. phpt"

--TEST--  version string  --SKIPIF--  
   --FILE--  
   --EXPECTF--  string(%d) "PHP %s (cli) (built: %s)%s  Copyright (c) 1997-20%d The PHP Group  Zend Engine v%s, Copyright (c) 1998-20%d Zend Technologies  "  Done

How to run PHP Automation TestScript

RunningTestBefore you proceed, you must first beTestPHP source code is compiled into executable objects.

Then you need to import several environment variables. Table 2 describes how to set the main environment variables.

Table 2. PHP Automation TestEnvironment variable settings in

Environment variable name	Environment variable value	Example
TEST_PHP_EXECUTABLE	SetTestObject PHP, or "auto ". When "auto" is set, if it is CGI mode, it is ". /sapi/cgi/php-cgi "; if it is in CLI mode, it is". /sapi/cli/php ".	TEST_PHP_EXECUTABLE = $ HOME/php-5.2.8/sapi/cli/php
TEST_PHP_DETAILED	Set whether detailed log output is required. Set the value to 1 or 0.	TEST_PHP_DETAILED = 1
TEST_PHP_USER	Set whether special user directories are required.	TEST_PHP_DETAILED = "/usr/test1"
TEST_PHP_LOG_FORMAT	Set the log format. Set the value to a subset of the "LEOD" substring. L representsTestGenerate the ". log" file. E Indicates ". exp", O indicates ". out", and D indicates ". diff ".	TEST_PHP_LOG_FORMAT = "LD"

In this example, set the environment variables in the Bash environment as follows:

List 2. Example of setting Environment Variables

export HOME=/home/user_dir/  export TEST_PHP_EXECUTABLE=$HOME/php-5.2.8/sapi/cli/php  export TEST_PHP_DETAILED=1  export TEST_PHP_LOG_FORMAT="LEOD"

After this settingTestThe PHP executable object is the "php" executable file compiled under the directory "$ HOME/php-5.2.8/sapi/cli.

RunTestBefore, you also needTestThe script PHPT is edited and saved as the ". phpt" file. Here we useOfficialBuilt-inTestFor example, run the script 001. phpt as follows:

Listing 3. PHPT TestSummary report example

bash-2.03$ cd $HOME/php-5.2.8/  bash-2.03$ $HOME/php-5.2.8/sapi/cli/php run-tests.php  \  $HOME/php-5.2.8/sapi/cli/tests/001.phpt

IfTestThe actual output of the use case is the same as the expected value.TestThe result is as follows:

Listing 4. PHPT TestSummary report example

=====================================================================  CWD         : /home/user_dir/php-5.2.8/sapi/cli/php  PHP         : /home/user_dir/php-5.2.8/sapi/cli/php  PHP_SAPI    : cli  PHP_VERSION : 5.2.8  ZEND_VERSION: 2.1.0  PHP_OS      : Linux rhas05 2.6.9-55.ELhugemem #1                        SMP Fri Apr 20 17:20:11 EDT 2007 i686 i686 i386 GNU/Linux  INI actual  :  More .INIs  :  Extra dirs  :  =====================================================================  Running selected tests.  PASS Test version string [001.phpt]  =====================================================================  Number of tests :    1                 1  Tests skipped   :    0 (  0.0%) --------  Tests warned    :    0 (  0.0%) (  0.0%)  Tests failed    :    0 (  0.0%) (  0.0%)  Tests passed    :    1 (100.0%) (100.0%)  ---------------------------------------------------------------------  Time taken      :    0 seconds  =====================================================================

IfTestIf a failure occurs, in addition to the screen output failure result, several files are generated in the current running directory for analysis.TestCause of failure. The types of generated log files are set by the Environment Variable TEST_PHP_LOG_FORMAT. For details, see table 2. If "LEOD" is set, the generated log file includes the five types of files listed in table 3.

Table 3. PHP Automation TestOutput script file

Log File Name	Log File Content	Settings in TEST_PHP_LOG_FORMAT
001. out	RunTestThe actual output result after the statement.	O
001. exp	The expected result in the script, that isTestThe content of the reverse CT * segment in the script.	E
001. log	The actual output result and the expected result in the script, that is, a collection of ". exp" and ". out.	L
001. diff	The actual output result and the expected result in the script are compared using the diff command.	D
001. php	Actually executed PHPTestStatement inTestParsingTestThe FILE segment of the script is obtained.	Always generate

In the preceding example, only one script is run at a time. PHP also supports manyTestRun the script together. Table 4 lists the three types of PHP supportedTestMethod.

Table 4. PHP Automation TestMethod

TestMethod	Parameters	Example
Run only oneTestUse Cases	TestScript Name	001. phpt 002. phpt
ExecuteTestUse Cases	TestDirectory Name of the use case	Test_dir/
ExecuteTestUse Cases	-R plus listTestCase file name	-R record_file

Back to Top

PHP Automation TestPrinciples and implementation of the framework

In the running example in listing 3, the actual execution statement is "$ HOME/php-5.2.8/sapi/cli/php run-tests.php $ HOME/php-5.2.8/sapi/cli/tests/001. phpt ". Among them, "001. phpt" isTestScript; run-tests.php is PHPTestThe driver script of isOfficialThe $ HOME/php-5.2.8/sapi/cli/php is the PHP executable object that runs the driver script. Note that the PHP executable object set in the environment variable TEST_PHP_EXECUTABLE andTestThe '$ HOME/php-5.2.8/sapi/cli/php' in the script points to the same executable object, but its meaning is different. The PHP set in the environment variable TEST_PHP_EXECUTABLE isTestObject, and runTest$ HOME/php-5.2.8/sapi/cli/php is used only to driveTestRun the script.

As shown in figure 1, PHPTestThe relationship between two PHP executable objects and two PHP scripts. InTestFirst, run the script "php-5.2.8" (step 1) by "$ HOME/run-tests.php/sapi/cli/php ). In the run-tests.php script, the environment variable value "TEST_PHP_EXECUTABLE = $ HOME/php-5.2.8/sapi/cli/php" will be resolved (step 1 ). Here, the PHP executable object isTestPHP. In the run-tests.php scriptTestScript "001. the FILE segment in phpt is parsed into the PHP script "001.php" (step 1), and the PHP executable object set by "TEST_PHP_EXECUTABLE" is used to execute "001.php" (step 2) and get the actual output results. After comparing the actual output results with the expected results of the CT segment, the outputTestResult (step 1 ). The above running process can be obtained from the analysis of the "run-tests.php" script.

Figure 1. PHP TestRunning Relationship Diagram

Through the analysis of the above examples, we can see that PHPOfficialTestOfAutomationMainly dependent on the run-tests.php script andTestUse Case script PHPT. The "run-tests.php" script is PHPTestAutomationFramework builder. From the code list 5, we can see that PHPAutomationTestThe main work of the framework.

Listing 5. PHP Automation TestCode snippet

$test_cnt = count($test_files);  if ($test_cnt) {  putenv('NO_INTERACTION=1');  verify_config();  write_information($html_output);  usort($test_files, "test_sort");  $start_time = time();  if (!$html_output) {  echo "Running selected tests.\n";  } else {  show_start($start_time);  }  $test_idx = 0;  run_all_tests($test_files, $environment);  $end_time = time();  if ($html_output) {  show_end($end_time);  }  if ($failed_tests_file) {  fclose($failed_tests_file);  }  if (count($test_files) || count($test_results)) {  compute_summary();  if ($html_output) {  fwrite($html_file, "
 \n" . get_summary(false, true));  }  echo "=================================================";  echo get_summary(false, false);  }  if ($html_output) {  fclose($html_file);  }  if ($output_file != '' && $just_save_results) {  save_or_mail_results();  }         if (getenv('REPORT_EXIT_STATUS') == 1 and preg_match('/FAILED(?: |$)/',   \                     implode(' ', $test_results))) {  exit(1);  }

InTestBefore, thisAutomationTestThe framework is valid according to the user-specifiedTestDetermine the number of filesTestUse Case, and write it into the test_cnt variable for counting. The verify_config () function is used to verify and prepare the settings and ini settings of various environment variables. The write_information () function will output thisTestEnvironment Information, includingTestPHP version, Extention used, ZEND version, INI settings,TestThe system version andTestTime record.

TestUse cases occur in the run_test () function called by the function run_all_tests. Run_test () will be parsedTestThe meaning of each section in the script.TestThe record and settings will be cleanTestAfter the environment is ready, prepare various intermediate files and log files, and run the actual PHP executable object specified by the Environment Variable TEST_PHP_EXECUTABLETestStatement. Finally, the running result andTestCompare the expected values in the script. If the comparison fails, the results will be recorded in the log files set by the user one by one.

In allTestThe variable end_time is recordedTestEnd Time of, and use the compute_summary () function to calculate successful, failed, skipped, and so on.TestNumber and output the result.

This can be summarized as follows:AutomationTestThe framework consists of the following parts:

1. TestPreparations: including environment preparation andTestParsing scripts, suchTestCleaning of the legacy environment, this timeTestThe required environment variables are read and setTestParameter Parsing,TestParsing script names, preparing various output files, and so on.
2. TestScript running in: ParsingTestThe paragraphs in the script are organizedTestStatement, runTestStatement to obtain the actual running result.
3. TestResult comparison and output:TestThen, compare the actual output result with the expected value, including comparison of various formatting methods and Regular Expression methods. Write the results to the specified file as required, and outputTestSummary report of the results.

Back to Top

PHP Automation Test MethodApplication

Understand PHPOfficialAutomationTestOperationMethodYou can create your own PHPTTestScript to help PHP Perform Function Black BoxTest. However, the application of PHPOfficialAutomationTestIdea and framework constructionMethodYou can also build your ownAutomationTestTool for other program developmentTest. In the customAutomationTestSome problems that need to be paid attention to during the framework, and these problems areOfficialAutomationTestThe framework is well resolved. For example:

1. TestScript structure. PHPOfficialAutomationTestThe framework can be completed with a simple PHPT script.TestGenerate all required information for the use case. Use the section in the PHPT scriptTestName,TestCondition,TestThe steps and expected values are clear at a glance, which is very helpful for reading and analyzing. WhenTestPersonnel inTestYou need to analyzeTestYou only need to open the corresponding PHPT script to clearly understand the cause of failure.TestUse Case intention and detailsTestStatement. This greatly reducesTestAnalysis time and difficulty. Moreover, this paragraph structure is scalable.
2. TestEnvironment guarantee. InTestIs the environment "clean" directly affected?TestResult accuracy. Okay.AutomationThe framework should strive to provideTestA "clean" environment. The environment includes the environment variables set by the system and the environment variables set by the user,TestTemporary files that may be used during running. If the previousTestIf the global environment is changed, the nextTestThe result is inaccurate. InaccurateTestThe results may bring a very high false positive rate.
3. TestThe art of result comparison. InTestThe actual output results of PHP are compared with the expected results.OfficialAutomationTestMethodCleverly applied Regular Expression comparison and format comparisonMethod. These two typesMethodThe introductionTestThe false positive rate is greatly reduced. Because if there is no regular expression comparison or format comparisonMethod, ThenTestOnlyTestExpected results are recorded in the expected values, which may becomeTestHard code of the script, in RegressionTestOr the expected result is caused by random values.Test.
4. Retain sufficient log information.TestAfter the endTest,TestAnalyze the cause of the failure. At this time,TestLogs will play an important role. Sufficient log information is retainedTestThe site that fails will helpTestFind the cause of the failure and report it to the developer as soon as possible. PHPOfficialAutomationTestThe framework allows you to flexibly select the desired log type. In various logsTestThe results are stored in the ". out" log file and ". diff" is left to the user. The ". diff" file can helpTestStaff can quickly find the expected value and actual valueTestResult differences for faster discoveryTestThe problem of failure.

However, in PHPOfficialAutomationTestThe framework also has some imperfections for users, such:TestIf an exception occurs, the complete output is not allowed.TestSummary Report. IfTestIt takes up to ten hours for a person to completeTest, AndTestAn exception occurred in the ninth hour,TestBut you cannot give allTestOfTestReport, which will be annoying. ThereforeAutomationTestThe implementation of the framework should be well considered.TestHandle exceptions.

Back to Top

Conclusion

PHPOfficialAutomationTestMethodBlack BoxTestPersonnel provide a goodAutomationTestFramework example.TestPersonnel can not only use PHPTTestComplete scripts for PHPTest, You can also learn this example to build your ownAutomationTestFrameworks and toolsTestOfAutomationBetter ServiceTestWork.