Code Coverage Tool EMMA

Get functional test coverage with EMMA

Test coverage is one of the important metrics for evaluating test integrity. EMMA is a test coverage collection tool for Java code. During testing, using EMMA makes the process of collecting and reporting test coverage more flexible and simple. In this article, the author introduces Emma to the functional testing process, detailing the use of Emma in functional testing, steps, and analysis of coverage results.

1 reviews

Umbrella Yunfei ([email protected]), software engineer, Author11 Company

June 26, 2008

• Content

Develop and deploy your next application on the IBM Bluemix cloud platform.

Get started with your trial

Introduction

EMMA is an open source, Java-oriented program test coverage collection and reporting tool. It collects the coverage information during the test execution by inserting the compiled Java bytecode file, and displays the coverage results by supporting multiple report formats. Emma's use of bytecode inserts not only ensures that Emma does not bring "dirty code" to the source code, but also ensures that Emma is free of the source code, a feature that makes it possible for Emma to apply a functional test.

Note :

• Before you use Emma to collect coverage information in your tests, you need to download the Emma.jar package from Emma's website. You can also get more resources about EMMA on this website.
• EMMA can only collect the coverage of Java code.

The article appendix provides a sample code that contains a war package and a JAR package, where the war package needs to be installed to run on WebSphere Portal Server. In the actual testing process, they can be replaced with the corresponding object being measured.

Back to top of page

Benefits of using EMMA in functional testing

The data that EMMA collects includes class coverage, method coverage, block coverage, and line coverage, which are organized in packages.

Most functional tests, testers generally do not directly get the source code, source code is not the focus of the tester's concern. In the specific testing process, functional testers generally take a meaningful function module as the focus of testing, while the coverage of classes and methods that reflect a certain functional meaning is more valuable in functional testing. Therefore, in functional testing, class coverage and method coverage are the focus of the testers ' concerns, and the line and block coverage are used as a reference for testing.

The test coverage report contains two aspects of the content, testing the covered parts and the parts that were not covered by the test. While a full test coverage does not represent a complete problem for the subject being tested, the coverage and coverage ratios of the test can increase the tester's confidence in the test effort, and guide the test execution and direction of the test. On the other hand, when test case execution occurs, the test report for each test case can also provide the scope of the suspect code to help with code error correction.

The parts that are not covered in the test coverage report are also valuable:

• Indicates that the test may not be complete and that some features and code are not covered by the test.
• Provides guidance for the design of test cases. Under the guidance of coverage reports, testers have a destination to discuss with the developer to determine whether the part that is not covered is the blank of the test or the part that does not need testing.
• Help developers discover useless code and provide a basis for modifying and perfecting the code.

When you use EMMA to get test coverage, the percentage of classes, methods, and so on is reported to make it easier for testers to evaluate tests. Testers design documents against coverage reports and test cases, and requirements documents can quickly find deficiencies in testing. By discussing with the developer, you can better assess the strength of the Test and guide further testing. Therefore, the introduction of coverage information in functional testing, can improve the test results report, ensure the quality of testing and strength, to ensure that the test by quality, according to the amount of completion.

Especially in the Agile development and testing process advocated today, the development and testing cycle is very short, effective coverage information can help testers more accurately control the test results and cycle, tracking problems, ensure the normal release of software.

Back to top of page

EMMA's application in test execution

In this section, you will step through the process and steps of EMMA's use of the functional testing process. In order to make the whole introduction process easy to understand, an example program is provided in the article appendix, which describes the use of EMMA's commands by manipulating the sample program.

Plug-in components to be tested

Emma tracks the execution of the tested component by inserting the tested component, so inserting the tested component is the first step in using Emma to obtain coverage information. Testers should first discuss with the developer to determine which part contains the files (Java files) that meet the requirements of the plug-in, which part needs to consider the coverage information and then choose the appropriate way to plug in.

• Insert Preparation

Before performing a plug-in operation, the Java virtual machine should be extended first, and the Emma.jar will be placed under the JRE directory used by the component under test as an extension of the JRE so that Emma can be called. Emma.jar contains the implementation of the Emma core function module and the class files required by the Emma Runtime, which are required to use Emma.

Because the sample tested component runs in Websphere Portal Server and runs with the default JRE, Emma.jar is placed under "/opt/websphere/portalserver/java/jre/lib/ext". In the actual test, the path is replaced accordingly.

• Plug and Fit

EMMA provides the "InStr" command to complete the plug-and-install operation. Plug-in operations can be made easy by selecting the appropriate commands for JAR packages, war packages, war packages, class files, and directories to be inserted. The following 1-4 describes the plug-in commands in different situations using specific examples.

1. Plug-in catalogs and class files

For a class file, it is implemented by specifying the directory where the class file resides.

Listing 1. Install commands on class files

/opt/websphere/portalserver/java/jre/bin/java Emma Instr-m overwrite-ip/opt/websphere/portalserver/installedapps/ Numberquizweb_10yggsru.ear/numberquizweb.war/web-inf/classes-dmetadata.out.file=/root/emma/number_ coverage.emEMMA:processing Instrumentation path emma:instrumentation path processed in 682 msemma: [5 Class (es) instrume nted, 0 resource (s) copied]emma:metadata merged into [/root/emma/number_coverage.em] {in + ms}

1. Plug-in JAR package

The JAR package can be mounted as a whole. By inserting the whole jar, the process of extracting and compressing the jar package can be avoided, and the efficiency of inserting is improved.

Listing 2. For JAR-Package-mount commands

/opt/websphere/portalserver/java/jre/bin/java Emma Instr-m OVERWRITE-CP  testws.jar-dmetadata.out.file=/root/ emma/number_coverage.ememma:processing Instrumentation path emma:instrumentation path processed in 675 MsEMMA: [7 Class ( ES) instrumented, 4 resource (s) copied]emma:metadata merged into [/root/emma/number_coverage.em] {in $ MS}

1. Cartridge War/ear Bag

Since the War/ear package needs to be run in a specific environment, it needs to be installed in a specific Java EE container before inserting it, and then as a directory for installation.

Listing 3. The War/ear package insert command

/opt/websphere/portalserver/java/jre/bin/java Emma Instr-m overwrite  -ip numberquizweb.war-dmetadata.out.file=  /root/emma/number_coverage.ememma:processing Instrumentation path emma:instrumentation path processed in 610 MsEMMA: [5 Class (es) instrumented, 0 resource (s) copied]emma:metadata merged into [/root/emma/number_coverage.em] {in 94 MS}

1. Optional cartridge

EMMA supports inserting the entire jar package and directory, but if the jar package or directory contains system files or files that are not cared for during testing, it should be selectively inserted because the presence of these files can affect the percentage of test results. EMMA offers the option of selecting inserts to enable selective mounting.

Listing 4. Select the plug-in command

/opt/websphere/portalserver/java/jre/bin/java Emma Instr-m overwrite-cp  Testws.jar-ix +org.wstest.service.*- dmetadata.out.file=/root/emma/number_coverage.ememma:processing Instrumentation Path Emma:instrumentation path Processed in 637 Msemma: [4 Class (ES) instrumented, 6 resource (s) copied]emma:metadata merged into [/root/emma/number_cov Erage.em] {in 107 ms}

The above command selects the same JAR package as in Listing 2, because it contains only the contents of the org.wstest.service.*, so only 4 classes are inserted.

The 1-4 above describes the common commands used in the process of inserting, and explains some of the parameters used in the command.

The parameter "M" represents the mode after which the file output is inserted. There are three values to choose from: "Copy", "Overwrite" and "Fullcopy". The "copy" and "Fullcopy" modes will change the directory in which the plug-in files are located and require testers to manually generate the required packages for them, which is more complex to use. "Overwrite" mode directly with the plug-in file overwrite the pre-insert file, easy to use. However, because the files generated at the same time can only be inserted once, in "Overwrite" mode, the file before the installation is lost, the tester cannot repeat the installation operation, it is recommended to insert the files and packages need to be installed before the backup.

The parameter "IP" and "CP" are used to provide the plug-in path, where "CP" is used to indicate a folder, "IP" specifies a separate file or JAR package.

The parameter "Dmetadata.out.file" is used to specify the path to the saved metadata file for the plug-in.

EMMA specifies the inclusion and exclusion of the file through the "IX" parameter, where the file after the "+" symbol is the file containing the file, "-" after the content is excluded.

• Merging meta-data

After the plug-in operation, a number of files named "*coverage.em" are generated under the specified path, which holds the meta-information of the plug-in, which is primarily a record of the placement of the interpolation in the code under test in the insertion process. If you specify these files to the same file during the plug-in process, EMMA defaults to merging the metadata. If the tester does not specify a path, or if it wants to obtain a separate metafile, the files are generated in the default or specified directory, respectively. Testers can also manually merge these meta-files by using the "merge" command to ensure comprehensiveness of the generated coverage reports. Note: The merge operation does not support reverse operation.

Listing 5. Merge meta-data command

/opt/websphere/portalserver/java/jre/bin/java Emma Merge-input <path1>/coverage1.em,<path2>/ Coverage.em-out <path>/coverage2.em

The parameter after "input" is the name of the file to be merged, and the parameter after "out" is the result file after the merge.

After you have completed the above operation, you have completed the preparation for collecting coverage information. The tester can then perform normal testing, and during the run of the test, EMMA will track and record the execution trajectory to get coverage information.

Run test cases to get coverage reports

After the plug-in is complete, testers can run test cases according to the test plan. EMMA will record code execution information and record the results in memory during the test execution. Each time the JVM is stopped, the in-memory record execution information is purged and saved to the "*.ec" file. However, during the actual testing process, the JVM's stop is difficult to control, so testers can manually write out the in-memory execution information on a regular basis. In this case, the in-memory record is output, but the contents of the memory are not purged. Listing 5-7 describes the commands for gathering coverage information and generating coverage information reports.

Listing 6. Collect coverage information from a remote machine

/OPT/WEBSPHERE/PORTALSERVER/JAVA/JRE/BIN/JAVA-CP Emma.jar Emma Ctl–connect auscsdpfvtvm15.bto.ibm.com:47653- Command coverage.dump,/root/emma/number_coverage.ecemma:processing control command sequence emma:executing [ Coverage.dump (/root/emma/number_coverage.ec,,true)] EMMA:coverage.dump:runtime coverage data remotely merged into                           [/root/emma/number_coverage.ec] {in-Ms}emma:coverage.dump:command completed in Msemma:control command Sequen CE complete

Listing 7. Collect coverage Information locally

/OPT/WEBSPHERE/PORTALSERVER/JAVA/JRE/BIN/JAVA-CP Emma.jar Emma Ctl-connect auscsdpfvtvm15.bto.ibm.com:47653- Command coverage.get,/root/emma/number_coverage.ecemma:processing control command sequence emma:executing [ Coverage.get (/root/emma/number_coverage.ec,true,true)] EMMA:coverage.get:local copy of coverage data merged           into [/root/emma/number_coverage.ec] {In-A-ms} EMMA:coverage.get:command completed in Msemma:control command sequence complete

The information collected is stored in "coverage.ec", and "coverage.ec" is a binary format file, so it is difficult to see the coverage results.

Listing 8. Generate Coverage reports

/OPT/WEBSPHERE/PORTALSERVER/JAVA/JRE/BIN/JAVA-CP Emma.jar Emma Report-r html-in/root/emma/number_coverage.em,/root /EMMA/NUMBER_COVERAGE.EC-DREPORT.HTML.OUT.FILE=/ROOT/EMMA/NUMBER_COVERAGE.HTML-DREPORT.METRICS=CLASS:80 (, method:75) emma:processing Input files Emma:2 file (s) Read and merged in msemma:writing [HTML] [/root/emma/ Coverage/number_coverage.xml]

In the process of generating coverage reports, testers can set satisfactory coverage criteria through the "dreport.metrics" parameter according to the test requirements. In the example command, the satisfaction of class coverage is set to 80%.

Test reports can be exported in HTML, text, and XML in three formats. Figure 1, Figure 2 is an example of an HTML-formatted report. Coverage reports are organized in package, class, and method level three units. Figure 1 is the execution of the Index class, where the red color indicates that the coverage has not met the satisfactory coverage criteria. Figure 2 is the implementation of the Quizbran class in package Org.numberquiz, in general, the class coverage is 100%, the method is 91%. The complete test coverage report for the sample program can be found in the appendix.

Figure 1. Index test Report Figure 2. Quizbran Test Report

Generating separate coverage reports for each individual test case during functional testing can be a great help to the test process:

• When a test case fails or throws an exception, the code corresponding to the test case can be found through the coverage report, providing the tester with a range of possible error codes. This report not only helps testers describe errors in more detail when they submit a problem, but also provides clues to the developer's ability to track issues and shorten the time-to-resolution cycle.
• Testers can get the corresponding relationship between the code and the function module from the independent test report, and better understand the role of the test case.
• Independent test reports can help testers improve the design of test cases, remove duplicate test cases, and split test cases with more coverage points.

In order to obtain a separate test report, the execution information in memory needs to be purged before each test case is executed. There are currently two ways to clear records, and testers can choose the right approach as needed during the test.

• Restart the JVM each time you run through a test case. This method completely clears the execution information in memory, but every time the JVM is restarted it can cause a lot of trouble for the test.
• Using the Coverage.reset command, this command clears the execution information of the methods, blocks, and rows logged in memory without restarting the JVM, but does not clear the class overwrite information. This method can be selected if the user focuses on the coverage information of the method.

Listing 9. Clear in-memory coverage Information command

/OPT/WEBSPHERE/PORTALSERVER/JAVA/JRE/BIN/JAVA-CP Emma.jar Emma ctl-connectauscsdpfvtvm15.bto.ibm.com:47653- Command coverage.resetEMMA:processing control command sequenceemma:executing [Coverage.reset ()]emma:coverage.reset: Coverage reset for 5 classes {in 0 Ms}emma:coverage.reset:command completed in msemma:control command sequence Compl Ete

Merge coverage Results

After the test cases are completed, the test coverage information can be combined to get the coverage report for the entire test. The coverage results file is implemented by merging the "*.ec" file with the "merge" command.

Also, because the test coverage in EMMA is obtained by associating the code information with the "*.em" file, when the code changes, the tests that have already run do not have to be completely duplicated, simply merge the resulting "*.ec" file (the newly obtained "*.ec" file), and then associate the latest "*.em "File to get coverage information after the code changes, which facilitates the test of EMMA's support for version changes. When generating a new test report, it is important to note that the "*.ec" time must be later than the "*.em" file.

Listing 9. Merge Coverage Results command

/opt/websphere/portalserver/java/jre/bin/java Emma Merge–input Coverage1.ec,coverage2.ec,coverage3.ec–output Coverage.ec

If a test report is generated, "com.vladium.emma.EMMARuntimeException: [Class_stamp_mismatch] runtime version of CLASS in the Coverage data is not consistent with the version of this class in the metadata, possibly because stale metadata is being u SED for report generation "error, stating that code has been modified before and after generating a new" *.em ", and that the class file where the modified code is located is not overwritten in the new test, which requires re-executing this part of the test to ensure that the modified part is re-executed.

Back to top of page

Use coverage reports to summarize and evaluate test procedures

So far, the coverage reports for the test coverage and the overall testing of each test case have been made. These reports can help testers summarize and analyze test results and improve test design.

In the current functional testing, the testers mainly use the number of test cases performed and the problem report during the test process to evaluate the test process, so the design of the test case is directly related to the adequacy of the test. Testers are often unable to get information from the current test results report, which parts are overwritten and which parts are not covered, causing the tester to have no confidence in the test results at the end of the test. In addition, in many cases, testers have completed tests of certain situations, and anomalies are often overlooked.

Coverage reports provide a clear view of how testers view test coverage. Although 100% coverage does not prove to be a problem, it provides a reference for testers. When coverage is low, testers can find the cause by using coverage report hints.

Discuss test results with developers

Under the guidance of coverage results, test case design, and execution documentation, testers can more clearly discuss test results with developers, and they can find out more clearly:

• The part that the tester ignored.
• The part of the test case design that is overwritten and not covered in the test execution.
• Useless code in the program.

For example: In the actual testing process, found that a call Syslog.messages packet coverage has been very low, some classes and methods are never overwritten. The results associated with these classes did not appear during the test execution. Therefore, after confirming with the developer, the discovery of the call to the content is forgotten when the code is written (the contents of the red box in Figure 3).

Figure 3. Coverage report analysis improved test design

With coverage reports, testers can improve the design of test cases:

• Remove the test cases where the coverage repeats.
• For test cases with too many coverage points, you can split them to ensure that the test cases are targeted.
• For parts that are not covered in the test, increase the test case to ensure test integrity.

Back to top of page

Conclusion

In functional testing, testers generally do not have direct access to the source code of the object under test, and the class and method coverage can be used as important criteria for measuring and evaluating tests. EMMA captures and reports coverage information from tests based on bytecode interpolation, making coverage successful for functional testing. By using EMMA in a functional test, you can improve your test design, help with problem analysis, and better evaluate your tests to ensure the quality of your tests.

Code Coverage Tool EMMA