Test Coverage rate measured with cobertura

Cobertura is an open-source tool that performs basic detectionCodeAnd observe what code is executed and what code is not executed when the test package is run to measure the test coverage rate. In addition to finding untested code and bugs, cobertura can optimize the code by marking useless and unexecuted code. It can also provide internal information for actual API operations. Elliotte Rusty Harold will share with you how to use cobertura with best practices for code coverage.

Although test-first programming and unit testing are no longer new concepts, test-driven development remains the most important programming innovation in the past 10 years. Some of the best programmers have been using these technologies for the past half century, but in recent years, these technologies are widely regarded as the key to developing robust defect-free software within time and cost budgets. However, the test-driven development cannot exceed the test level. The test improves the code quality, but this is only for the part of the Code actually tested. You need a tool to tell youProgramIn this way, you can write test code for these parts and find more bugs.

Mark doliner's cobertura (CoberturaIn Spanish) is a free GPL tool to complete this task. Cobertura uses additional statements to record which rows are tested and which rows are not tested when a test package is executed. In this way, cobertura measures bytecode to monitor the test. Then it generates an HTML or XML report, indicating which packages, classes, methods, and rows in the Code are not tested. You can write more test code for these specific areas to discover all hidden bugs.

Read cobertura output

First, check the generated cobertura output. Figure 1 shows the report generated by running cobertura on the jaxen test package (see references ). From this report, we can see thatOrg. jaxen. expr. ITERAlmost 100% in the package) to very poor (inOrg.jaxen.dom.html.

Figure 1. jaxen package-level coverage statistics

 

Cobertura calculates coverage by testing the number of lines and the number of branches to be tested. During the first test, the difference between the two test methods is not very important. Cobertura also calculates the average McCabe complexity for classes (see references ).

You can dig into HTML reports to understand the coverage of specific packages or classes. Figure 2 showsOrg. jaxen. FunctionPackage coverage rate. In this package, the coverage rate ranges fromSumfunctionClass 100%IdfunctionOnly 5% of classes are supported.

Figure 2. code coverage in the org. jaxen. Function package

 

Further dive into a separate class and check which line of code is not tested. Figure 3 showsNamefunctionClass. The row number is displayed in the leftmost column. The next column shows the number of times this row was executed during the test. It can be seen that 112nd rows were executed 100 times, and 114th rows were executed 28 times. The Rows highlighted in red are not tested at all. This report shows that although the method has been tested in general, many branches are not actually tested.

Figure 3. code coverage in the namefunction class

 

Cobertura is the branch of jcoverage (see references ). The GPL version of jcoverage has not been updated for one year, and there are some long-term bugs that cobertura has fixed. The original jcoverage developers no longer continue to develop open source code. They switched to the commercial version of jcoverage and jcoverage +. jcoverage + is a closed block developed from the same code base.Source codeProduct. The wonder of open source code is that a product will not die out because the original developer decides to make their work get the appropriate reward.

Confirm missing tests

Using the cobertura report, you can find untested parts of the code and write and test them for them. For example, Figure 3 shows that jaxen needs to be tested and usedName ()The function tests the text node, comment node, processing command node, attribute node, and namespace node.

If there are a lot of uncovered code, as cobertura reported here, it will be very time-consuming to add all the missing tests, but it is also worth it. It does not have to be completed at once. You can start with the least tested code, such as all the unoverwritten packages. After testing all the packages, you can write some test code for each class that is displayed as not covered. After a special test is performed on all classes, you need to write test code for all methods that are not covered. After testing all the methods, you can start to analyze the necessity of testing untested statements.

(Almost) No untested code is left

Is there anything that can be tested but should not be tested? It depends on who you are asking. In the JUnit FAQ, J. B. rainsberger wrote, "The general idea is: ifItselfThere will be no problems, so it will be too simple and there will be no problems. The first example isGetx ()Method. Assume thatGetx ()The method only provides the value of an instance variable. In this case, unless a problem occurs with the compiler or interpreterGetx ()There will be no problems. Therefore, you do not need to testGetx (), Test it will not bring any benefits. ForSetx ()The same is true for methods. However, ifSetx ()If the method does require any parameter verification or side effects, it is necessary to test it ."

Theoretically, writing test code for unoverwritten Code may not necessarily find bugs. However, in practice, I have never encountered any bugs. Untested code is full of bugs. The fewer tests you perform, the more bugs hidden in the code that are not found.

I disagree. I can't remember the number of bugs found in the code "simple enough to avoid problems. Indeed, some getter and setter are very simple and cannot cause problems. However, I have never been able to tell which methods are really simple and will not make any mistakes, but what methods just look like this. It is not difficult to compile the test code that covers a simple method like setter and getter. This takes a small amount of time to compensate for unexpected bugs found in these methods.

Generally speaking, it is easy to reach 90% of the test coverage after the measurement starts. To increase the coverage rate to 95% or higher, you need to worry about it. For example, you may need to load support libraries of different versions to test bugs in libraries of all versions. Or you need to re-build the code to test the code that is usually not executed. Classes can be expanded to change their protected methods to public methods, so that they can be tested. These techniques seem to be an alternative, but they helped me discover more bugs in half of the time.

It is not always possible to get a perfect code coverage rate of 100%. Sometimes you will find that no matter how code is transformed, there are still some rows, methods, or even the entire class that cannot be tested. The following are some examples of the challenges you may encounter:

• Only code executed on a specific platform. For example, in a well-designed GUI application, the code for adding an exit menu item can run on Windows PC, but it cannot run on Mac.

• Catch exceptions that do not occurCatchStatement, suchBytearrayinputstreamThrown during read OperationsIoexception.
• Some methods in non-public classes are never actually called, but must be implemented to satisfy an interface contract.
• Code blocks that handle Virtual Machine bugs, for example, cannot recognize UTF-8 encoding.

Considering the above and similar situations, I think some extreme programmers are impractical and may be ironic to automatically delete all untested code. The failure to always achieve absolutely perfect test coverage does not mean that there will be no better coverage.

However, the residual code is more common than statements and methods that cannot be executed. It does not have any effect and removing the code from the code base will not have any impact. Sometimes you can use reflection to access private members to test untested code. You can also write test code for untested package-protected code and place the test class in the package where the class to be tested is located. But it is better not to do this. All codes that cannot be accessed through published (public and protected) interfaces should be deleted. Code that cannot be executed should not be part of the code base. The smaller the code base, the easier it will be understood and maintained.

Do not miss measurement unit test packages and classes. I have noticed more than once that some test methods or classes are not actually run by the test package. This usually indicates that there is a problem in the Naming Convention (for example, naming a methodTesSomereallycomplexconditionInstead of naming itTestSomereallycomplexcondition), Or forget to add a class to the masterSuite ()Method. In other cases, unexpected conditions cause the code in the test method to be skipped. No matter what the situation is, although the test code has been compiled, but it does not really run it. JUnit will not tell you that it does not run all tests as you think, but cobertura will tell you. It is usually easy to correct an unexecuted test.

Back to Top

Run cobertura

After learning about the benefits of measuring code coverage, let's discuss how to use cobertura to Measure code coverage. Cobertura is designed to run in ant. No ide plug-ins are available yet, but it may take a year or two.

First, you must add a task definition in the build. xml file. The following top-levelTaskdefElement limits the cobertura. jar file to the current working directory:

<Taskdef classpath = "cobertura. Jar" resource = "tasks. properties"/>

Then, you needCobertura-InstrumentTask, which adds the log code to the compiled class file.TodirAttribute specifies where the measurement class is placed.FilesetThe child element specifies which. class files to measure:

<Target name = "instrument"> <cobertura-instrument todir = "target/instrumented-classes"> <fileset dir = "target/classes"> <include name = "**/ *. class "/> </fileset> </cobertura-instrument> </Target>

Run the test with the same type of ant tasks that normally run the test package. The only difference is that the class to be measured must appear in the class path before the original class appears in the class path, and the cobertura JAR file must be added to the class path:

<Target name = "cover-test" depends = "instrument"> <mkdir dir = "$ {testreportdir}"/> <JUnit dir = ". /"failureproperty =" test. failure "printsummary =" yes "fork =" true "haltonerror =" true "> <! -- Normally you can create this task by copying your existing JUnit target, changing its name, and adding these next two lines. you may need to change the locations to point to wherever you 've put the cobertura. JAR file and the instrumented classes. --> <classpath location = "cobertura. jar "/> <classpath location =" target/instrumented-classes "/> <classpath> <fileset dir =" $ {libdir} "> <include name = "*. jar "/> </fileset> <pathelement Path =" $ {testclassesdir} "/> <pathelement Path =" $ {classesdir} "/> </classpath> <batchtest todir = "$ {testreportdir}"> <fileset dir = "src/Java/test"> <include name = "**/* test. java "/> <include name =" org/jaxen/JavaBean/* test. java "/> </fileset> </batchtest> </JUnit> </Target>

The jaxen project uses JUnit as its testing framework, but cobertura is not affected by the framework. It works as well in testng, artima suiterunner, httpuni, or a system developed in your own basement.

Finally,Cobertura-ReportTask generation: the HTML file that you see at the beginning of this article:

<Target name = "coverage-Report" depends = "cover-test"> <cobertura-Report srcdir = "src/Java/main" destdir = "cobertura"/> </Target>

SrcdirAttribute specifies where the original. Java source code is located.DestdirAttribute specifies the name of the directory where cobertura places the output HTML.

After adding similar tasks to your ant compilation file, you can generate an overwrite report by typing the following command:

% Ant instrument % ant cover-test % ant coverage-Report

Of course, you can also change the name of the target task, or combine the three tasks into a target task.