Don't Be Fooled By The Coverage Report
From NeoWiki
- Are your test coverage measurements leading you astray?
Andrew Glover, President, Stelligent Incorporated
31 Jan 2006
- Test coverage tools bring valuable depth to unit testing, but they're often misused. This month, Andrew Glover brings his considerable expertise in this area to his new series, In pursuit of code quality. This first installment takes a closer look at what the numbers on the coverage report really mean, as well as what they don't. He then suggests three ways you can use your coverage to ensure code quality early and often.
Do you remember what it was like before most developers jumped on the code quality bandwagon? In those days, a skillfully placed main() method was considered both agile and adequate for testing. Yikes! We've come a long way since then. I, for one, am immensely grateful that automated testing is now an essential facet of quality-centric code development. And that's not all I'm grateful for. Java™ developers have a plethora of tools for gauging code quality through code metrics, static analysis, and more. Heck, we've even managed to categorize refactoring into a handy set of patterns!
All these new tools make it easier than ever to ensure code quality, but you do have to know how to use them. In this series of articles, I'll focus on the sometimes arcane details of ensuring code quality. In addition to familiarizing you with the variety of tools and techniques available for code quality assurance, I'll show you how to:
- Define and effectively measure the aspects of your code that most impact quality.
- Set quality assurance goals and plan your development efforts accordingly.
- Decide which code quality tools and techniques actually meet your needs.
- Implement best practices (and weed out poor ones) so that ensuring code quality early and often becomes a painless and effective aspect of your development practice.
I'll start this month with a look at one of the most popular and easy additions to a Java developer's quality assurance toolkit: test coverage measurement.
Contents |
Beware fools gold
It's the morning following a nightly build and everyone's standing around the water cooler. Developers and management alike are exchanging bold NFL-style pats on the backside when they learn that a few particularly well-tested classes have coverage rates in the high 90s! The collective confidence of the team is at an all-time high. "Refactor with abandon!" can be heard in the distance as defects become a distant memory and the responsibility of the weak and inferior. But there is one small, dissenting voice that says:
Ladies and Gentlemen: Don't be fooled by the coverage report.
Now, don't get me wrong: There's nothing foolish about using test coverage tools. They're a great addition to the unit testing paradigm. What's important is how you synthesize the information once you've got it, and that's where some development teams make their first mistake.
High coverage rates simply mean that a lot of code was exercised. High coverage rates do not imply that code was exercised well. If you're focusing on code quality, you need to understand exactly how test coverage tools work, as well as how they don't; then you'll know how to use these tools to obtain valuable information, rather than just settling for high coverage goals, as many developers do.
Test coverage measurement
Test coverage tools are generally easy to add into an established unit testing process, and the results can be reassuring. Simply download one of the available tools, slightly modify your Ant or Maven build script, and you and your colleagues have a new kind of report to talk about around the water cooler: The Test Coverage Report. It can be a big comfort when packages like foo and bar show astonishingly high coverage, and it's tempting to rest easy when you believe that at least a portion of your code is certifiably "bug free." But to do so would be a mistake.
There are different types of coverage measurements, but most tools focus on line coverage, also known as statement coverage. In addition, some tools report branch coverage. A test coverage measurement is obtained by exercising a code base with a test harness and capturing data that corresponds to code having been "touched" throughout the lifetime of the test process. The data is then synthesized to produce a coverage report. In Java shops, the test harness is commonly JUnit and the coverage tool is usually something like Cobertura, Emma, or Clover, to name a few.
Line coverage simply indicates that a particular line of code was exercised. If a method is 10 lines long and 8 lines of the method were exercised in a test run, then the method has a line coverage of 80%. The process works at the aggregate level as well: If a class has 100 lines and 45 of them were touched, then the class has a line coverage of 45%. Likewise, if a code base comprises 10,000 non-commenting lines of code and 3,500 of them were exercised on a particular test run, then the code base's line coverage is 35%.
Tools that report branch coverage attempt to measure the coverage of decision points, such as conditional blocks containing logical ANDs or ORs. Just like line coverage, if there are two branches in a particular method and both were covered through tests, then you could say the method has 100% branch coverage.
The question is, how useful are these measurements? Clearly, all of this information is easy to obtain, but it's up to you to be discerning about how you synthesize it. Some examples clarify my point.
Code coverage in action
I've created a simple class in Listing 1 to embody the notion of a class hierarchy. A given class can have a succession of superclasses -- like Vector, whose parent is AbstractList, whose parent is AbstractCollection, whose parent is Object:
Listing 1. A class that represent a class hierarchy
package com.vanward.adana.hierarchy;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
public class Hierarchy {
private Collection classes;
private Class baseClass;
public Hierarchy() {
super();
this.classes = new ArrayList();
}
public void addClass(final Class clzz){
this.classes.add(clzz);
}
/**
* @return an array of class names as Strings
*/
public String[] getHierarchyClassNames(){
final String[] names = new String[this.classes.size()];
int x = 0;
for(Iterator iter = this.classes.iterator(); iter.hasNext();){
Class clzz = (Class)iter.next();
names[x++] = clzz.getName();
}
return names;
}
public Class getBaseClass() {
return baseClass;
}
public void setBaseClass(final Class baseClass) {
this.baseClass = baseClass;
}
}
As you can see, Listing 1's Hierarchy class holds a baseClass instance and its collection of superclasses. The HierarchyBuilder in Listing 2 creates the Hierarchy class through two overloaded static methods dubbed buildHierarchy:
Listing 2. A class hierarchy builder
package com.vanward.adana.hierarchy;
public class HierarchyBuilder {
private HierarchyBuilder() {
super();
}
public static Hierarchy buildHierarchy(final String clzzName)
throws ClassNotFoundException{
final Class clzz = Class.forName(clzzName, false, HierarchyBuilder.class.getClassLoader());
return buildHierarchy(clzz);
}
public static Hierarchy buildHierarchy(Class clzz){
if(clzz == null){
throw new RuntimeException("Class parameter can not be null");
}
final Hierarchy hier = new Hierarchy();
hier.setBaseClass(clzz);
final Class superclass = clzz.getSuperclass();
if(superclass !=
null && superclass.getName().equals("java.lang.Object")){
return hier;
}else{
while((clzz.getSuperclass() != null) &&
(!clzz.getSuperclass().getName().equals("java.lang.Object"))){
clzz = clzz.getSuperclass();
hier.addClass(clzz);
}
return hier;
}
}
}
It's testing time!
What would an article about test coverage be without a test case? In Listing 3, I define a simple, sunny-day scenario JUnit test class with three test cases, which attempt to exercise both the Hierarchy and HierarchyBuilder classes:
Listing 3. Test that HierarchyBuilder!
package test.com.vanward.adana.hierarchy;
import com.vanward.adana.hierarchy.Hierarchy;
import com.vanward.adana.hierarchy.HierarchyBuilder;
import junit.framework.TestCase;
public class HierarchyBuilderTest extends TestCase {
public void testBuildHierarchyValueNotNull() {
Hierarchy hier = HierarchyBuilder.buildHierarchy(HierarchyBuilderTest.class);
assertNotNull("object was null", hier);
}
public void testBuildHierarchyName() {
Hierarchy hier = HierarchyBuilder.buildHierarchy(HierarchyBuilderTest.class);
assertEquals("should be junit.framework.Assert",
"junit.framework.Assert",
hier.getHierarchyClassNames()[1]);
}
public void testBuildHierarchyNameAgain() {
Hierarchy hier = HierarchyBuilder.buildHierarchy(HierarchyBuilderTest.class);
assertEquals("should be junit.framework.TestCase",
"junit.framework.TestCase",
hier.getHierarchyClassNames()[0]);
}
}
Because I'm an avid tester, I naturally want to run some coverage tests. Of the code coverage tools available to Java developers, I tend to stick with Cobertura because I like its friendly reports. Also, Cobertura is an open source project, which forked the pioneering JCoverage project.
The Cobertura report
Running a tool like Cobertura is as simple as running your JUnit tests, only with a middle step of instrumenting the code under test with specialized logic to report on coverage (which is all handled via the tool's Ant tasks or Maven's goals).
As you can see in Figure 1, the coverage report for HierarchyBuilder illustrates a few sections of code that weren't exercised. In fact, Cobertura claims that HierarchyBuilder has as 59% line coverage rate and a 75% branch coverage rate.
Figure 1. The Cobertura report
So, my first shot at coverage testing failed to test a number of things. First, the buildHierarchy() method, which takes a String as a parameter, wasn't tested at all. Second, two conditions in the other buildHierarchy() method weren't exercised either. Interestingly, it's the second unexercised if block that is a concern.
I'm not worried at this point because all I have to do is add a few more test cases. Once I've reached those areas of concern, I should be good to go. Note my logic here: I used the coverage report to understand what wasn't tested. Now I have the option to use this data to either enhance my testing or move on. In this case, I'm going to enhance my testing, because I've left a few important areas uncovered.
Cobertura: Round 2
Listing 4 is an updated JUnit test case that adds a handful of additional test cases in an attempt to fully exercise HierarchyBuilder:
Listing 4. An updated JUnit test case
package test.com.vanward.adana.hierarchy;
import com.vanward.adana.hierarchy.Hierarchy;
import com.vanward.adana.hierarchy.HierarchyBuilder;
import junit.framework.TestCase;
public class HierarchyBuilderTest extends TestCase {
public void testBuildHierarchyValueNotNull() {
Hierarchy hier = HierarchyBuilder.buildHierarchy(HierarchyBuilderTest.class);
assertNotNull("object was null", hier);
}
public void testBuildHierarchyName() {
Hierarchy hier = HierarchyBuilder.buildHierarchy(HierarchyBuilderTest.class);
assertEquals("should be junit.framework.Assert",
"junit.framework.Assert",
hier.getHierarchyClassNames()[1]);
}
public void testBuildHierarchyNameAgain() {
Hierarchy hier = HierarchyBuilder.buildHierarchy(HierarchyBuilderTest.class);
assertEquals("should be junit.framework.TestCase",
"junit.framework.TestCase",
hier.getHierarchyClassNames()[0]);
}
public void testBuildHierarchySize() {
Hierarchy hier = HierarchyBuilder.buildHierarchy(HierarchyBuilderTest.class);
assertEquals("should be 2", 2, hier.getHierarchyClassNames().length);
}
public void testBuildHierarchyStrNotNull() throws Exception{
Hierarchy hier =
HierarchyBuilder.buildHierarchy("test.com.vanward.adana.hierarchy.HierarchyBuilderTest");
assertNotNull("object was null", hier);
}
public void testBuildHierarchyStrName() throws Exception{
Hierarchy hier =
HierarchyBuilder.buildHierarchy("test.com.vanward.adana.hierarchy.HierarchyBuilderTest");
assertEquals("should be junit.framework.Assert",
"junit.framework.Assert",
hier.getHierarchyClassNames()[1]);
}
public void testBuildHierarchyStrNameAgain() throws Exception{
Hierarchy hier =
HierarchyBuilder.buildHierarchy("test.com.vanward.adana.hierarchy.HierarchyBuilderTest");
assertEquals("should be junit.framework.TestCase",
"junit.framework.TestCase",
hier.getHierarchyClassNames()[0]);
}
public void testBuildHierarchyStrSize() throws Exception{
Hierarchy hier =
HierarchyBuilder.buildHierarchy("test.com.vanward.adana.hierarchy.HierarchyBuilderTest");
assertEquals("should be 2", 2, hier.getHierarchyClassNames().length);
}
public void testBuildHierarchyWithNull() {
try{
Class clzz = null;
HierarchyBuilder.buildHierarchy(clzz);
fail("RuntimeException not thrown");
}catch(RuntimeException e){}
}
}
When I run the test coverage process again with the new test cases, I get a much more complete report, as shown in Figure 2. I've now covered the untested buildHierarchy() method as well as hitting both if blocks in the other buildHierarchy() method. HierarchyBuilder's constructor is private, however, so I can't test it via my test class (nor do I care to); therefore, my line coverage still hovers at 88%.
Figure 2. Who says there are no second chances?
As you can see, using a code coverage tool can uncover important code that doesn't have a corresponding test case. The important thing is to exercise caution when viewing the reports (especially ones with high values), for they can hide nefarious subtleties. Let's look at a couple more examples of code issues that can hide behind high coverage rates.
The trouble with conditionals
As you may well know by now, many variables found in code can have more than one state; furthermore, the presence of conditionals creates multiple paths of execution. With these caveats in mind, I've defined an absurdly simple class with one method in Listing 5:
About the author
Andrew Glover is president of Stelligent Incorporated, a JNetDirect company. Stelligent Incorporated helps companies address software quality early with effective developer testing strategies and frameworks, software metric analysis and correlations, and continuous integration techniques that enable development teams and management to monitor code quality continuously. He is the co-author of Java Testing Patterns (Wiley, September 2004).
- "When you have learned to snatch the error code from the trap frame, it will be time for you to leave.", thus spake the master programmer.