Don't Be Fooled By The Coverage Report

From NeoWiki

Are your test coverage measurements leading you astray?

Andrew Glover, President, Stelligent Incorporated

31 Jan 2006

Test coverage tools bring valuable depth to unit testing, but they're often misused. This month, Andrew Glover brings his considerable expertise in this area to his new series, In pursuit of code quality. This first installment takes a closer look at what the numbers on the coverage report really mean, as well as what they don't. He then suggests three ways you can use your coverage to ensure code quality early and often.

Do you remember what it was like before most developers jumped on the code quality bandwagon? In those days, a skillfully placed main() method was considered both agile and adequate for testing. Yikes! We've come a long way since then. I, for one, am immensely grateful that automated testing is now an essential facet of quality-centric code development. And that's not all I'm grateful for. Java™ developers have a plethora of tools for gauging code quality through code metrics, static analysis, and more. Heck, we've even managed to categorize refactoring into a handy set of patterns!

All these new tools make it easier than ever to ensure code quality, but you do have to know how to use them. In this series of articles, I'll focus on the sometimes arcane details of ensuring code quality. In addition to familiarizing you with the variety of tools and techniques available for code quality assurance, I'll show you how to:

• Define and effectively measure the aspects of your code that most impact quality.
• Set quality assurance goals and plan your development efforts accordingly.
• Decide which code quality tools and techniques actually meet your needs.
• Implement best practices (and weed out poor ones) so that ensuring code quality early and often becomes a painless and effective aspect of your development practice.

I'll start this month with a look at one of the most popular and easy additions to a Java developer's quality assurance toolkit: test coverage measurement.

Beware fools gold

It's the morning following a nightly build and everyone's standing around the water cooler. Developers and management alike are exchanging bold NFL-style pats on the backside when they learn that a few particularly well-tested classes have coverage rates in the high 90s! The collective confidence of the team is at an all-time high. "Refactor with abandon!" can be heard in the distance as defects become a distant memory and the responsibility of the weak and inferior. But there is one small, dissenting voice that says:

Ladies and Gentlemen: Don't be fooled by the coverage report.

Now, don't get me wrong: There's nothing foolish about using test coverage tools. They're a great addition to the unit testing paradigm. What's important is how you synthesize the information once you've got it, and that's where some development teams make their first mistake.

High coverage rates simply mean that a lot of code was exercised. High coverage rates do not imply that code was exercised well. If you're focusing on code quality, you need to understand exactly how test coverage tools work, as well as how they don't; then you'll know how to use these tools to obtain valuable information, rather than just settling for high coverage goals, as many developers do.

Test coverage measurement

About the author

Andrew Glover is president of Stelligent Incorporated, a JNetDirect company. Stelligent Incorporated helps companies address software quality early with effective developer testing strategies and frameworks, software metric analysis and correlations, and continuous integration techniques that enable development teams and management to monitor code quality continuously. He is the co-author of Java Testing Patterns (Wiley, September 2004).

"When you have learned to snatch the error code from the trap frame, it will be time for you to leave.", thus spake the master programmer.