JUnit 4 vs. TestNG

From NeoWiki

Why TestNG is still the better framework for large-scale testing

Andrew Glover, President, Stelligent Incorporated

29 Aug 2006

With its new, annotations-based framework, JUnit 4 has embraced some of the best features of TestNG, but does that mean it's rendered TestNG obsolete? Andrew Glover considers what's unique about each framework and reveals three high-level testing features you'll still find only in TestNG.

JUnit 4.0 was released early this year, following a long hiatus from active development. Some of the most interesting changes to the JUnit framework -- especially for readers of this column -- are enabled by the clever use of annotations. In addition to a radically updated look and feel, the new framework features dramatically relaxed structural rules for test case authoring. The previously rigid fixture model has also been relaxed in favor of a more configurable approach. As a result, JUnit no longer requires that you define a test as a method whose name starts with test, and you can now run fixtures just once as opposed to for each test.

These changes are most welcome, but JUnit 4 isn't the first Java™ test framework to offer a flexible model based on annotations. TestNG established itself as an annotations-based framework long before the modifications to JUnit were in progress.

In fact, TestNG pioneered testing with annotations in Java programming, which made it a formidable alternative to JUnit. Since the release of JUnit 4, however, many developers are asking if there's still any difference between the two frameworks. In this month's column, I'll discuss some of the features that set TestNG apart from JUnit 4 and suggest the ways in which the two frameworks continue to be more complementary than competitive.

Contents 1 Similar on the surface 1.1 A simple test case 1.2 Even more flexibility 2 Dependency testing 2.1 Skipping, not failing 2.2 Fail and rerun 2.3 Parametric testing 2.4 Advanced parametric testing 3 Why choose one? 4 Resources 5 About the author

Similar on the surface

JUnit 4 and TestNG have some important attributes in common. Both frameworks facilitate testing by making it amazingly simple (and fun), and they both have vibrant communities that support active development while generating copious documentation.

Where the frameworks differ is in their core design. JUnit has always been a unit-testing framework, meaning that it was built to facilitate testing single objects, and it does so quite effectively. TestNG, on the other hand, was built to address testing at higher levels, and consequently, has some features not available in JUnit.

A simple test case

At first glance, tests implemented in JUnit 4 and TestNG look remarkably similar. To see what I mean, take a look at the code in Listing 1, a JUnit 4 test that has a macro-fixture (a fixture that is called just once before any tests are run), which is denoted by the @BeforeClass attribute:

Listing 1. A simple JUnit 4 test case

package test.com.acme.dona.dep;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import org.junit.BeforeClass;
import org.junit.Test;

public class DependencyFinderTest {
  private static DependencyFinder finder;

  @BeforeClass
  public static void init() throws Exception {
    finder = new DependencyFinder();
  }

  @Test
  public void verifyDependencies() throws Exception {
    String targetClss = "test.com.acme.dona.dep.DependencyFind";
    Filter[] filtr = new Filter[] { 
      new RegexPackageFilter("java|junit|org")};

    Dependency[] deps = 
      finder.findDependencies(targetClss, filtr);

    assertNotNull("deps was null", deps);
    assertEquals("should be 5 large", 5, deps.length);	
  }
}

JUnit users will immediately note that this class lacks much of the syntactic sugar required by previous versions of JUnit. There isn't a setUp() method, the class doesn't extend TestCase, and it doesn't even have any methods that start with test. This class also makes use of Java 5 features like static imports and, obviously, annotations.

Even more flexibility

In Listing 2, you see the same test, but this time it's implemented using TestNG. There is one subtle difference between this code and the test in Listing 1. Do you see it?

Listing 2. A TestNG test case

package test.com.acme.dona.dep;

import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotNull;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Configuration;
import org.testng.annotations.Test;

public class DependencyFinderTest {
  private DependencyFinder finder;

  @BeforeClass
  private void init() {
    this.finder = new DependencyFinder();
  }

  @Test
  public void verifyDependencies() throws Exception {
    String targetClss = 
      "test.com.acme.dona.dep.DependencyFind";

    Filter[] filtr = new Filter[] { 
      new RegexPackageFilter("java|junit|org")};

    Dependency[] deps = 
      finder.findDependencies(targetClss, filtr);
  
    assertNotNull(deps, "deps was null" );
    assertEquals(5, deps.length, "should be 5 large");		
  }
}

Obviously the two listings are pretty similar, but if you look closely, you'll see that the TestNG coding conventions are more flexible than JUnit 4's. In Listing 1, JUnit forced me to declare my @BeforeClass decorated method as static, which consequently required me to also declare my fixture, finder, as static. I also had to declare my init() method as public. Looking at Listing 2, you see a different story because those conventions aren't required. My init() method is neither static nor public.

Flexibility has been one of the strong points of TestNG right from the start, but that's not its only selling point. TestNG also offers some testing features you won't find in JUnit 4.

Dependency testing

One thing the JUnit framework tries to achieve is test isolation. On the downside, this makes it very difficult to specify an order for test-case execution, which is essential to any kind of dependent testing. Developers have used different techniques to get around this, like specifying test cases in alphabetical order or relying heavily on fixtures to properly set things up.

These workarounds are fine for tests that succeed, but for tests that fail, they have an inconvenient consequence: every subsequent dependent test also fails. In some situations, this can lead to large test suites reporting unnecessary failures. For example, imagine a test suite that tests a Web application that requires a login. You might work around JUnit's isolationism by creating a dependent method that sets up the entire test suite with a login to the application. Nice problem solving, but when the login fails, the entire suite fails too -- even if the application's post-login functionality works!

Skipping, not failing

Unlike JUnit, TestNG welcomes test dependencies through the dependsOnMethods attribute of the Test annotation. With this handy feature, you can easily specify dependent methods, such as the login from above, which will execute before a desired method. What's more, if the dependent method fails, then all subsequent tests will be skipped, not marked as failed.

Listing 3. Dependent testing with TestNG

import net.sourceforge.jwebunit.WebTester;

public class AccountHistoryTest  {
  private WebTester tester;

  @BeforeClass
  protected void init() throws Exception {
    this.tester = new WebTester();
    this.tester.getTestContext().
     setBaseUrl("http://div.acme.com:8185/ceg/");
  }

  @Test
  public void verifyLogIn() {
    this.tester.beginAt("/");		
    this.tester.setFormElement("username", "admin");
    this.tester.setFormElement("password", "admin");
    this.tester.submit();		
    this.tester.assertTextPresent("Logged in as admin");
  }

  @Test (dependsOnMethods = {"verifyLogIn"})
  public void verifyAccountInfo() {
    this.tester.clickLinkWithText("History", 0);		
    this.tester.assertTextPresent("GTG Data Feed");
  }
}

In Listing 3, two tests are defined: one for verifying a login and another for verifying account information. Note that the verifyAccountInfo test specifies that it depends on the verifyLogIn() method using the dependsOnMethods = {"verifyLogIn"} clause of the Test annotation.

If you ran this test through TestNG's Eclipse plug-in (for example) and the verifyLogIn test failed, TestNG would simply skip the verifyAccountInfo test, as shown in Figure 1:

Figure 1. Skipped tests in TestNG

TestNG's trick of skipping, rather than failing, can really take the pressure off in large test suites. Rather than trying to figure out why 50 percent of the test suite failed, your team can concentrate on why 50 percent of it was skipped! Better yet, TestNG complements its dependency testing setup with a mechanism for rerunning only failed tests.

Fail and rerun

The ability to rerun failed tests is especially handy in large test suites, and it's a feature you'll only find in TestNG. In JUnit 4, if your test suite consists of 1000 tests and 3 of them fail, you'll likely beforced to rerun the entire suite (with fixes). Needless to say, this sort of thing can take hours.

Anytime there is a failure in TestNG, it creates an XML configuration file that delineates the failed tests. Running a TestNG runner with this file causes TestNG to only run the failed tests. So, in the previous example, you would only have to rerun the three failed tests and not the whole suite.

You can actually see this for yourself using the Web testing example from Listing 2. When the verifyLogIn() method failed, TestNG automatically created a testng-failed.xml file. The file serves as an alternate test suite in Listing 4:

Listing 4. Failed test XML file

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite thread-count="5" verbose="1" name="Failed suite [HistoryTesting]" 
  parallel="false" annotations="JDK5">
  <test name="test.com.acme.ceg.AccountHistoryTest(failed)" junit="false">
    <classes>
      <class name="test.com.acme.ceg.AccountHistoryTest">
        <methods>
          <include name="verifyLogIn"/>
        </methods>
      </class>
    </classes>
  </test>
</suite>

This feature doesn't seem like such a big deal when you're running smaller test suites, but you quickly come to appreciate it as your test suites grow in size.

Parametric testing

Another interesting feature available in TestNG and not in JUnit 4 is parametric testing. In JUnit, if you want to vary the parameter groups to a method under test, you are forced to write a test case for each unique group. This isn't so inconvenient in most cases, but every once in a while, you'll come across a scenario where the business logic requires a hugely varying number of tests.

JUnit testers often turn to a framework like FIT in this case because it lets you drive tests with tabular data. But TestNG provides a similar feature right out of the box. By placing parametric data in TestNG's XML configuration files, you can reuse a single test case with different data sets and even get different results. This technique is perfect for avoiding tests that only assert sunny-day scenarios or don't effectively verify bounds.

In Listing 5, I define a TestNG test in Java 1.4 that accepts two parameters, a classname and a size. These parameters verify a class hierarchy (that is, if I pass in java.util.Vector, I expect HierarchyBuilder to build a Hierarchy of 2).

Listing 5. A TestNG parametric test

package test.com.acme.da;

import com.acme.da.hierarchy.Hierarchy;
import com.acme.da.hierarchy.HierarchyBuilder;

public class HierarchyTest {
  /**
   * @testng.test
   * @testng.parameters value="class_name, size"
   */
  public void assertValues(String classname, int size) throws Exception{
    Hierarchy hier = HierarchyBuilder.buildHierarchy(classname);
    assert hier.getHierarchyClassNames().length == size: "didn't equal!";
  }
}

Listing 5 shows a generic test that can be reused over and over again with varying data. Think about this for a minute. If you had 10 different parameter combinations to test in JUnit, you would be forced to write 10 test cases. Each one would essentially do the same thing, only varying the parameters to the method under test. But with parametric tests, you can define one test case and then push your desired parameter patterns (for example) into TestNG's suite files. That's what I've done in Listing 6:

Listing 6. A parametric suite file in TestNG

<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd">
<suite name="Deckt-10">
  <test name="Deckt-10-test">
    <parameter name="class_name" value="java.util.Vector"/>
    <parameter name="size" value="2"/> 	
    <classes>  		
      <class name="test.com.acme.da.HierarchyTest"/>
    </classes>
  </test>  
</suite>

The TestNG suite file in Listing 6 only defines one combination of parameters for the test (class_name equal to java.util.Vector and size equal to 2), but the possibilities are endless. As an added benefit, moving the test data into the non-code artifact of an XML file means that non-programmers can also specify data.

Advanced parametric testing

While pulling data values into an XML file can be quite handy, tests occasionally require complex types, which can't be represented as a String or a primitive value. TestNG handles this scenario with its @DataProvider annotation, which facilitates the mapping of complex parameter types to a test method. For example, for the verifyHierarchy test in Listing 7, I've utilized an overridden buildHierarchy method that takes a Class type, asserting that Hierarchy's getHierarchyClassNames() method returns a proper String array:

Listing 7. DataProvider usage in TestNG

package test.com.acme.da.ng;

import java.util.Vector;

import static org.testng.Assert.assertEquals;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import com.acme.da.hierarchy.Hierarchy;
import com.acme.da.hierarchy.HierarchyBuilder;

public class HierarchyTest {

  @DataProvider(name = "class-hierarchies")
  public Object[][] dataValues(){
    return new Object[][]{
      {Vector.class, new String[] {"java.util.AbstractList", "java.util.AbstractCollection"}},
      {String.class, new String[] {}}
    };
  }

  @Test(dataProvider = "class-hierarchies")
  public void verifyHierarchy(Class clzz, String[] names) throws Exception{
    Hierarchy hier = HierarchyBuilder.buildHierarchy(clzz);
    assertEquals(hier.getHierarchyClassNames(), names, 
      "values were not equal");		
  }
}

The dataValues() method provides data values through a multidimensional array, which matches the parameter values of the verifyHierarchy test method. TestNG iterates over the data values and accordingly invokes the verifyHierarchy twice. The first time, the Class parameter is set to Vector.class and the String array parameter holds two "java.util.AbstractList" and "java.util.AbstractCollection" values as Strings. How's that for handy?

Why choose one?

I've talked about the features that differentiate TestNG for me, but there are a few more that aren't yet available in JUnit. For example, TestNG uses test groups, which can categorize tests according to features such as run times. It also works in Java 1.4 with javadoc-style annotations, as you saw in Listing 5.

As I said at the beginning of this column, JUnit 4 and TestNG are similar on the surface. But whereas JUnit is designed to hone in on a unit of code, TestNG is meant for high-level testing. Its flexibility is especially useful with large test suites, where one test's failure shouldn't mean having to rerun a suite of thousands. Each framework has its strengths, and there's nothing stopping you from using both in concert.

Resources

Learn

• "TestNG makes Java unit testing a breeze" (Filippo Diotalevi, developerWorks, January 2005): TestNG isn't just really powerful, innovative, extensible, and flexible; it also illustrates an interesting application of Java annotations.
• "An early look at JUnit 4" (Elliotte Rusty Harold , developerWorks, September 2005): Obsessive code tester Elliotte Harold takes JUnit 4 out for a spin and details how to use the new framework in your own work.
• Using JUnit extensions in TestNG (Andrew Glover, thediscoblog.com, March 2006): Just because a framework claims to be a JUnit extension doesn't mean it can't be used within TestNG.
• Statistical Testing with TestNG (Cedric Beust, beust.com, February 2006): Advanced testing with TestNG, written by the project's founder.
• "Rerunning of failed tests" (Andrew Glover, testearly.com, April 2006): A closer look at rerunning failed tests in TestNG.
• In pursuit of code quality: "Resolve to get FIT" (Andrew Glover, developerWorks, February 2006): The Framework for Integrated Tests facilitates communication between business clients and developers.
• JUnit 4 you (Fabiano Cruz, Fabiano Cruz's Blog, June 2006): An interesting entry on JUnit 4 ecosystem support.
• Code coverage of TestNG tests (Improve your code quality forum, March 2006): Join the discussion on integrating code coverage tools with TestNG.
• In pursuit of code quality series (Andrew Glover, developerWorks): See all the articles in this series ranging from code metrics to testing frameworks to refactoring.

Get products and technologies

• Download TestNG: Take TestNG for a test drive.
• Download JUnit: Find out what's new with JUnit 4.

Discuss

• Participate in the discussion forum.

About the author

Andrew Glover is president of Stelligent Incorporated, which helps companies address software quality with effective developer testing strategies and continuous integration techniques that enable teams to monitor code quality early and often. Check out Andy's blog for a list of his publications.