Continuous Testing Tools (not only) for Java – Infinitest vs JUnit Max

Tools like Infinitest and JUnit Max run Java unit tests as early as possible, immediately after a relevant code change has taken place. This way you get an immediate feedback and errors and their causes can be identified immediately. Especially when doing test driven development (TDD), this saves you a lot of keystrokes, because you don’t have to manually run your test cases all the time. Test runs are ordered to let those tests with the highest risk of failure run first.

Infinitest

Until recently, Infinitest was distributed under a commercial license and even now, Google’s first hit points to a page where you can buy Infinitest. Since November 2010, it is available as an open source project, hosted on github and licensed under the GPLv3. Whereas the previous version 4 was IDE neutral and came with a Swing UI, the current versions integrate with Eclipse (Update Site: http://infinitest.github.io ) and IntelliJ. The current stable release (5.1.84) supports JUnit. Experimental support for TestNG currently (January 2011) only exists in the GIT source repository.

The Eclipse Plugin displays the current test status in terms of a green/yellow/red bar in the bottom status bar. The tests are running continuously in the background and trigger visual hints, but don’t actively interfere with the development flow. The individual test results are listed in the problems view, and error markers appear in the editor view at the affected code lines and also on relevant files in the package explorer. If an error is caused by an exception thrown by  some code under test, Infinitest offers a Quick Fix to show you the stack trace and find out which test was failing.

Having only a minimal global settings page, the Eclipse Plugin gives a very lean impression. In fact, all essential settings are stored in the infinitest.filters and infinitest.args files at the Eclipse project root, as described in the user guide . These files can be put under version control, allowing the whole team to share their settings. Using these files you can define filters to ignore long running tests via regular expressions and you can configure special VM settings like additional heap space.

All in all, Infinitest gives a quite mature impression. Even with larger workspaces, the tool works well. One interesting source is the blog entry about Infinitest written by Alex Ruiz.

Since testing is done based on changes of class files, there is a good chance that alternative languages like Groovy or Scala will get better support in the future. During some experiments with Groovy test cases, the classes were detected as JUnit test classes, but errors during the test run could not be traced back to line numbers. Also the background process got stuck at that point for about a minute. The execution of Scala tests seems to work.

JUnit Max

The JUnit Max Eclipse Plugin was developed by Kent Beck and can be licensed as a yearly subscription for $100 per developer. As name and author would suggest, there is only support for the JUnit test framework, not for TestNG.

From the functional perspective, there are only some small differences between JUnit Max (currently at version 1.2.22) and Infinitest. They share the signal light in the status bar and errors are reported in the problems view and markers placed at the corresponding bits of code.

In addition, it is possible to revert code changes that have been made since the last known green state. Settings that can be configured via preference dialogs and are much more extensive than in Infinitest. Developers can disable continuous testing at the project and package levels. Unfortunately, settings are stored internally within Eclipse instead of inside the project folder, so that you cannot add them to version control and share them between team members.

Alternative programming languages are not supported by JUnit Max. Corresponding tests are ignored completely. Also changes on code under test written in other languages are not detected, even if the code is covered by Java JUnit tests. Sadly, the plugin always stopped working after a short time showing an EOFException in the console view, until the next restart of Eclipse. This seems however to be a local problem with my machine (it’s always the same) – my collegues didn’t have that problem.

What more to say

Finally I should mention that using any of these tools cannot replace the additional full test run before checking in, even if you didn’t filter anything. There are cases, where the change detection algorithms reach their limits, for example if the code uses reflection.

Both tools can help increase productivity, especially when doing TDD. My personal favorite is Infinitest, in particular because of the more flexible and versioning friendly filtering options.