nit Testing Framework
nit Testing FrameworkTopics of the lecture:
JUnit: The rationale
JUnit: The recipe
JUnit: The practice
JUnit: The extensions
JUnit: The test–driven development
Note: the examples used in this lecture are based on the JUnit version 3.8.1. The 16 Feb 2006 version 4.0 is significantly different and uses Java 1.5 features at its core. The tests prepared for pre-JUnit4.0 can be run through the latest version using special adapter classes provided with JUnit 4.0.: or (preferred) write for JUnit 4 by looking at the modern documentation.
Unit testing is the only kind of testing which can and should be exercised by the code writers themselves. Without making sure that smallest individual pieces of code (units) are correct, it is not possible to attain correctness of an integrated system. The traditional unit testing techniques like debugging, however, are inevitably disruptive (even when carried out with the best tools), because they are not like coding, they require a different mind set. Also, they cannot be easily composed to execute multiple tests at a time, and they require human attention to assess their results.
Testing with the JUnit framework offers two remarkable advantages: the business of writing tests are absolutely the same as the coding itself — the programmer does not have to "click a switch" to stop writing the code, and start writing the tests, the two go together hand-in-hand. The code and the test harness are kept synchronized, and often writing tests precedes writing code. Secondly, the acknowledging of the test results is as easy as hearing a bell ring (or the silence, of their absence). This effect is achieved through a systematic and very effective use of assertions which is a way (delivered as a language feature, or simple and short programming solution) to specify the desired outcome of a test and compare it with the actual outcome. The assertion succeeds if the two are identical, and fails if they are not. The interpretation of the test results is immediate.
JUnit allows the programmer to group the unit tests like a file system. This test organization gives an easy way to do selective testing, which in turn makes it easy to perform regression tests to ensure that new code modifications have not introduced new faults.
For every class to be tested (which belongs to the production code) a special testing class is written (it belongs to the test code).
Let's consider the following mathematical example. Suppose, I am writing a class Triangle which allows me to define a triangle on a two-dimensional plane, find out all major geometrical characteristics of a triangle object (like the length of its sizes, its angles, its area etc) , and also find out whether different triangle objects are similar (in the geometrical sense), or equal ("congruent"). If I knew some of the advanced mathematics (like projective geometry), I would try to define how to add two triangles together or something else exotic, but we shall not go too far in this example.
All testing of the Triangle class will done in the TestTriangle class, which will extend the JUnit TestCase class. There will be reciprocity between methods of the production code class, and its testing counterpart. Namely, every method in the production code which is responsible for some testable functionality will be matched by one (or more) testing method in the test code. For the Triangle class, we can present this correspondence in the following table:
| Triangle.java method |
TestTriangle.java method |
|---|---|
| Triangle() | testCreateTriangle() |
| area() | testArea() |
| smallestAngle() | testSmallestAngle() |
| shortestSide() | testShortestSide() |
| isDegenerate() | testDegeneracy() |
| isSimilar() | testSimilarity() |
| equals() | testEquals() |
| ……… | ……… |
| doWhatever() | testWhatever() |
Emphasize once again: It's not important to JUnit that a method name in the test code matches a method name in the production code: the only requirement that the method in the test code begins with "test".
JUnit 4.x: this
requirement has changed:
in JUnit 4.x, you only need to precede the method definition with the
annotation @Test
But it is often a good starting point to have indicative, matching names. Another method is to create a test for each property or specification, and name the tests accordingly: one specification may be implemented across several mehods, so cannot be tested at the finest unit level. But since we are concentrating on unit testing, i.e. testing methods, naming a test for each method is appropriate.
Every test method in the test code will contain one or more assert calls defined in the Assert class of the framework. assert's are polymorphic methods which represent the atoms of unit testing. They check the equality of two items of data (the actual and the required return values of a method, or the actual state and the required state of an object), and make it clear whether the equality holds (success) or not (error or failure). Some of the JUnit assertions are the following:
| assert methods |
|---|
| assertEquals([String], expected, actual) |
| assertNull([String], Object) |
| assertSame([String], expected, actual) |
| assertTrue([String], boolean) |
| fail([String]) |
Some of the above assertions have Not counterparts, eg assertNotSame() and assertFalse(). The assertEquals() assertion has an important variant assertEquals(String, float, float, float tolerance), which is recommended if the compared values are floating point numbers (floats or doubles). The same approach is recommended, of course, for writing relevant methods in the production code (isSimilar(), equals() methods in Triangle). Note also, that the fail() assertion is provided to immediate abort the test, to mark a section of code that should not be reached in the normal execution thread (e.g., after an exception is expected).
JUnit distinguishes between failures and errors. A failure is anticipated and checked for with assertions. Errors are unanticipated problems like an ArrayIndexOutOfBoundsException. A test method can be declared to throw an exception; it can be tested with the assertion fail(), but a better way is to let JUnit to catch it and report as error.
When one of the assert calls fails, that test method is aborted (no remaining assertions will be executed during the run), but this is not a failure of the test, this is the success — we found a bug (if, of course, our test was correct)! The test methods which follow the one with the failed assertion will be executed during the same run, such that all assertion failures (or, errors) will be identified and flagged up. After the run is complete, we can examine the reported messages. Since different functionalities are isolated in different tests, to find the cause of a particular failure is quite easy.
A testMethod() can exercise all or part of the production code functionality. If the number of equivalence classes and boundary values among the inputs which a method in the production code can take (or, among the object's states at which the method can be called) is significant, different types of inputs (or, object states) can be tested in separate test methods.
Each test can be run independently from every other test (from the same test class): this is important, since it allows us to tests in any order, in any selection, any time. Therefore, one has to be able to reset some parts of the pre-run setup, and clean up after a test has run. The JUnit provides two fixtures: methods which can be overridden to achieve the desired setup for every individual test method. The setUp() method allows to initialize a set of data on which a test can be run; the tearDown() method releases any permanent resources you allocated in setUp() (eg, disconnect database, destroy a socket etc). If there are more than one test method in the test class, the setUp() method will be called before each of the method runs, and tearDown() will be called after each of the method runs. This is how the independence of individual tests is achieved. Of course, you can define some parameters of the testing environment globally for all the tests which will be performed (by making appropriate initializing outside the fixture). This brings us to the notion of the test suite.
Thus, the test class (for our Triangle example) may look like this:
The @Test, @Before, @After notations are
only for JUnit 4.
In JUnit 3 these are omitted altogether.
package comp2100.junit;
import static org.junit.Assert.*;
import org.junit.Test;
public class TestTriangle extends TestCase {
@Test
// @Test is JUnit 4 notation: marks a testcase (Java 5)import org.junit.Test;
public TestTriangle(String name) { super(name); }
@Before
protected void setUp() { /* number of triangles created */ }
@After
protected void tearDown() { … }
@Test
public void testTriangle() { … }
@Test
public void testSmallestAngle() { … }
@Test
public void testSharpestVertex() { … }
@Test
public void testSmallesSide() { … }
@Test
public void testSimilarity() { … }
@Test
public void testEquals() { … }
@Test
public void testArea() { … }
@Test
public void testDegeneracy() { … }
}
In more detail: if the Triangleclass contain a constructor Triangle(Point p1, Point p2, Point p3) (here the class Point model a two-dimensional point; it can also be — must! — tested with JUnit).
public Triangle(Point p1, Point p2, Point p3) {
vertex1 = p1;
vertex2 = p2;
vertex3 = p3;
side1 = vertex2.distance(vertex3);
side2 = vertex3.distance(vertex1);
side3 = vertex1.distance(vertex2);
}
public void testCreateTriangle() {
Assert.assertNotNull("What?! no triangles!?", new Triangle(p1, p2, p3);
}
(assuming that Points p1, p2, p3 are defined as the TestTriangle fields, or in the setUp() method).
Another method isSimilar() in Triangle is defined as
public boolean isSimilar(Triangle t) {
boolean s = false;
s = (Math.abs(smallestAngle() - t.smallestAngle()) +
Math.abs(largestAngle() - t.largestAngle()) < tolerance);
return s;
}
is tested in TestTriangle like this:
public void testSimilarity() {
Point q1 = new Point(-1, 9);
Point q2 = new Point(3, 0);
Point q3 = new Point(0, 3);
Triangle t1 = new Triangle(new Point(1, 1), new Point(1, 2);
Triangle t2 = new Triangle(q1, q2, q3);
Triangle t3 = new Triangle(q2, q3, q1);
Triangle tr = new Triangle(new Point(-1, -1), new Point(1, 1), new Point(1, 3));
assertTrue(t1.isSimilar(tr));
assertTrue(t2.isSimilar(t3));
assertTrue(t2.isSimilar(new Triangle(q3, q1, q2)));
}
It's always more informative to use "commented" versions of assertions, and include into String part a useful information regarding the tested method. That is, the last assert could be modified as follows:
assertTrue("Scaling and shifting does not change similarity, does it?",
t1.isSimilar(tr));
Test suites are not needed for JUnit 4 run with eclipse.
In JUnit 3: after the test methods are defined, one has run them. The JUnit framework provides two ways: static and dynamic. In static way, you override runTest() method inherited from TestCase. With the use of an anonymous inner class, TestCase object can be instantiated as follows:
TestCase testSim = new TestTriangle("test similarity") {
public void runTest() {
testSimilarity();
}
};
(here each test is given a name like "test similarity" for identification if it fails).
The dynamic way relies on the reflection mechanism to implement runTest(). The name of the test object should be tied up to the name of the test method:
TestCase testArea= new TestTriangles("testArea");
By using the test object name, JUnit (via reflection package) dynamically finds and invokes the test method.
Finally, all the test method can be run together by adding the corresponding test objects into a test suite. In JUnit this requires the definition of a static method called suite(), which is like a main() method for running tests. The actual test suite is a TestSuite object. The suite() methods adds all the desired test objects to the test suite, and returns it. Bypassing the creation of the test objects (apart from the two above), the suite() method is defined as follows:
public static Test suite() {
TestSuite suite = new TestSuite("Triangle Tests");
suite.addTest(testSim);
suite.addTest(testArea);
suite.addTest(new TestTriangle("testShortestSide"));
suite.addTest(new TestTriangle("testSmallestAngle"));
suite.addTest(new TestTriangle("testEquals"));
/* and so on for all the methods the test class */
return suite;
}
Creating the test classes which can be run separately is a good thing. But JUnit also offers the way to combine different test cases into larger collections, which can be organized according to the production code structure. These collections of tests are also suites. Because both TestCase and TestSuite implement the JUnit interface Test, a thing which you can do to a TestCase object you also can do the the TestSuite object. Thus, for instance, you can add a suite to a suite. When writing up Triangle, I in addition defined classes Point and PlanarVector which too should be tested in the way similar to Triangle. So, I created test classes TestPoint and TestPlanarVector. Imagine also, that in my previous life (which I remember vaguely), I was an algebraist [Alexei Khorev], I have a Complex class as a left over which I may re-use. I have a test class for it, too, TestComplex. All these tests can be combined into one test suite. I can define this suite in a separate class, and define the main() method which will invoke one of the Runner (see below) classes run() method:
package comp2100.junit;
import junit.framework.Test;
import junit.framework.TestSuite;
public class Comp2100JunitTests {
public static void main(String[] args) {
junit.textui.TestRunner.run(suite());
}
public static Test suite() {
TestSuite suite = new TestSuite("All Comp2100 JUnit Tests");
TestSuite suite1 = new TestSuite("Triangle Tests");
suite.addTest(new TestSuite(comp2100.junit.TestComplex.class));
suite1.addTest(new TestSuite(comp2100.junit.TestTriangle.class));
suite1.addTest(new TestSuite(comp2100.junit.TestPoint.class));
suite1.addTest(new TestSuite(comp2100.junit.TestPlanarVector.class));
suite.addTest(suite1);
return suite;
}
}
Note that the suite is defined in such a way that "geometrical" and "algebraic" parts are in different sub-suites as it should be. This capability to hierarchically organize the test collection to mirror the hierarchy of the production code (or, following another principle) results in a great flexibility with which the test suites can be carried out.
Finally, we have to actually run the tests.
This can be done using the JUnit
TestRunner classes,
or you can run them directly from within eclipse.
The older framework provides both the command line
mode, and the graphical user interface (there is a Swings
based GUI runner, and — if you can put up with its ugliness —
a AWT runner) for running tests. To run a test on the command line
execute the following command:
java junit.textui.TestRunner comp2100.junit.TestTriangle
will generate an output on the StdOut which may look like
........ Time: 0.132 OK (8 tests)
(if all tests are successful), or like
....F..F.. Time: 0.106 There were 2 failures: 1) testSimilarity(comp2100.junit.TestTriangle)junit.framework.AssertionFailedError at comp2100.junit.TestTriangle.testSimilarity(TestTriangle.java:50) 2) testDegenerate(comp2100.junit.TestTriangle)junit.framework.AssertionFailedError at comp2100.junit.TestTriangle.testDegenerate(TestTriangle.java:69) FAILURES!!! Tests run: 8, Failures: 2, Errors: 0
(if some tests fail).
To bring up the JUnit GUI, use this command
java junit.swingui.TestRunner TestTriangle
which will open a GUI window
The all successful tests are marked by the green progress bar, but if at least one test fails, the bar turns red:
You have to restart the graphical interface each time you change the code. This can be avoided by using LoadingTestCollector. This feature can be disabled by unchecking the "Reload classes every run" checkbox.
| "Keep the bar green to keep the code clean" |
|---|
The GUI interface allows the user to select only part of the whole suite (a particular sub-suite, or a single test) and run it separately.
On our student system we have a useful shell script which makes running the JUnit test even easier. It's called (well, you guessed it) junit, and it can be run with three options: -text, -swing and -awt (I advise you not to exercise the third option). If you need to run JUnit tests on the class MyTests, type in the command:
junit -text MyTests
If you want to run Swing GUI, use the command
junit -swing &
and select the desired test using the menu.
JUnit in its heart is a simple library. One can say, that it abides the UNIX tools philosophy: one, preferably simple tool for a particular functionality, and for nothing else. If one wants to add more functionality for testing, this should come as a separate tool, not as an extension of the JUnit itself.
When the production code unit depends on other parts of the system, than to test the unit in isolation is impossible, but writing the "fat" test to give it enough context is often prohibitive, and defeats the very idea of unit testing, since it introduces strong coupling between into the process which is meant to be localised. A feasible solution to this problem involves the well known trick of using stubs, which in the unit test context are called Mock Objects. A mock object is a replacement of a real object in the context of testing. The code under test only refers to the outside object by its interface. The mock object is an instance of a class which implements this interface in the manner which makes it usable in a unit test, but without introducing complexities of the entire system. One can say, that Mock Objects is an OO version of method stubs from structural programming. To ease the task of writing the mock object classes, one can use the Easy-Mock class library, a Java API for creating mock objects dynamically.
JUnit does not take parameters, alas! The ability to manage test parameters come with additional coding which has little to do with the test creation. JTestCase helps in seperating test data from test code. You can organize all your test cases of multiple unit tests into one data file — an XML file, and bulk-load them into memory via sets of easy-to-use APIs that JTestCase provides. In a word, JTestCase provides a way for java unit tests to be test-case-oriented and full-test-automatable.
A remarkable tool: It generates skeletons of TestCases based on your application source code! Even less for you to code! Works with major modern IDEs, and is synchronised with the code refactoring.
We talked about path and branch coverages in the previous lecture. The JUnit framework does nor provide automatic means to select a set of tests which will exercise a particular execution path; this remains a programmer's "prerogative". The test coverage can be characterized by different kind of metrics: a statement coverage metrics (how many times during the test a particular statement was executed), a branch (decision) coverage (number of times a particular branch was followed during a test), path coverage, etc. Tools which provide this sort of information (the metrics), are available:
The utmost form of practicing the JUnit is the test driven development, when writing the tests does not simply precedes writing the production code, but actually guides it. The test writing helps to aks right questions about what the item of interest (class, method) in the production code could be and what would be its behavoiur. This approach can be exercised at the small (well, again unit) scale design, but this is exactly where the now fashionable agile development claims its rights. You can learn more about TDD from the book by K. Beck, "Test-Driven Development".
The JUnit design represents a remarkable case study for application of the so called Design Patterns in object-oriented design. The UML diagram which describes the framework design shows the use of four standard design patterns: Composite, Adapter, Template Method and Command.
Copyright © 2006,2007,2008 The Australian National University
$Revision: 1.4 $ $Date: 2009/03/26 03:01:52 $ $Author: cwj $
Feedback & Queries to
comp2100@cs.anu.edu.au