Unit Test Framework

The previous topic turned the runtime class path into a portable artifact and showed that what goes into it is a decision. That decision was bounded there by two paths: the compile class path and the runtime class path. This lesson makes the same decision for a third path. While writing tests, one more class path sits at hand — the path the test sees — and this path is not the same as the path production sees. Our question is: what does this third path carry, and what happens when a type that never appears on the production class path is missing?

The second half of the question is a separate topic, established in the M21/K02 Unit Testing and Test-Driven Development course: test classification, assertion writing, test naming — none of these are established again here. The one new question in this lesson is which class path the types the test needs to run stand on. Before answering that question, we first need to build by hand the framework itself, on top of which we will build a class path question — the runner that calls setup, body and teardown in sequence.

These two questions stand closer together than they appear. A framework applies a sequence that is never written in the source, over and over for every case; this repetition means a series of calls that appears nowhere in a file’s own source. For those calls to compile and run, they must be written somewhere — and that place is not the class path production code reads. Moving to the class path question without building the framework would mean measuring an empty abstraction; this is why the lesson first writes the calls by hand, then counts which type those calls need.

The Sequence the Framework Establishes

• TE1. Setup and teardown are each modeled here as a Runnable; a real framework’s test discovery (file scanning, name matching) is not built here — only the setup, body, teardown order and the way a failure is reported are modeled.
• TE2. Assertion failure is reported by throwing an AssertionError; the runner catches this and logs the case as “failed,” and otherwise writes “passed.” The teardown step runs under every condition — whether the body fails or not.

From the outside, what a test framework does amounts to making three calls in a fixed order: setup first, then the body, then teardown last. An assertion is a comparison and reports its failure with an exception; the framework catches this exception and marks the case as “failed,” and if it does not catch one, it writes “passed.” Setup runs again for every case — the state one case leaves behind does not leak into the next — and teardown runs whether the case passes or fails; otherwise one case failing would corrupt the next one’s setup. The runner below builds these three calls and the assertion by hand.

// Framework.java — a small runner that calls setup, body and teardown in sequence
public class Framework {

    static void runCase(String name, Runnable setup, Runnable body, Runnable teardown) {
        setup.run();
        try {
            body.run();
            System.out.println(pad(name) + "passed");
        } catch (AssertionError error) {
            System.out.println(pad(name) + "failed: " + error.getMessage());
        } finally {
            teardown.run();
        }
    }

    static void assertEqual(String field, int expected, int actual) {
        if (expected != actual) throw new AssertionError(field + " (expected " + expected + ", got " + actual + ")");
    }

    static String pad(String s) { return String.format("%-16s", s); }

    public static void main(String[] args) {
        int[] quantity = {0};
        runCase("store-filled",
                () -> quantity[0] = 10,
                () -> assertEqual("quantity", 10, quantity[0]),
                () -> quantity[0] = 0);

        int[] field = {0};
        Runnable setup = () -> field[0] = 5;
        Runnable teardown = () -> field[0] = 0;

        runCase("before-check", setup, () -> assertEqual("field", 5, field[0]), teardown);

        setup.run();
        runCase("after-check", () -> {}, () -> { teardown.run(); assertEqual("field", 5, field[0]); }, () -> {});
    }
}

store-filled    passed
before-check    passed
after-check     failed: field (expected 5, got 0)

The first case is a plain pass: setup makes quantity 10, the body reads and confirms it, teardown resets it to zero. The real question is in the second and third case. Both carry the same assertion — field must equal 5 — but one checks the assertion before teardown, the other after teardown. Teardown resets field to zero; this is why the same assertion passes in the first case and fails in the second. The runner itself does not know this distinction — it only makes the three calls in sequence and catches whatever exception the body throws. before-check and after-check each appear as one line in the output, and both are counted as “run” in the runner’s list; the difference between them can only be read in the result column.

The conclusion that follows is this: the order the framework calls is not a guarantee, it is a contract. The setup-body-teardown order always stays the same, but when the body reads — the state setup left behind, or the state teardown left behind — is in the hands of whoever writes it. The same assertion text measures two different truths when written in two different positions, and the framework does not arbitrate between them; it only records which one threw and which one did not. A case being counted as “run” does not mean that case was set up correctly — it only means the three calls were made in order and a result was produced. The test classification and assertion writing that M21/K02 established do not change here; the only thing that changes is that where the assertion is written inside the body determines the result.

The source of this distinction is the finally block: the teardown call runs independently of the body’s result, because the runner placed it inside finally, not outside the try block. The teardown step therefore works the same way both in a passing case like store-filled and a failing case like after-check — it resets field to zero. The difference is whether the assertion runs before or after this reset, and this is not information the runner itself holds — it is a decision made by whoever writes the body. The same holds for a real framework: the order in which setup and teardown are called is documented and guaranteed, but how the body uses that order — which moment, which state it reads — is not under the framework’s control.

The Test Class Path: A Third Path

• TE3. The distinction between the compile class path and the runtime class path is carried over from the previous topic (Build Tools) and is not rebuilt here; the one question this lesson adds is what a third class path carries.
• TE4. “Type count” here is a count of compiled .class files. Each of the three sources below contains no inner class, so each produces exactly one .class file; the count therefore lines up with the number of sources, but in general the type count can differ from the number of source files.
• TE5. Running “without the framework type” means a real class loader searches the given class path for that type and does not find it — the loader here is a URLClassLoader and is built on top of the platform class loader, so it does not inherit anything from the application’s own class path.

The runner above and the assertion helper inside it were part of the Framework class — but in a real program, a helper of this kind stands apart from the code under test. The measurement below actually compiles three separate classes: Store (production code), Assertion (the framework itself, just an equality check the test needs) and StoreTest (the test that uses both). Store compiles only on its own class path; Assertion compiles in a separate directory; StoreTest compiles against both. It then runs against two separate class paths: one contains all three, the other leaves Assertion out.

// ThirdPath.java — the test class path really carries extra types the production class path does not
import java.io.*;
import java.lang.reflect.InvocationTargetException;
import java.net.*;
import java.nio.file.*;
import java.util.*;
import java.util.spi.ToolProvider;

public class ThirdPath {

    static Path write(Path dir, String name, String source) throws IOException {
        Files.createDirectories(dir);
        Path p = dir.resolve(name + ".java");
        Files.writeString(p, source);
        return p;
    }

    static boolean compile(Path output, List<Path> classPath, List<Path> sources) throws IOException {
        Files.createDirectories(output);
        List<String> arg = new ArrayList<>(List.of("-d", output.toString()));
        if (!classPath.isEmpty()) {
            StringJoiner sj = new StringJoiner(File.pathSeparator);
            for (Path p : classPath) sj.add(p.toString());
            arg.add("-classpath");
            arg.add(sj.toString());
        }
        for (Path s : sources) arg.add(s.toString());
        ToolProvider compiler = ToolProvider.findFirst("javac").orElseThrow();
        StringWriter silentOutput = new StringWriter();
        return compiler.run(new PrintWriter(silentOutput), new PrintWriter(silentOutput),
                arg.toArray(new String[0])) == 0;
    }

    static String run(List<Path> classPath, String mainClass) {
        URL[] url = new URL[classPath.size()];
        try {
            for (int i = 0; i < url.length; i++) url[i] = classPath.get(i).toUri().toURL();
            try (URLClassLoader loader = new URLClassLoader(url, ThirdPath.class.getClassLoader().getPlatformClassLoader())) {
                Class<?> cls = Class.forName(mainClass, true, loader);
                cls.getMethod("main", String[].class).invoke(null, (Object) new String[0]);
                return "ok";
            }
        } catch (InvocationTargetException e) {
            return e.getCause().getClass().getSimpleName();
        } catch (Throwable t) {
            return t.getClass().getSimpleName();
        }
    }

    static long typeCount(Path dir) throws IOException {
        try (var walk = Files.walk(dir)) {
            return walk.filter(p -> p.toString().endsWith(".class")).count();
        }
    }

    public static void main(String[] args) throws Exception {
        Path root = Files.createTempDirectory("thirdpath");
        Path source = root.resolve("source");
        Path prodDir = root.resolve("production"), frameworkDir = root.resolve("framework"), testDir = root.resolve("test");

        Path sStore = write(source, "Store",
                "public class Store {\n"
              + "  private int quantity;\n"
              + "  public Store(int initial) { quantity = initial; }\n"
              + "  public void add(int amount) { quantity += amount; }\n"
              + "  public void reduce(int amount) {\n"
              + "    if (amount > quantity) throw new IllegalStateException(\"insufficient quantity\");\n"
              + "    quantity -= amount;\n"
              + "  }\n"
              + "  public int quantity() { return quantity; }\n"
              + "}\n");
        Path sAssertion = write(source, "Assertion",
                "public class Assertion {\n"
              + "  public static void equal(int expected, int actual) {\n"
              + "    if (expected != actual) throw new AssertionError(\"expected \" + expected + \", got \" + actual);\n"
              + "  }\n"
              + "}\n");
        Path sStoreTest = write(source, "StoreTest",
                "public class StoreTest {\n"
              + "  public static void main(String[] a) {\n"
              + "    Store s = new Store(10);\n"
              + "    s.add(5);\n"
              + "    s.reduce(3);\n"
              + "    Assertion.equal(12, s.quantity());\n"
              + "    System.out.println(\"case passed\");\n"
              + "  }\n"
              + "}\n");

        compile(prodDir, List.of(), List.of(sStore));
        compile(frameworkDir, List.of(), List.of(sAssertion));
        compile(testDir, List.of(prodDir, frameworkDir), List.of(sStoreTest));

        System.out.println("type count on the production class path : " + typeCount(prodDir));
        System.out.println("type count on the test class path       : "
                + (typeCount(prodDir) + typeCount(frameworkDir) + typeCount(testDir)));

        System.out.println("full test class path run                 : " + run(List.of(prodDir, frameworkDir, testDir), "StoreTest"));
        System.out.println("without framework type (production+test) : " + run(List.of(prodDir, testDir), "StoreTest"));
    }
}

type count on the production class path : 1
type count on the test class path       : 3
case passed
full test class path run                 : ok
without framework type (production+test) : NoClassDefFoundError

The numbers confirm the claim. When Store compiles and runs on its own, the production class path carries exactly one type. The full set needed to compile and run StoreTest is three types: the production code, the framework itself, and the test itself. These two extra types — Assertion and StoreTest — never appear anywhere in the production program; they do not need to go into the distributed executable output, and if they do, it grows for no reason. The test class path is therefore not a superset of the production class path, it is a separate path: it carries types production does not need, and it also covers the types production does carry.

The direction is one-sided. Compiling StoreTest requires putting Store and Assertion on the class path, because the test names both of them by name in its source. Compiling Store, on the other hand, needs neither Assertion nor StoreTest — the production code was written with no knowledge that test types exist. The distinction the packaging topic established applies here without change: if an executable output is produced from the production class path, no type like Assertion or StoreTest ever enters it; if one does, that output is bigger than it needs to be and carries more types than it should. The test class path being a third path also means it must not mix into the production output — the two paths mixing together is the most common form a leaked dependency takes.

Placing the three class paths side by side makes the ordering clear: the compile class path asks for Store together with Assertion to compile StoreTest, the runtime class path carries only Store and knows nothing of Assertion, and the test class path carries all three at once. All three paths can point to the same .class files on the same disk; the difference between them is not in the files’ content, it is in which directories are given to which command. A type being “required on the test class path” means that type was never written in the production code, and arose only from the test’s own need — and this distinction is invisible without actually building the two paths separately.

In this lesson, Assertion and StoreTest depended on each other at compile time, but Assertion being present on the runtime class path at run time was a separate requirement from compiling — and the last line indeed separates the two. The next lesson carries this observation one step further: when the type the test needs replaces a type the code under test already uses, the same compiled test can give two different results across two different runtime class paths.

The last two lines show this by actually running. When StoreTest runs on the full class path — production, framework and test types together — the Assertion.equal call is found and the case returns “ok.” The same StoreTest.class file, run this time only with the production and test directories, without the framework directory, has the loader find the Assertion name nowhere, and it throws NoClassDefFoundError. The file itself did not change, it was not recompiled; only the class path given to it narrowed. This is the most concrete proof of the claim that “the test class path is a third path”: the type the framework supplies is required both for the test to compile and for it to run, but is required for neither compiling nor running production.

Summary

• The test framework was modeled here as a small runner that calls setup, body and teardown in sequence and catches assertion failure with an exception.
• The same assertion passed when checked before teardown and failed when checked after teardown; the runner counted both as “run,” and the order itself did not guarantee correctness.
• The production class path carried a single type; the test class path carried the framework’s and the test’s own types in addition to that same type — three types in total.
• The same compiled StoreTest.class, run on a class path missing the framework type, gave NoClassDefFoundError without being recompiled; the file did not change, only the class path changed.
• Test classification and assertion writing were established in the M21/K02 Unit Testing and Test-Driven Development course and were not repeated here.

Next Step

In this lesson, the types that entered the test class path were types we wrote by hand ourselves, standing entirely apart from the code under test. But a dependency the test needs can end up having to take the place of a type the production code already uses — a type that carries the same name, provides the same signature, but behaves differently. The next lesson runs the same compiled test once against a class path where the real type stands, and again against another class path where a fake type carrying the same name stands, and counts that the test gets two different results without being recompiled.