Parameterized Tests

In the previous lesson, a single test body ran against three separate class paths, but the same three numbers were used in every run. In this lesson the body stays fixed, and the side that changes is the data: instead of writing the body three times to run the same check with three separate inputs, we write the data set as a declaration and build the body once. The question is: when a body is called against an N-row data set, how many separate results does it produce, how many fail, and how do we know which row is the one that failed?

The second half of this question looks separate at first glance, but it is not: running the same body N times assumes that the N runs are independent of each other. Because the body is written once, one row’s setup passes through the same code as the next row’s setup — and whether that code touches a shared variable never shows up in the data set itself. This lesson first counts three independent results with a correctly built body, then runs the same data with a broken setup and shows when the assumption collapses.

The data set itself is kept to three rows here, but the number itself is not what matters — what matters is that the same body can be called N times regardless of what N is, and that each call can produce its own result. Three rows was chosen because it is the smallest number that can still be traced by hand; the same setup bug surfaces the same way in a hundred-row data set, only tracing it gets harder.

The Data Set Stands in a Declaration, the Body Is Written Once

• TE11. The data set is declared here as an array of records (Row[]). Real frameworks’ mechanism for reading a data source from a file, an annotation or a method reference is not built here — the data stands directly in the source code.
• TE12. The body here is a loop that walks the data array; the same three rows are called once each, in the same order, each with its own input. The call count equals the row count.

The ROWS array below carries three rows; each row holds an initial quantity, a reduction and the expected result together. The independentRun body processes these three rows in order and builds its own Store instance for each row — setup lives inside the row and runs again on every turn of the loop. Below, the same data set is also run with a broken version where setup is moved outside the loop; the difference between the two is this lesson’s real subject.

// DataSet.java — one body, a run multiplied by the data set, and shared state's order dependency
import java.util.*;

public class DataSet {

    static class Store {
        private int quantity;
        Store(int initial) { quantity = initial; }
        void reduce(int amount) { quantity -= amount; }
        int quantity() { return quantity; }
    }

    record Row(int initial, int reduce, int expected) {}

    static final Row[] ROWS = {
        new Row(10, 3, 7),
        new Row(5, 2, 3),
        new Row(8, 8, 0),
    };

    // -- body written once, each of the N runs builds its own Store ------------------------
    static List<String> independentRun(Row[] rows) {
        List<String> log = new ArrayList<>();
        int failed = 0;
        for (int i = 0; i < rows.length; i++) {
            Row r = rows[i];
            Store s = new Store(r.initial());
            s.reduce(r.reduce());
            boolean passed = s.quantity() == r.expected();
            if (!passed) failed++;
            log.add("row " + i + ": " + (passed ? "passed" : "failed, actual=" + s.quantity()));
        }
        log.add("failed count: " + failed);
        return log;
    }

    // -- setup mistakenly moved outside the loop, Store is shared across runs --------------
    static List<String> sharedRun(Row[] rows) {
        List<String> log = new ArrayList<>();
        int failed = 0;
        Store s = new Store(rows[0].initial());
        for (int i = 0; i < rows.length; i++) {
            Row r = rows[i];
            s.reduce(r.reduce());
            boolean passed = s.quantity() == r.expected();
            if (!passed) failed++;
            log.add("row " + i + ": " + (passed ? "passed" : "failed, actual=" + s.quantity()));
        }
        log.add("failed count: " + failed);
        return log;
    }

    public static void main(String[] args) {
        System.out.println("-- independent run: each row builds its own Store --");
        independentRun(ROWS).forEach(System.out::println);

        System.out.println("-- shared run, original data order --");
        sharedRun(ROWS).forEach(System.out::println);

        Row[] reversedRows = { ROWS[2], ROWS[1], ROWS[0] };
        System.out.println("-- shared run, same data, order reversed --");
        sharedRun(reversedRows).forEach(System.out::println);
    }
}

-- independent run: each row builds its own Store --
row 0: passed
row 1: passed
row 2: passed
failed count: 0
-- shared run, original data order --
row 0: passed
row 1: failed, actual=5
row 2: failed, actual=-3
failed count: 2
-- shared run, same data, order reversed --
row 0: passed
row 1: failed, actual=-2
row 2: failed, actual=-5
failed count: 2

independentRun passes all three rows. This is the expected result, because each row builds its own Store from scratch — all three are unaware of each other, none touches another’s state. The body was written in one place (the loop body), but ran three times and produced three separate results. The failure count (failed count) and each row’s own status (row i: passed/failed) are recorded separately in the log — when a row fails, which row it was and the value it returned stay on record, not just a single total number.

This is a different way of counting from the “one assertion” idea in the previous lesson. There, one test produced one result; here, one body is called N times and gives rise to N separate assertions. A three-row data set means three separate assertions, and all three must be evaluated independently — one failing while the other two pass is still a meaningful result, because each row tests its own input on its own. Writing the body once is a shorthand here, reducing repetition in the source text; but it does not change the fact that there are three separate calls in the compiled or running program. The ROWS array and the independentRun/sharedRun methods are not outside the distinction the previous lesson established either — neither of them is written in production code, both stand only on the test class path.

Shared Setup, an Order-Dependent Result

• TE13. Which row a failure came from is accumulated here as a list element. Real frameworks’ report format (file name, line number, stack trace) is not built here; the record is only the row index and the resulting value.
• TE14. The shared state here is a single Store instance moved outside the loop. Where in the loop setup runs — on every turn, or once — determines whether the rows are independent of each other.
• TE15. Order dependency is shown here with a reversed copy of the same data set. In a real test suite, the order in which rows (or tests) run is usually undocumented and can change from run to run; this lesson samples only two possible orders.

sharedRun carries the same three rows, the same assertion — the only difference is that Store is built just once, outside the loop. This models a bug where setup should run again for every row but mistakenly runs only for the first row: the s object is built from the first row’s initial value, and then every row’s reduce call accumulates on the same object. In the original order, the first row passes — because the shared Store was built with exactly that row’s expected initial value — but the second and third rows no longer see their own expected initial state, they see the state the previous rows left behind, and both fail.

Tracking the numbers makes the difference clear. The shared Store is built with 10, which is ROWS[0].initial(). The first row calls reduce(3), the quantity becomes 7, the expected value is also 7 — it passes. The second row never sees its own initial value of 5; instead it sees the 7 the first row left behind, and calling reduce(2) makes the quantity 5, which fails because the expected value is 3. The third row also does not see its own initial value of 8, but the 5 the second row left behind; calling reduce(8) drops the quantity to -3, far from the expected 0. Each row’s own data is correct, each row’s own arithmetic is correct — what is wrong is which state they start from.

The real demonstration is in the last two runs. The same three data rows are run again with only their order reversed. The result again takes the shape “one row passes, two fail” — but the row that passes changes. In the original order, the passing row was the data set’s first row (10 → 7); in the reversed order, the passing row is whichever row arrives first in that order (8 → 0). The data set itself did not change, none of the rows’ content was edited — only which row was processed first changed, and this single change flipped which row passes from top to bottom. Whether a row passes or not is no longer tied to that row’s own data, but to its position.

The same tracing can be done in the reversed order. The shared Store is now built with 8 — the initial value of the third row, the first element of the reversed array. That row calls reduce(8), the quantity becomes 0, the expected value is also 0 — it passes. The second data row, now arriving second, does not see its own initial value of 5, but the 0 the previous step left behind; calling reduce(2) makes the quantity -2, far from the expected 3, and it fails. Finally, the first data row does not see its own initial value of 10, but -2; calling reduce(3) makes the quantity -5, far from the expected 7, and it fails. In both runs, only the first processed row passes — because the shared Store is built with exactly that row’s expected initial value — and the rest inherit the state the previous rows left behind and fail. Which row is “processed first” depends on the data array’s order, not on the row’s own content.

This is a concrete source of the flaky test concept established in the M21/K02 Unit Testing and Test-Driven Development course: state shared between tests (or, as here, between the same test’s rows) makes the result dependent on run order. The concept itself was established there; what is new in this lesson is where the source stands — whether setup runs inside or outside the loop is a decision that never shows up in the declared data set itself. It is not possible to recover this decision by looking at the two runs’ logs; it only becomes visible by changing the data order and running again.

This is why most real test frameworks deliberately shuffle the order in which rows or tests run: two separate runs giving the same order every time keeps a bug arising from shared state hidden. If the order stays fixed on every run, a divergence like the one this lesson showed in the second run might never appear — the suite says “passed,” only because the order is always the same. Once the order is shuffled, the same buggy setup passes sometimes one row, sometimes another, and the total failure count can appear to change from run to run. This variability is the symptom of the bug — a sign that it must be sought not in the test itself, but in state shared between tests.

The rule that follows is this: parameterization multiplies a body, but it does not guarantee that the multiplied runs are independent of each other. What guarantees independence is each run executing its own setup on its own — as in independentRun. When setup runs once and is shared across N runs, no matter how correctly the data set is written, the result becomes dependent on the data set’s order. A row passing no longer shows that the row is correct, only that the row’s position happened to fit.

The root of the bug is an aliasing problem: where two separate rows expect two separate Stores, a single Store reference is shared between them. This sharing is not visible in the source text — the name s appears on the same line on every turn of the loop, the only thing that changes is which Row record is being processed. The variable’s scope (outside the loop) and its intended lifetime (a single row) are separated here, and this separation cannot be seen from the source text; it is seen only by running with several rows, in several orders. This observation brings the course’s measuring question down to the level of test data: which scope a setup runs in is, just like which class path a type stands on, a decision that is not written explicitly in the source text but that determines the result.

It would also be misleading to think that moving setup outside the loop has no cost — quite the opposite, that move is itself usually the result of chasing some gain. independentRun builds a new Store for every row; in an N-row data set this means N separate setup calls. If setup is an expensive operation — opening a real connection, building a large structure — this repetition can look costly, and sharing setup by running it once becomes tempting. This lesson shows the price of that temptation: the speed gained from shared setup is paid for by losing independence between rows. Wanting both properties at once — a setup that is both shared and independent — is a contradiction; which one takes priority is a decision made by whoever writes the body, and this lesson only measures the outcome of that decision.

There is no single correct answer to this choice, only conscious and unconscious forms of it. If a body shares setup deliberately — for example, when rows read a common, unchanging resource that none of them writes to — sharing is not a bug. The bug is sharing happening, without anyone realizing it, in a setup that involves writing. This lesson’s sharedRun method models the second case: when setup was moved outside the loop, which row would see which state was never considered, the only goal was reducing repetition.

Summary

• The data set was written as a declaration, the body was built once and called as many times as there was data — three times; each call recorded its own row’s result.
• With independent setup, all three rows passed; the failure count and which row it came from were logged separately.
• When setup was moved outside the loop and a single Store was shared between rows, two rows failed, but which row passed became dependent on data order.
• When the same three rows ran in reverse order, the passing row changed; the data content stayed the same, only the processing order changed.
• Shared state making the result order-dependent is a concrete source, materialized in this lesson, of the flaky test concept established in M21/K02.

Next Step

In this lesson, the thing that was shared was an object a test body built itself — a bug belonging to the test’s own arrangement, outside the code under test. The code under test (Store) itself never changed; only the test side’s setup changed. The next lesson moves shared state outside the code under test: a dependency mocked in a unit test is put back with its real implementation in an integration setup, and the same test body now runs against a larger class path. Where the defect moves — how a divergence that stays silent in the unit setup turns into a failure in the integration setup — is the next lesson’s question.