Fixtures and Parametrization

The previous lesson measured that the discovery rule decides which tests run: the same file found 3, 0, and 4 tests across three rules, and the report stayed the same across all three. There, the tests were independent of each other; each one called a function, compared the returned value, and left no trace behind.

Real suites are not like this. Most tests need a setup: an object gets built, a directory gets opened, a counter gets reset. When this setup gets shared, tests become each other’s neighbors — and neighborhood gives rise to an order. This lesson’s question: when run order changes, how many distinct verdicts does the same suite give, and what makes that number singular?

Shared State Makes Order Matter

The shared setup’s five-test suite is the plainest model of this. Five tests operate on a single environment object. Three do not touch the environment and pass under every condition. One pollutes the environment: it increments a counter by one and still passes. One assumes the environment is unpolluted; it passes if the counter is zero, fails otherwise.

None of the five tests is defective on its own. The polluting test does its job, and the dependent test does its job. The defect lies in the two running together, in a particular order. In writing order, the polluting test gets called before the dependent test, and the dependent test fails; in the order that moves the dependent test earlier, and in reverse order, the dependent test gets called first and passes.

This was built as theory under “fast and independent tests” in the Unit Testing and Test-Driven Development course of the Software Quality and Testing curriculum. Theory is not repeated here; what gets measured is how many distinct verdicts the same suite gives.

Who Decides the Order

The order not being written into any test is this measurement’s starting point. The order tests get called in is a decision the runner makes, and this decision comes from at least four places.

The first is discovery order: whatever order the runner collected files in is the order tests run in, and the order of filenames changes when a file gets renamed. The second is selection: running only failing tests, or only a certain tag, takes the rest out and changes the remaining tests’ neighborhood. The third is distribution: when tests get split across more than one worker, each worker runs its own subset in its own order, and how the split happens depends on worker count. The fourth is deliberate shuffling: some setups tie order to a value decided fresh on every run, because they want to catch order dependency early.

What the four share: none of them is written in the project’s source files. Order falls into the same class as the previous lesson’s discovery rule — the part of the verdict that sits outside the code. This lesson’s measurement changes exactly that part and reads the result: three orders, the same five tests.

Fixture: Taking Setup Out of the Test

A fixture is a unit, called by the framework, that takes the setup a test needs out of the test’s body. The test itself does not build the environment; the fixture builds it and hands it to the test ready-made. Besides reducing repetition, this does a second and more important thing: it turns when the setup happens into an explicit decision.

This decision is called the fixture’s scope, and it takes three values in the measurement.

No fixture. The environment gets built once, and the five tests share it. The trace the polluting test leaves is visible to every test that runs after it.

Each test. The fixture rebuilds the environment before every test. The polluting test’s trace never reaches the next test.

Session. The fixture gets built once at the start of the run session, and every test, across every run, shares the same object. This scope gets chosen for expensive setups — so that a resource costly to build does not get rebuilt for every test.

All three scopes have valid uses; the measurement gives not which one is “correct,” but what each one does to distinct verdict count.

A fixture’s other half is teardown: taking back what got built. An in-memory object needs no teardown; the new object replaces the old one, and the old one drops away on its own. For a file, a directory, or a connection, though, teardown gets written explicitly and has to run even if the test fails; this is where the context manager protocol built in the Python Fundamentals course applies directly. A fixture with missing teardown thinks it removed the sharing while it leaves a residue, and the measurement becomes order-dependent again.

Parametrization: Same Body, Separate Input

Testing the same assertion with more than one input has two paths. In the first, the inputs get put into a loop inside a single test; in the second, every input becomes a separate test, and the body gets shared. The second is called parametrization.

Parametrization shares something with the fixture: both take repetition out of the test’s body. Where they part is what they take out. A fixture takes out setup and manages sharing between tests; parametrization takes out input and multiplies a single body across more than one case. For this reason, the two do not substitute for each other, they get used together: every parametrized case runs with its own fixture.

The difference is invisible while tests pass. Both test the same assertion with the same inputs, and both give a green report. The difference shows up when an input fails: in the looped writing, the first deviation fails the test and the loop stops there, so the remaining inputs never get tried at all. The report carries a single line, and that line holds the test’s name — not which input failed. In the parametrized writing, each input runs under its own name; the failing input shows up by name in the report, and the others keep running.

The measurement’s assumptions:

The Measurement’s Assumptions

• TE9 — The five-test suite and three orders are taken from the shared setup; the TESTS list and the tests’ behavior are not changed. The lesson only adds fixture scope.
• TE10 — The oracle is the setup itself: we know all five tests pass in a clean environment because we wrote it that way. The correct verdict in every order is 5/5.
• TE11 — Three orders get tried: writing order, the order that moves the dependent test earlier, and reverse order. The order list is made of the tests’ indices and is not written into any test.
• TE12 — In the no-fixture scope, the environment gets built once at the start of each run; it does not get rebuilt within the run.
• TE13 — In the each-test scope, the environment gets rebuilt right before every test in the order is called.
• TE14 — In the session scope, the environment gets built once for all three runs, and carries over between runs too.
• TE15 — Distinct verdict count is the number of mutually different passed-test counts the three orders give.
• TE16 — The assertion tested in the parametrization measurement is this: every order should give 5/5. This assertion comes from the oracle, and it comes out false in the no-fixture suite under writing order.
• TE17 — The looped writing returns at the first deviation; the parametrized writing runs all three orders. Duration is never measured; what gets counted is orders run and report line count.
• TE18 — Report line count is the number of records the framework writes to its result list; an input that never runs never enters the report.

The Measurement

"""Fixtures and parametrization: who sets up shared state, and when.

Part 1 - three orders, three fixture regimes.
Part 2 - the same three orders in a single test and in a parametrized test.
"""


class Environment:
    """State the tests share."""

    def __init__(self):
        self.counter = 0


def t_clean_a(env):
    return True


def t_polluter(env):
    env.counter += 1
    return True


def t_dependent(env):
    return env.counter == 0


def t_clean_b(env):
    return True


def t_clean_c(env):
    return True


TESTS = [("clean_a", t_clean_a), ("polluter", t_polluter),
         ("dependent", t_dependent), ("clean_b", t_clean_b), ("clean_c", t_clean_c)]

ORDERS = {"writing order": [0, 1, 2, 3, 4],
          "dependent first": [0, 2, 1, 3, 4],
          "reverse": [4, 3, 2, 1, 0]}


def run(order, regime="none", session=None):
    """Fixture scope: none, rebuilt each test, or built once for the session."""
    env = session if regime == "session" else Environment()
    passed = 0
    for i in order:
        if regime == "each test":
            env = Environment()
        _, func = TESTS[i]
        if func(env):
            passed += 1
    return passed, len(order)


print(f"{'fixture scope':<18s} " + " ".join(f"{label:>13s}" for label in ORDERS)
      + "  distinct")
for regime in ("none", "each test", "session"):
    session, passes = Environment(), []
    for order in ORDERS.values():
        passes.append(run(order, regime, session)[0])
    print(f"{regime:<18s} " + " ".join(f"{p:>11d}/5" for p in passes)
          + f" {len(set(passes)):9d}")

print()


def single_test():
    """Tries the three orders in one body, one loop; stops at the first deviation."""
    ran = 0
    for label, order in ORDERS.items():
        ran += 1
        passed, total = run(order)
        if passed != total:
            return [("three orders", "failed")], ran
    return [("three orders", "passed")], ran


def parametrized_test():
    """Each order is a separate test; all three run."""
    report, ran = [], 0
    for label, order in ORDERS.items():
        ran += 1
        passed, total = run(order)
        report.append((label, "passed" if passed == total else "failed"))
    return report, ran


print(f"{'style':<16s} {'report lines':>12s} {'orders run':>11s} "
      f"{'named as failing':>26s}")
lines, ran_counts = [], []
for label, func in (("single test", single_test), ("parametrized", parametrized_test)):
    report, ran = func()
    failing = [r[0] for r in report if r[1] == "failed"]
    lines.append(len(report))
    ran_counts.append(ran)
    print(f"{label:<16s} {len(report):12d} {ran:11d} "
          f"{(failing[0] if failing else '-'):>26s}")
print(f"same three orders: report lines {len(set(lines))}, orders run {len(set(ran_counts))} "
      f"distinct outcomes; failing verdict 1 in both")

fixture scope      writing order dependent first       reverse  distinct
none                         4/5           5/5           5/5         2
each test                    5/5           5/5           5/5         1
session                      4/5           4/5           4/5         1

style            report lines  orders run           named as failing
single test                 1           1               three orders
parametrized                3           3              writing order
same three orders: report lines 2, orders run 2 distinct outcomes; failing verdict 1 in both

Two Distinct Verdicts and Two Paths to Singular

The first row pays out the course’s fourth reading. The no-fixture suite passes 4/5 in writing order, 5/5 in the order moving the dependent test earlier, 5/5 in reverse order — 2 distinct verdicts. The five tests’ code never changed; the only thing that changed was call order. Same input, same code, two distinct outcomes.

This number’s practical counterpart: a suite like this is green in some runs, red in others, and when red, the failing test is not the test where the defect lives. The report points to dependent, while the test polluting the environment is polluter. Order dependency carries diagnosis to the wrong place.

The second row gives what the fixture pays off. When the environment gets rebuilt before every test, all three orders give 5/5, and the count drops to 1 distinct verdict. The polluting test still increments the counter, but the object it increments never reaches the next test. The fixture did not fix a defect — the defect was never in the tests — it removed the sharing.

The third row is the measurement’s real warning. The session-level fixture also drops the count to 1 distinct verdict, but this time at 4/5. Pollution accumulates on the first run and carries over into later runs too; the dependent test no longer passes in any order. The result became stable, and the value it stabilized at is wrong.

The sentence that follows is this lesson’s measurement axis at its sharpest: distinct outcome count dropping to 1 does not mean the outcome it drops to is correct. Reproducibility does not substitute for correctness; it is only the precondition for correctness being arguable at all. A suite that gives the same wrong answer on every run is reproducible, and still defective.

What separates the second and third rows, for this reason, is not the scope’s name, it is which verdict the setup gives. Both produce a single number; one cuts off pollution before it is ever born, the other makes pollution permanent. What shows a suite has escaped order dependency is not distinct verdict count being 1, it is that single verdict matching the oracle — and the oracle, in the shared setup, says all five of the five tests pass in a clean environment.

A warning holds in the reverse direction too. 2 distinct verdicts could be seen here because three orders got tried; had a single order run, the suite would have given either 4/5 always or 5/5 always, and order dependency would never have shown up. The condition for order dependency to be measurable at all is running order across more than one value.

Report Lines and Running Inputs

The lower table gives what parametrization pays off. When the three orders get tried inside a single test, the runner writes 1 report line, and that line holds the test’s name — “three orders.” Which order failed is not in the report; and since it returns at the first deviation, 1 order runs and the remaining two never get tried at all. This second point outweighs the first: the report is not only missing information, the measurement itself gets done incompletely.

In the parametrized writing, the same three orders produce 3 report lines and 3 orders run. The failing line shows up by name: “writing order.” Since the other two orders keep running, the reader of the report can see whether the defect sits in all three orders or in just one — and this distinction carries diagnosis straight to the polluter test.

The two writings’ failing-verdict count is the same: both turn the suite red. What differs is the information the report carries and the count of inputs that actually run. Parametrization does not change the test’s verdict, it raises the verdict’s resolution.

This distinction has a cost too, and it should not be hidden. In a parametrized suite, test count grows with input count; ten inputs means ten tests, and those ten tests take up ten lines in the report. The looped writing gets by with one line. Parametrization, that is, buys diagnostic information with report volume. What the measurement says is not which one to pick, it is what the choice changes: report lines 2 distinct outcomes, orders run 2 distinct outcomes, failing verdict 1.

Summary

• Shared state ties a test’s correctness to its neighbor; while none of the five tests is defective on its own, the suite gives 2 distinct verdicts by order: 4/5 in writing order, 5/5 in the other two.
• A fixture takes setup out of the test and makes the real decision visible: when the setup happens. This decision is called the fixture’s scope.
• When the environment rebuilds fresh for every test, all three orders give 5/5, and the count drops to 1 distinct verdict; the fixture does not fix a defect, it removes the sharing.
• A session-level fixture also drops the count to 1, but at 4/5; distinct outcome count becoming singular does not say the outcome it becomes singular at is correct.
• Parametrization does not change the verdict, it raises its resolution: the same three orders give 1 report line and 1 order run in a single test, and 3 and 3 in the parametrized writing.

Next Step

The fixture made the count singular by rebuilding shared state fresh for every test — but it could do this because what it built was in its own hands: an in-memory object. Had what the fixture built been a package registry over a network, a clock, or a filesystem, “rebuilding” would not have been an option. The next lesson looks at this outward-reaching leg of testing: how many distinct outcomes does the same test give across three separate modes of an external resource, and what does the mock object that replaces the external resource drop that number to — and on what basis do we say the outcome it drops to is correct?