Test Frameworks

The previous lesson made the environment and dependencies singular: the interpreter version got pinned, the lock file wrote down the resolution’s decision, and the same declaration now gives a single version set. No question is left to ask on the setup side. But the setup becoming singular does not mean the code running on top of it gives the same verdict every time.

This lesson’s question: does the code itself give the same result on every run, and who tells us that? What tells us is a test framework. A framework is not a product, it is a class of tool; no product name appears in this lesson. A framework’s jobs get modeled within the lesson, while the framework the standard library provides gets used directly.

The Framework’s Four Jobs

A test framework looks like it does one job — “runs the tests” — but it actually does four separate jobs, and only one of them decides the outcome.

Discovery. Decides which callables count as tests. This decision rests on a naming convention, a directory pattern, or a marker, and does not look at the test’s content. The rule’s source can be the framework’s default, or the project’s setting; either way, the rule sits outside the test.

Run. Calls every discovered test, decides the order, and inserts setup and teardown steps in between when needed. Order itself is a decision too, and it is what the next lesson measures.

Assertion. Compares the expected against the actual inside the test. In Python, the plainest form of this is the assert statement; when the statement comes out false, it raises AssertionError, and the framework counts this exception as a failed test. Using an exception as flow control was built in the Python Fundamentals course; here the same mechanism gets used for reporting.

Report. Writes how many tests ran, how many passed, and the names of the ones that failed; the reader takes the verdict from these lines.

Of these four jobs, only assertion says anything about correctness. Discovery, run, and report are configuration decisions — and what this lesson measures is what configuration decisions do to the outcome.

All four sit together, so they look like one job. Yet only assertion is written in the project’s source files; the discovery rule and run order sit, in most setups, in a separate configuration file or in the run command’s options. Part of the decision is inside the code, part is outside — and when the outside part changes, no trace stays in the code files. For this course’s question, this split is exactly what is being sought: same source, separate configuration, how many distinct outcomes.

The Assertion Needs an Oracle

An assertion compares two things: the value the code produces, against the expected value. Where the expected value comes from is not visible inside the test; whoever wrote it got it from somewhere. This source is called an oracle, built as a concept in the Quality and Test Fundamentals course of the Software Quality and Testing curriculum. Theory is not repeated here; what the oracle adds to the measurement is.

In our shared setup, the oracle is us: we wrote the correct discount behavior, the clean environment, and the correct version set. This lesson’s test cases, though, are written not from the oracle, but from the function’s observed behavior. That is, the expected values were obtained by writing down whatever the function gives today. A test written this way verifies that the function’s behavior has not changed; it does not verify that the behavior is correct. What this difference costs gets measured in this topic’s last lesson; for now, it is enough to note: a test passing does not say the expected value is correct.

How the assertion is written is a choice too. When the assert statement’s sentence comes out false, it raises AssertionError, and the report is left with only “the statement at this line came out false.” The framework’s comparison methods, though, capture both values and write them to the report. This difference shortens diagnosis time but does not change the verdict; the measurement’s upper table shows exactly this.

A Test Without an Assertion Passes Silently

The modeled runner’s rule is a simple one: call the test, count it as passed if AssertionError does not come. This rule has a direct consequence — a test containing no assertion at all always passes. The runner does not count the assertion, it counts the exception; if no assertion is ever written, there is no exception to raise either.

This is not the framework’s defect, it is its limit. The framework does not look at the test’s body; if it did, it would have to know which call counts as an assertion, and that knowledge is application-specific. helper_amount in the measurement is a concrete example: it carries no assertion, so it would count as passed even if discovered. The same holds for a test whose assertion was accidentally commented out, or left inside a condition that never gets evaluated.

The consequence: a passing test may have passed for at least two separate reasons — the assertion held, or there was no assertion at all. The report does not tell the two apart. To tell them apart, the place to look is not the report, it is the test itself; and a mechanism that measures this over the code gets built in this topic’s last lesson.

The Same Three Cases, Two Writing Styles

A unit test has two common writing styles. In the first, every test is a function, and the assertion is written with the assert statement; the runner calls the function and counts the test as failed if it catches an exception. In the second, tests are a class’s methods, and the assertion is written with the framework’s comparison methods; the standard library’s unittest framework uses this style.

The difference between the two is a writing difference. The second style writes both values to the report on a failing assertion; the first only says the statement came out false. This is a report detail and does not change the verdict itself — the measurement shows this.

The class-based style has a second contribution: hooks that run before and after every test can be defined. These hooks take repetition out of the test and are the next lesson’s subject; here, only three plain tests are written, so no hook gets used, and the two styles genuinely do the same job. This was necessary for the comparison to be meaningful: the two styles can only be set side by side once they run the same three cases with the same setup.

What the Discovery Rule Decides

The discovery rule decides which callables in a file count as tests. When the rule changes, the file’s content does not change, the set of running tests does. The file in the measurement carries five callables: three with a t_ prefix, one with a check_ prefix, and one a helper function.

These three rules model a real situation. As a team grows, tests get written by more than one person, and not everyone carries the same naming habit; some use a test_ prefix, some t_, some a check_-like prefix. If the discovery rule looks for only one of these, the others sit in the file, visible to the eye, tracked in version control — and never run.

What matters here: a test that does not run does not count as failed. The framework never saw it, so it has no place in the report. The report says “no failures,” and this sentence is the same whether five tests ran or none did. The discovery rule, for this reason, is not a detail, it is part of the verdict.

The measurement’s assumptions:

The Measurement’s Assumptions

• TE1 — The runner is modeled within the lesson: it calls every test, counts it as failed if it catches AssertionError, and does nothing else. No framework carrying a product name is used.
• TE2 — The class-based suite is run with the standard library’s unittest framework; the report stream is swallowed, and only the numbers get read.
• TE3 — The three test cases are the shared setup’s test set, and the expected values are written from the function’s observed behavior, not from an oracle.
• TE4 — The discount function’s behavior is not changed; the lesson only adds tests and a discovery rule.
• TE5 — The discovery rule is a name prefix and does not look at the test’s body. Three rules get tried: the t_ prefix, the test_ prefix, and the combination of the t_ and check_ prefixes.
• TE6 — helper_amount in the file is not a test, it is a helper; no rule collects it, and if it did, it would count as passed because it carries no assertion.
• TE7 — Distinct outcome count is the number of mutually different result tuples observed across a measurement; duration is never measured in a run.
• TE8 — The report carries only the count of failed tests; a test that never runs has no name in the report, and the reader of the report cannot see this.

The Measurement

"""A framework does four jobs: discovery, run, assertion, and report. The
runner is modeled within the lesson.

Part 1 - the same three cases, two writing styles.
Part 2 - the same file, three discovery rules.
"""
import io
import unittest


def discount(amount, member, coupon):
    """The shared setup's three-decision function; its behavior is not changed."""
    rate = 0
    if member:
        rate += 10
    if coupon:
        rate += 15
    if rate > 20:
        rate = 20
    return amount - amount * rate // 100


CASES = (("member_only", (100, True, False), 90),
         ("coupon_only", (100, False, True), 85),
         ("both", (100, True, True), 80))


def runner(tests):
    """Modeled runner: calls every test, counts passed and failed."""
    passed, failed = 0, []
    for name, func in tests:
        try:
            func()
        except AssertionError:
            failed.append(name)
        else:
            passed += 1
    return passed, failed


def plain_suite():
    def make(args, expected):
        def test():
            assert discount(*args) == expected
        return test
    return [(name, make(args, exp)) for name, args, exp in CASES]


class ClassSuite(unittest.TestCase):
    def test_member_only(self):
        self.assertEqual(discount(100, True, False), 90)

    def test_coupon_only(self):
        self.assertEqual(discount(100, False, True), 85)

    def test_both(self):
        self.assertEqual(discount(100, True, True), 80)


def class_run():
    suite = unittest.TestLoader().loadTestsFromTestCase(ClassSuite)
    result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    return result.testsRun - len(result.failures) - len(result.errors), len(result.failures)


def t_member_only():
    assert discount(100, True, False) == 90


def t_coupon_only():
    assert discount(100, False, True) == 85


def t_both():
    assert discount(100, True, True) == 80


def check_two_hundred():
    assert discount(200, True, True) == 160


def helper_amount(amount):
    return amount


FILE = {"t_member_only": t_member_only, "t_coupon_only": t_coupon_only,
        "t_both": t_both, "check_two_hundred": check_two_hundred,
        "helper_amount": helper_amount}

RULES = {"t_ prefix": ("t_",), "test_ prefix": ("test_",),
         "t_ or check_": ("t_", "check_")}


def discover(prefix):
    return [(name, f) for name, f in FILE.items() if name.startswith(prefix)]


print(f"{'writing style':<22s} {'tests':>5s} {'passed':>6s} {'failed':>6s}")
plain_passed, plain_failed = runner(plain_suite())
print(f"{'plain function suite':<22s} {len(CASES):5d} {plain_passed:6d} {len(plain_failed):6d}")
class_passed, class_failed = class_run()
print(f"{'class-based suite':<22s} {len(CASES):5d} {class_passed:6d} {class_failed:6d}")
print(f"same three cases, two writing styles: "
      f"{len({(plain_passed, len(plain_failed)), (class_passed, class_failed)})} distinct outcomes")

print()
print(f"{'discovery rule':<18s} {'found':>8s} {'passed':>6s} {'failed':>6s} {'report':>16s}")
found, report = [], []
for name, prefix in RULES.items():
    tests = discover(prefix)
    passed, failed = runner(tests)
    found.append(len(tests))
    report.append(len(failed))
    print(f"{name:<18s} {len(tests):8d} {passed:6d} {len(failed):6d} "
          f"{'failed ' + str(len(failed)):>16s}")
print(f"same file, three discovery rules: found tests {len(set(found))} distinct outcomes, "
      f"report {len(set(report))} distinct outcomes")
print(f"tests that never run under the narrowest rule: {max(found) - min(found)}; "
      f"the report does not say this")

writing style          tests passed failed
plain function suite       3      3      0
class-based suite          3      3      0
same three cases, two writing styles: 1 distinct outcomes

discovery rule        found passed failed           report
t_ prefix                 3      3      0         failed 0
test_ prefix              0      0      0         failed 0
t_ or check_              4      4      0         failed 0
same file, three discovery rules: found tests 3 distinct outcomes, report 1 distinct outcomes
tests that never run under the narrowest rule: 4; the report does not say this

Reading the Numbers

The upper table gives what the writing style costs: the same three cases, 1 distinct outcome across two separate stylings. Both the plain function suite and the class-based suite pass 3 of 3 tests. Writing style is a preference, and it does not change the verdict. This does not mean choosing between frameworks is unimportant — but what the choice costs is not the outcome, it is the report’s detail.

The lower table gives what configuration costs, and the number changes. Same file, three discovery rules: found tests give 3 distinct outcomes (3, 0, and 4), the report gives 1 distinct outcome. In all three rules, the report says “failed 0.”

The second row is this measurement’s real finding. The rule looking for the test_ prefix finds no test at all in this file, zero tests run, and its report is “failed 0.” This report is indistinguishable from the report where three tests passed. A suite being green is true under two separate conditions: the tests ran and passed, or no test ran at all.

The third row shows the same fact from the other direction. A test named check_two_hundred sits in the file, its assertion is correct and it passes — but the default rule, looking only for the t_ prefix, never collects it. The difference between the narrowest and widest rules is 4 tests, and all of that difference stays outside the report.

The rule that follows: a test suite’s verdict depends on the discovery rule as much as on the tests themselves. The report counts the tests that ran, not the ones that did not; for this reason, the report’s “failed 0” line is not proof on its own. Proof in full requires reading the found test count too — and, as the measurement shows, this number can take three separate values without the code itself ever changing.

Read together, the two tables make the course’s measurement axis visible for the first time here. Two separate questions got asked of the same input — same three cases, same function, same file. The first was “how did you write it,” and it did not change the outcome: 1 distinct outcome. The second was “what are you running,” and it did change the outcome: 3 distinct outcomes. What threatens reproducibility is not the first, it is the second, because the second is written not in a file, but in the setting that configures the run, and it does not get read alongside the code.

This distinction’s practical counterpart is clear too. A suite’s green report only carries information once the count of running tests is known too; a green report read without that number looks the same in a setup where no test ran at all. For this reason, the running test count gets kept alongside the report, and its drop counts as an event as attention-worthy as a failing test.

Summary

• A test framework does four jobs: discovery, run, assertion, and report. Of these, only assertion says anything about correctness; the other three are configuration decisions.
• An assertion takes its expected value from an oracle. A test passing does not show that expected value is correct.
• The same three cases pass 3/3 with both the plain function suite and the class-based suite; the two writing styles give 1 distinct outcome, and the difference sits only in the report’s detail.
• The same file finds 3, 0, and 4 tests across three discovery rules — 3 distinct outcomes; the report says “failed 0” in all three and stays at 1 distinct outcome.
• A test that never runs does not count as failed; under the narrowest rule, 4 tests stay outside the report, and a green report does not say this.

Next Step

The discovery rule decided which tests would run; but the running tests are not independent one by one. In the shared setup’s five-test suite, one pollutes shared state, and another assumes an unpolluted state. In a suite like this, which order you call the tests in is a configuration detail — does that detail change the verdict? The next lesson runs the same suite in three separate orders, reads the number of distinct verdicts that come out, and measures two mechanisms that take repetition out of a test: the fixture, which rebuilds shared state fresh for every test, and parametrization, which runs the same body with more than one input.