Coverage Measurement

The previous lesson made documentation claims testable and measured that four, then six, of the seven claims that went stale out of twelve turned audible. The question there was “does the claim get heard when it falsifies.” Now it is time to look from the other direction: when tests run, where in the code did execution go?

The number that answers this question is called coverage. Coverage does not say whether a test suite is good; it counts which parts of the code ran during a run. This lesson’s job is to show exactly what that number measures and draw its limit — because drawing that limit wrong is the most expensive misreading in this entire topic.

What a Coverage Counter Counts

The coverage counter in the measurement is modeled within the lesson; no tool name appears. The mechanism the model rests on is Python’s trace hook: the interpreter gets handed a function, and it calls that function at certain events during execution. One of these events is the line event, and it fires every time execution moves to a new source line.

The counter is nothing more than this. When the hook gets called, the number of the running line gets recorded; once the run ends, the set of recorded numbers gives the covered lines. The only information the hook sees is “which line are we at” — which test is running, what the assertion expects, whether the result is correct: none of it reaches the hook.

This fact decides the measurement’s entire outcome in advance, and it is not a gap. The trace hook cannot see the test’s expectation, because the expectation is a comparison inside the test, and the hook watches execution, not meaning. A coverage counter extracts the geography of execution.

The function’s eight lines correspond to the eight statements in its body, and their numbers get read from the code itself: the definition line and the docstring get left out, leaving eight executable lines.

Two Metrics: Line and Branch

Line coverage asks whether a line ran at least once. Branch coverage asks whether a condition’s both true and false outcomes got taken. The difference between the two shows up when a conditional’s body never runs: line coverage counts the condition line as covered, branch coverage records that only one branch got taken.

These metrics and the hierarchy between them were built as theory in the Unit Testing and Test-Driven Development course of the Software Quality and Testing curriculum; not repeated here, only referenced. Both get measured together in this lesson, because the claim’s strength depends on it: if only line coverage were full, one could object “you would have seen it with a stronger metric.” The measurement closes off that objection — both metrics are full, and the defect stays invisible anyway.

Branch coverage gets extracted from execution in the model too: for every condition, whether the line inside that condition’s body ran on that call gets checked. If it ran, the true branch was taken; if not, the false branch was. Three conditions, six branches.

Beyond these metrics, a third exists, not used here: path coverage, that is, how many of the conditions’ mutually independent combinations got tried. This function’s three conditions have four reachable combinations, and the measurement’s second table lists exactly those four. This lesson does not write path coverage as a separate number, because it does not weaken the argument, it strengthens it: three of the four are in the test set, and the defective one is among them.

The Oracle Is Written Separately

The shared setup’s discount function makes three decisions: membership ten points, coupon fifteen points, and a cap. The defect is in the third decision — the cap is written as 20, while the contract says 25. The oracle, the correct behavior, is written as a separate function and applies the cap of 25.

The test set’s expected values, though, are written not from the oracle, but from the function’s observed behavior. This is the same situation built in the first lesson, and it is not a setup detail, it is a real writing habit: when writing a test for an existing function, expected values often get decided by running the function and looking at its output. A test written this way verifies that behavior has not changed; not that it is correct.

The measurement’s assumptions:

The Measurement’s Assumptions

• TE41 — The discount function and the oracle are taken from the shared setup; neither’s behavior is changed. The lesson only adds the coverage counter and a fourth test.
• TE42 — The coverage counter is modeled with a trace hook. The hook counts only the line event and only the discount function’s frame; it accesses no other information.
• TE43 — The eight lines counted are the executable statements in the function’s body, and their numbers get read from the code itself; the definition line and the docstring are excluded.
• TE44 — Branch coverage is two outcomes for each of the three conditions, six branches total. That a branch was taken gets inferred from whether the line inside that condition’s body ran.
• TE45 — The test set is the shared setup’s three tests, and their expected values are written from the function’s observed behavior.
• TE46 — The oracle comparison happens outside the measurement, in a separate table; the test set never calls the oracle.
• TE47 — The input space is held with amount fixed at 100, and the four combinations of two boolean flags get tried. Changing the amount does not change the defect, only its size.
• TE48 — A call being “in the set” means it equals one of the test set’s three inputs.
• TE49 — The fourth test uses the same input but takes its expected value from the oracle; there is no other difference.
• TE50 — Passing test count is the count of tests where the expected value equals the actual value; there is no partial pass.
• TE51 — Ratios are written to four decimal places; the smallest measurable line difference in this set is 1/8, that is, 0.1250.
• TE52 — Duration is never measured, no file is left in the repository. What gets counted is covered lines, taken branches, passing tests, and calls deviating from the oracle.

The Measurement

"""Coverage counter: which line ran, which branch was taken, which call
deviates from the oracle.

The counter is modeled within the lesson with a trace hook; what gets
measured is where the code ran.
"""
import sys


def discount(amount, member, coupon):
    """Three-decision function; the defect is in the third decision."""
    rate = 0
    if member:
        rate += 10
    if coupon:
        rate += 15
    if rate > 20:
        rate = 20
    return amount - amount * rate // 100


def correct_discount(amount, member, coupon):
    """Oracle: the correct behavior applies a cap of 25."""
    rate = (10 if member else 0) + (15 if coupon else 0)
    if rate > 25:
        rate = 25
    return amount - amount * rate // 100


NAMES = ("rate=0", "if member", "rate+=10", "if coupon", "rate+=15",
         "if rate>20", "rate=20", "return")
BODY = discount.__code__.co_firstlineno + 1   # definition and docstring lines excluded
NUMBERS = sorted({s for _, _, s in discount.__code__.co_lines()
                   if s is not None and s > BODY})
LINES = dict(zip(NUMBERS, NAMES))
BRANCHES = {"if member": "rate+=10", "if coupon": "rate+=15", "if rate>20": "rate=20"}

TEST_SET = (((100, True, False), 90),
            ((100, False, True), 85),
            ((100, True, True), 80))
ALL = [(100, m, c) for m in (False, True) for c in (False, True)]


def trace_call(args):
    """Traces a single call and returns the names of the lines that ran."""
    seen = set()

    def hook(frame, event, arg):
        if frame.f_code is discount.__code__:
            if event == "line":
                seen.add(LINES[frame.f_lineno])
            return hook
        return None

    sys.settrace(hook)
    try:
        value = discount(*args)
    finally:
        sys.settrace(None)
    return value, seen


def coverage(calls):
    """Line and branch coverage: what gets counted is whether the line ran."""
    lines, branches = set(), set()
    for args in calls:
        _, seen = trace_call(args)
        lines |= seen
        for condition, body in BRANCHES.items():
            branches.add((condition, body in seen))
    return lines, branches


def passing(cases):
    return sum(1 for args, expected in cases if trace_call(args)[0] == expected)


calls = [a for a, _ in TEST_SET]
lines, branches = coverage(calls)
deviating = [a for a in ALL if discount(*a) != correct_discount(*a)]

print(f"{'metric':<14s} {'covered':>9s} {'total':>7s} {'ratio':>7s}")
print(f"{'line':<14s} {len(lines):9d} {len(NAMES):7d} {len(lines) / len(NAMES):7.4f}")
print(f"{'branch':<14s} {len(branches):9d} {2 * len(BRANCHES):7d} "
      f"{len(branches) / (2 * len(BRANCHES)):7.4f}")
print(f"passing tests {passing(TEST_SET)}/{len(TEST_SET)}")

print()
print(f"{'input':<26s} {'actual':>7s} {'oracle':>7s} {'deviation':>9s} {'in set':>7s}")
for a in ALL:
    g, d = discount(*a), correct_discount(*a)
    print(f"{'amount=%d member=%d coupon=%d' % (a[0], a[1], a[2]):<26s} {g:7d} {d:7d} "
          f"{('yes' if g != d else 'no'):>9s} "
          f"{('yes' if a in calls else 'no'):>7s}")
print(f"in {len(deviating)} of four combinations, actual behavior deviates from "
      f"the oracle; that call is in the set: "
      f"{'yes' if deviating[0] in calls else 'no'}")

print()
FOUR = TEST_SET + (((100, True, True), correct_discount(100, True, True)),)
lines4, branches4 = coverage([a for a, _ in FOUR])
print(f"{'test set':<16s} {'tests':>5s} {'passing':>7s} {'line':>7s} {'branch':>7s}")
print(f"{'three tests':<16s} {len(TEST_SET):5d} {passing(TEST_SET):7d} "
      f"{len(lines) / len(NAMES):7.4f} {len(branches) / (2 * len(BRANCHES)):7.4f}")
print(f"{'oracle fourth':<16s} {len(FOUR):5d} {passing(FOUR):7d} "
      f"{len(lines4) / len(NAMES):7.4f} {len(branches4) / (2 * len(BRANCHES)):7.4f}")
RATIOS = {(passing(TEST_SET), len(TEST_SET)), (passing(FOUR), len(FOUR))}
print(f"the fourth test leaves coverage at {len({len(lines), len(lines4)})} distinct "
      f"outcome(s), and raises the passing ratio to {len(RATIOS)} distinct outcomes")

metric           covered   total   ratio
line                   8       8  1.0000
branch                 6       6  1.0000
passing tests 3/3

input                       actual  oracle deviation  in set
amount=100 member=0 coupon=0     100     100        no      no
amount=100 member=0 coupon=1      85      85        no     yes
amount=100 member=1 coupon=0      90      90        no     yes
amount=100 member=1 coupon=1      80      75       yes     yes
in 1 of four combinations, actual behavior deviates from the oracle; that call is in the set: yes

test set         tests passing    line  branch
three tests          3       3  1.0000  1.0000
oracle fourth        4       3  1.0000  1.0000
the fourth test leaves coverage at 1 distinct outcome(s), and raises the passing ratio to 2 distinct outcomes

Full Coverage and Defective Behavior

The upper table places three numbers side by side, and all three look positive. The three-test suite covers eight of eight lines: 1.0000. It takes six of six branches: 1.0000. And 3/3 tests pass. Anyone looking at these three numbers concludes the function is tested.

The middle table gives the same function’s entire input space, and deviation sits in the last row. When membership and coupon come together, the function gives 80, the oracle gives 75 — the rate should be 25, but the cap pulls it down to 20. In 1 of four input combinations, actual behavior deviates from the oracle.

The link between the two tables is the measurement’s real finding. The deviating call — membership and coupon together — is inside the test set. That is, the test set runs exactly that call. The line where the defect lives — the line applying the cap — runs; the coverage counter sees it and records it as covered. Coverage coming out at 1.0000 is, in fact, because this call is in the set.

Why does the test pass, then? Because that test expects 80. Since the expected value was written from the function’s observed behavior, the function produces its defective value, and the comparison holds. The reason it passes is not a coverage gap, it is that the expectation wrote down the defective value.

This course’s third claim is exactly this: coverage sees the line, it does not see the expectation. And the reason was written in this lesson’s first section: the only information reaching the trace hook is “which line are we at.” The value the test expects never reaches that hook.

Same Coverage, Separate Verdict

The lower table settles this with an experiment. A fourth test gets added to the three-test suite; its input is the same — membership and coupon together — but its expected value comes from the oracle. The new test fails, and the suite drops to 3/4.

Coverage does not budge: line 1.0000, branch 1.0000. The same input was already in the set, so the set of lines that ran never changed. Coverage stays at 1 distinct outcome, while the passing ratio rises to 2 distinct outcomes.

These two numbers coming apart is the plainest proof of what coverage is. The change that found the defect never changed coverage at all. Getting closer to the defect by growing coverage is impossible in this example, because coverage is already at its ceiling. What was missing was not a call, it was an expectation — and an expectation sits outside what coverage measures.

It needs saying that the reverse holds too: low coverage is information, and real information. A line that never ran is a line no test ever tried, and anything can be true there. Coverage reliably says which code was never tested; it never says which code is correct. The number’s direction runs one way only.

This asymmetry has one more consequence. A line running and that line being tested are separate things, and the only thing that closes the gap between them is an assertion. By the coverage counter’s reckoning, a line is covered even when a test with no assertion at all runs it — the assertion-free test seen in the first lesson does exactly this. Raising coverage, that is, only requires calling the code, never checking the result; the number cannot tell the two apart.

How the Number Is Read

Three reading rules follow from this.

First: coverage is a lower bound, not a certificate. Low coverage is proof of a gap; high coverage is not proof of sufficiency. This asymmetry also explains why the number erodes once it gets set as a target — the cheapest way to raise it is not adding assertions, it is adding calls.

Second: a coverage percentage cannot be compared on its own. In this measurement, the smallest measurable line difference is 1/8, that is, 0.1250; in an eight-line function, there is no value between 1.0000 and 0.8750. A coverage number placed next to another codebase’s percentage says nothing unless both sides state what they count. This is the previous course’s measurement-discipline rule applied to this measurement.

Third: coverage is not a number that measures the quality of the expectation, and no number that does comes out of a coverage counter. What decides the expectation’s correctness is the oracle — a contract, a specification, or, as in this course, the setup itself. Without an oracle, every expectation written just re-records observed behavior, and the suite offers the code’s current state as its own proof.

Summary

• A coverage counter counts the trace hook’s line event; the only information it sees is which line execution is at. The test’s expectation never reaches this hook.
• A three-test suite covers 8 of 8 lines (1.0000), takes 6 of 6 branches (1.0000), and passes 3/3.
• Against that, in 1 of four input combinations, actual behavior deviates from the oracle and the test set runs that exact call; the reason it passes is not a coverage gap, it is that the expectation wrote down the defective value.
• A fourth test taking its expectation from the oracle drops the suite to 3/4 but does not change coverage: coverage stays at 1 distinct outcome, and the passing ratio rises to 2 distinct outcomes.
• Coverage is one-directional information: it says which code was never tested, not which code is correct. Coverage counts where the code ran, not what the test looked at.

Next Step

Everything measured across this topic was about the code running: which tests got found, in what order they ran, which source they looked at, which claim turned audible, which line executed. Coverage was the last in this sequence, and it counted where the code ran.

What remains is another face of the code: how it is written. A function producing the same behavior can be written in countless forms — different indentation, different line endings, different quoting and spacing choices — and none of this shows up in a coverage counter; the lines that run are the same lines. When these formatting decisions get left to a human, every review produces a fresh argument. The next lesson carries the question here: how many distinct writings of the same source exist, what does that number drop to, and who does the making-singular?