Reviewing Tests

The previous lesson assumed that in this setup a defect is missed for a single reason: no one looking at that axis. But there can be a second reason for a miss — the text the defect sits in was never read at all. The two reasons give the same result and show up in the same column of the table; yet the response called for is not the same.

This lesson measures that distinction, and it measures it on the test axis. The test axis is not chosen at random: tests sit in the same files as the change’s body, or in files adjacent to them, meaning they pass in front of the eyes of reviewers looking at other axes too. Its question is this: can a defect, sitting inside text that has been read, be missed in front of the eyes of multiple people who read that very text?

The Test Itself Is Also Reviewed

Test levels, the test pyramid, coverage metrics, and test-writing techniques were established in the Software Quality and Testing curriculum and are not repeated here. This lesson’s subject is not how to write a test, but what is found when the test itself is looked at in review.

The distinction sits here. A change’s tests occupy two separate roles. In the first role, a test is evidence: it shows the body is correct, and the reading looking at the implementation axis relies on it. In the second role, a test is a review object: it is itself written code and carries its own defects.

When the second role is skipped, the first collapses. A reading that decides on the body’s correctness by looking at whether the tests pass, without checking what the tests actually verify, has looked at nothing at all. The test axis is exactly this check.

Three Defects a Test Can Carry

Coverage. The new path the change introduces has not been tested. Tests exist, they pass, and their count may even have grown; but no test touches the changed behavior itself. This defect’s trace visible in review is the absence of an assertion corresponding to a branch in the changed body.

Meaningfulness. The test runs the new path but verifies nothing. A test that checks only that the call does not throw passes no matter what the body returns. The same class also includes tests whose expected value is computed with the same expression the body uses: an assertion producing the expected value from the body’s own expression is confirmed even if the body is wrong.

Fragility. The test is bound not to behavior but to an implementation detail: call order, an internal structure’s field name, a log line’s exact text. Such a test breaks without the behavior changing, and when it breaks, what needs fixing is not the body, it is the test.

The three are not equally visible in review. A coverage defect is a countable gap, surfacing once the body’s branches and the tests’ state are placed side by side. A meaningfulness defect requires reading the test’s text and escapes a reading that looks at test count, because a test verifying nothing is still a test and still counts. Fragility is the hardest of the three: the test passes today, and its cost is paid in a later, unrelated change that it stops for no real reason.

# taught review-comment example, not executed

coverage       Change introduces two new branches; tests call only the
               first branch. A case needs to be added for the second
               branch.

meaningfulness Test's only assertion is that the call does not throw. This
               test passes no matter what value the body returns; the
               returned value needs to be verified.

fragility      Test compares the exact text of a log line. The test will
               break when the log format changes even if behavior stays
               the same; the comparison can be tied to the outcome
               instead.

All three notes point at a spot in the test and say what needs to be added for it to close. This block is an example and is not run; the lesson’s numbers come only from the measurement block.

The Test Comes From the Same Hand as the Change

There is a structural reason the test axis requires a separate reading, and it ties back to the previous lesson’s question: the reading looking at the implementation axis had to derive what the body should do from somewhere, and one available source was the tests.

The problem is that tests are part of the same change. The body and the test are written at the same time, with the same understanding. If the body rests on a misunderstanding, the test records the same misunderstanding and confirms the body. In that case, the test passing is not evidence, it is an echo.

The consequence for review is direct: the test’s correctness has to be checked independently of the body’s correctness. The test axis is the only reading that does this, and what it looks at is not whether the test passes but what the test claims. An assertion repeating the same expression as the body, never calling one of the body’s branches, or comparing no result at all — these are seen by reading the test, not by running it.

The measurement does not model this dependency — defects in the test class are independent of the ones in the implementation class. The model’s simplicity favors the measurement here: had the dependency been added, the cost of closing the test axis would have come out larger than what shows in the table.

Missing Inside Text That Was Read

In the setup, finding a defect depends on two conditions: the defect’s class has to be on the reviewer’s axis, and the chunk the defect sits in has to be among the chunks the reviewer read. If either condition fails, the defect is missed.

At this topic’s scale, the second condition always holds: the change is 12 chunks and attention is also 12 chunks. So in this lesson no defect is missed because its text was unread. The measurement counts this instead of assuming it, and splits the missed into two: missed inside a read chunk, and missed inside an unread chunk.

The second column coming out zero shows that the entirety of the first column is axis miss. This is a direct answer to the lesson’s question: the missed defect does not sit somewhere no one looked, it sits somewhere everyone read.

The Measurement’s Assumptions

• RA23 — The shared setup is unchanged: 600 lines, 12 chunks, 24 defects, attention 12 chunks.
• RA24 — A chunk is 50 lines and the unit of reading is the chunk; a reviewer who reads a chunk is considered to have seen all of it.
• RA25 — The “other class in the same chunk” column gives the class names of defects found in that chunk that belong to a non-test axis.
• RA26 — The “other axis flagging in chunk” count is the number of non-test axes that have a defect of their own class in that chunk. A reviewer on that axis has, by the setup’s certainty, made a note in that chunk — the chunk has been read.
• RA27 — A defect missed-and-read is one whose chunk was read but whose class is not on the reviewer’s axis. A defect missed-and-unread is one whose chunk was never read.
• RA28 — Every row in the second part is a single single-axis reading; the missed columns are relative to that one reading.
• RA29 — Whether the test passes or fails is not in the measurement; what is measured is the defect the test carries as seen in review, not the result it gives when run.
• RA30 — Resolution at the chunk level is 1/12 = 0.083, at the defect level 1/24 = 0.042.

Measurement

"""Reviewing tests: defects missed inside chunks that were read.

Part 1 - the chunks test defects sit in and the other classes in that chunk.
Part 2 - for each single-axis reading, the split between missed-and-read and missed-and-unread.
"""
SEED = 20260815
AXES = ("interface", "implementation", "test", "documentation", "style")
UNWRITTEN = "unwritten requirement"
CLASSES = AXES + (UNWRITTEN,)
ATTENTION = 12
CHUNK = 50


def rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def change(lines, defect_count=24, seed=SEED):
    draw, defects = rng(seed), []
    chunk_count = max(1, lines // CHUNK)
    for i in range(defect_count):
        defects.append({"no": i + 1, "class": CLASSES[draw(6)],
                         "chunk": draw(chunk_count)})
    return {"lines": lines, "chunks": chunk_count, "defects": defects}


def review(d, axes, attention=ATTENTION):
    read = set(range(min(attention, d["chunks"])))
    return {k["no"] for k in d["defects"]
            if k["class"] in axes and k["chunk"] in read}


FULL = set(AXES)
d = change(600)
test_defects = [k for k in d["defects"] if k["class"] == "test"]

print(f"{'defect':>6s} {'chunk':>5s} {'other class in same chunk':<28s} "
      f"{'other axis flagging in chunk':>30s}")
for k in test_defects:
    neighbors = sorted({o["class"] for o in d["defects"]
                        if o["chunk"] == k["chunk"] and o["class"] in AXES
                        and o["class"] != "test"})
    print(f"{k['no']:6d} {k['chunk']:5d} {', '.join(neighbors):<28s} "
          f"{len(neighbors):30d}")
occupied = {k["chunk"] for k in d["defects"]}
mixed = sum(1 for o in range(d["chunks"])
            if len({k["class"] for k in d["defects"] if k["chunk"] == o}) > 1)
print(f"{d['chunks']} chunks, {len(occupied)} have at least one defect; "
      f"{mixed} chunks have defects from more than one class")

print()
print(f"{'axis':<15s} {'chunks read':>11s} {'found':>7s} "
      f"{'missed, read':>13s} {'missed, unread':>15s}")
for a in AXES:
    b = review(d, {a})
    read = set(range(min(ATTENTION, d["chunks"])))
    missed = [k for k in d["defects"] if k["no"] not in b]
    m_read = sum(1 for k in missed if k["chunk"] in read)
    m_unread = sum(1 for k in missed if k["chunk"] not in read)
    print(f"{a:<15s} {len(read):11d} {len(b):7d} {m_read:13d} {m_unread:15d}")

print()
FOUR = FULL - {"test"}
for name, axes in (("five axes", FULL), ("four axes, no test", FOUR)):
    b = review(d, axes)
    missed = [k for k in d["defects"] if k["no"] not in b]
    tm = sum(1 for k in missed if k["class"] == "test")
    print(f"{name:<20s} found {len(b):2d}  missed {len(missed):2d}  "
          f"test missed {tm}  unwritten missed "
          f"{sum(1 for k in missed if k['class'] == UNWRITTEN)}")

defect chunk other class in same chunk      other axis flagging in chunk
     3     9 implementation                                            1
     7     1 interface, style                                          2
    19     7 interface                                                 1
12 chunks, 11 have at least one defect; 7 chunks have defects from more than one class

axis            chunks read   found  missed, read  missed, unread
interface                12       3            21               0
implementation           12       6            18               0
test                     12       3            21               0
documentation            12       3            21               0
style                    12       7            17               0

five axes            found 22  missed  2  test missed 0  unwritten missed 2
four axes, no test   found 19  missed  5  test missed 3  unwritten missed 2

Same Chunk, Separate Axis

The top table shows the three test defects one by one, and none of the three stands alone.

Defect number 3 is in the ninth chunk, and that same chunk has an implementation defect. Defect number 7 is in the first chunk, where both an interface and a style defect sit. Defect number 19 is in the seventh chunk, next to an interface defect.

The column on the right gives what this means: 1, 2, 1. These are the number of axes that found a defect of their own class in that chunk. That is, in the first chunk, two separate reviewers made a note; both read those same fifty lines; and both missed the test defect inside that same fifty lines. The same holds once each for the ninth and seventh chunks.

The three defects sit in three separate chunks; at the chunk level this is 3/12, that is, 0.250. The chunks holding test defects correspond to a quarter of the change, and none of these chunks went unread. Given the smallest measurable difference at the chunk level is 1/12 = 0.083, this share sits comfortably inside the measurement band.

The bottom row gives the setup’s density: 11 of 12 chunks have at least one defect, and 7 chunks have defects from more than one class. The change is not crammed into a narrow spot; the classes sit mixed together. In nearly every chunk a reviewer reads, the chance is high that something they did not find sits next to something they did.

The Single Reason for a Miss

The middle table turns this lesson’s distinction into a number. In all five rows, chunks read is 12 and missed, unread is 0.

The zero column says this: at this scale, no defect is missed because its text was unread. Every missed defect — 21 in interface, 18 in implementation, 21 in test, 21 in documentation, 17 in style — sits inside a chunk that was read. Each has a single cause: that class is not on the reviewer’s axis.

This names review’s most deceptive trait. A reviewer can say they read the entirety of the change, and they are telling the truth; the reading really was done in full. But “I read all of it” does not mean “I looked at all of it.” Reading belongs to coverage, looking belongs to an axis, and the table shows the two are separate.

This has a consequence for review’s record. When a review closes by saying “I found nothing,” the information left behind is incomplete: which axes were looked at is not written down. Someone looking at the same change later cannot know which classes a closed review covered, and by the table, this is as decisive as the list of defects found.

What the record needs to carry is therefore not only the list of findings but the list of axes looked at. A note such as “read on interface and implementation, not on test and documentation” does not change the found count; it tells in advance which classes the missed defects will be in. This note is about the change and records review’s scope, not the reviewer’s success.

The bottom two rows give the size of the loss. A reading covering five axes finds 22; once the test axis is closed, 19. Missed climbs from 2 to 5, and the difference is exactly 3 test defects. The test axis’s share, in keeping with the rule measured in the previous lesson, equals the size of its class.

The practical weight of these three defects being missed is, in one sense, heavier than the others. A missed test defect does not stay only itself; it also breaks the test’s evidence role. A test suite with a coverage gap gives a false assurance to later readings working on the implementation axis — the tests are passing, but what they are passing is not the changed behavior.

Summary

• A test carries two roles in a change: evidence for the body’s correctness, and a review object in its own right. When the second role is skipped, the first is left unsupported.
• The defects a test can carry are three kinds: coverage (the new path is untested), meaningfulness (the assertion verifies nothing), fragility (the test is bound to an implementation detail).
• The three test defects sit in chunks 9, 1, and 7, and other axes gave findings in those chunks — 1, 2, 1 respectively; the reviewers who read those chunks passed over the test defect.
• All five axes give missed-in-unread-chunk 0: at this scale, every missed defect’s single cause is an axis not looked at. “I read all of it” does not mean “I looked at all of it.”
• Once the test axis is closed, found drops from 22 to 19 and missed climbs from 2 to 5; the three missed defects also weaken the test’s evidence role.
• Because a test is written at the same time and with the same understanding as the body, a passing test is not independent evidence; the test axis looks at what the test claims, not whether it passes.

Next Step

Three axes have been measured so far, and all three had code as their object: signature, body, test. The fourth axis’s object sits outside the code — the change’s reflection in documentation. When this axis is closed too, the loss is expected to equal the size of its class; but the documentation axis has a distinct side. A written document being wrong and a behavior never having been documented at all are not the same thing, and the two fall into separate classes in the setup. The next lesson closes the five axes one at a time to measure the loss, and separates what the documentation axis can find from what it cannot.