Documentation Review

The previous three lessons measured three axes, and all three had code as their object: signature, body, test. The fourth axis’s object sits outside the code. When a change alters a behavior, the text describing that behavior has to change too; if it does not, what is left is a text that contradicts what the code says.

This lesson measures that axis, and it widens the measurement by one step: the five axes are closed one at a time, and each closure’s loss is placed side by side. Then the documentation axis’s own particular limit is measured — the difference between a written document being wrong and a behavior never having been written down at all.

The Change’s Reflection in Documentation

The kinds of text the documentation axis looks at are separate, and not all of them are found in the same change. Comment lines next to the body, the public interface’s reference text, the repository’s introduction file and setup instructions, sample code snippets — each rests on a behavior, and becomes invalid once that behavior changes.

The distinction between document types, reader definitions, and writing discipline are the subject of the Technical Writing and Documentation course and are not repeated here. The only thing this lesson measures is whether documentation is looked at in review, and what is missed when it is not.

The documentation axis’s distinguishing trait is that its finding is discovered through a contradiction. The other axes look at a single text: interface looks at the signature, implementation at the body, test at the assertion. The documentation axis compares two texts — what the code does and what the text says it does. This is why a documentation defect cannot be found by looking only at the documentation file; reading the documentation correctly requires having read the code too.

This need to compare makes the documentation axis look expensive, though it can be made cheaper. What the code does has already been read on the interface and implementation axes; the documentation axis lays the second text on top of that same reading. Carrying two axes together in the same reading is the practical counterpart of the additivity measured in the third lesson — axes do not eat into one another’s share and can be run together.

Three Findings of the Documentation Axis

Stale text. The behavior has changed, the sentence describing it has not. The text is grammatically flawless and was once true; today it is wrong. Of the three, this is the one most often missed, because the text itself is invisible inside the change — since it did not change, it does not appear in the change.

Missing text. The change introduces a new public name or a new configuration option, but no text describing it was added. This finding’s trace is inside the change: the added name is visible, the description that should have been added alongside it is not.

Contradicting text. The same behavior is described in two separate places, and one was updated. Whichever the reader looks at, they believe it; only someone reading both at once sees that they contradict.

# taught review-comment example, not executed

stale          Function's description line says "returns empty list"; the
               body now returns an empty value. Description needs to be
               updated to match the new behavior.

missing        New configuration option was added, no counterpart in the
               setup instructions. Option needs to be added to the
               instructions along with its default value.

contradicting  Example in the introduction file uses the old call form,
               the reference text describes the new form. One of the two
               texts needs to be updated to match the other.

This block is an example and is not run; the lesson’s numbers come only from the measurement block.

Why the Documentation Axis Closes

Of the five axes, documentation is the one that closes most often. The reasons are outside the setup, but they change how the table should be read.

Documentation is left for last. The change’s body and tests are ready, documentation “will be written after merge.” In this case review has not closed the axis, it has postponed it; for the measurement, the two are the same, because no documentation defect is found in that round.

A documentation finding is not treated as a blocker. An implementation finding stops a merge; a documentation finding often does not — the finding is written and the change merges anyway. The table does not carry this distinction: a defect found is found, whether it closes is the next topics’ variable.

Documentation sits outside the change’s scope. The comment line next to the body is inside the change; the introduction file, the setup instructions, or the reference text may sit somewhere else, even in a different repository. Text outside the scope never reaches review at all, and no finding is produced even with the axis open.

All three are manageable decisions, and the number the table gives is the price of those decisions: closing the documentation axis means 3 defects are missed that round. What is useful is not the size of the number but knowing in advance which class the missed defects will belong to.

The Measurement’s Assumptions

• RA31 — The shared setup is unchanged: 600 lines, 12 chunks, 24 defects, attention 12 chunks. There is no unread chunk; every missed defect’s cause is an axis not looked at.
• RA32 — In the first part, every row is a single reviewer’s reading, and the only difference between rows is the closed axis. The first row closes no axis.
• RA33 — The “loss” column is the found count with all five axes open minus that row’s found count.
• RA34 — The class breakdown of missed is the count, by class, of defects not found in that row, written in descending order.
• RA35 — A documentation defect is a written text in the change contradicting the code. The setup keeps not where this text sits, but which chunk it falls into.
• RA36 — An unwritten-requirement defect is the defect of what was never written at all; falling into a chunk does not make it visible — the chunk only marks where the defect belongs.
• RA37 — When the documentation axis is open, only the documentation class is found; the sixth class is on no axis’s search list, and this measurement does not change that.
• RA38 — Resolution at the defect level is 1/24 = 0.042, at the chunk level 1/12 = 0.083.

Measurement

"""Documentation review: what escapes when an axis is closed.

Part 1 - the five axes closed one at a time; found, missed, and the class breakdown of missed.
Part 2 - the split between a written document's defect and a never-written requirement.
"""
SEED = 20260815
AXES = ("interface", "implementation", "test", "documentation", "style")
UNWRITTEN = "unwritten requirement"
CLASSES = AXES + (UNWRITTEN,)
ATTENTION = 12
CHUNK = 50


def rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def change(lines, defect_count=24, seed=SEED):
    draw, defects = rng(seed), []
    chunk_count = max(1, lines // CHUNK)
    for i in range(defect_count):
        defects.append({"no": i + 1, "class": CLASSES[draw(6)],
                         "chunk": draw(chunk_count)})
    return {"lines": lines, "chunks": chunk_count, "defects": defects}


def review(d, axes, attention=ATTENTION):
    read = set(range(min(attention, d["chunks"])))
    return {k["no"] for k in d["defects"]
            if k["class"] in axes and k["chunk"] in read}


FULL = set(AXES)
d = change(600)

print(f"{'closed axis':<16s} {'open':>4s} {'found':>7s} {'missed':>6s} "
      f"{'loss':>5s} {'missed class breakdown':<40s}")
full = review(d, FULL)
for a in ("-",) + AXES:
    open_axes = FULL if a == "-" else FULL - {a}
    b = review(d, open_axes)
    missed = [k for k in d["defects"] if k["no"] not in b]
    breakdown = {}
    for k in missed:
        breakdown[k["class"]] = breakdown.get(k["class"], 0) + 1
    text = ", ".join(f"{s} {n}" for s, n in
                      sorted(breakdown.items(), key=lambda x: -x[1]))
    print(f"{a:<16s} {len(open_axes):4d} {len(b):7d} {len(missed):6d} "
          f"{len(full) - len(b):5d} {text:<40s}")

print()
documentation = [k for k in d["defects"] if k["class"] == "documentation"]
unwritten = [k for k in d["defects"] if k["class"] == UNWRITTEN]
print(f"{'class':<20s} {'defects':>7s} {'chunks':<14s} {'found by documentation axis':>28s}")
print(f"{'documentation':<20s} {len(documentation):7d} "
      f"{str(sorted(k['chunk'] for k in documentation)):<14s} {'yes':>28s}")
print(f"{UNWRITTEN:<20s} {len(unwritten):7d} "
      f"{str(sorted(k['chunk'] for k in unwritten)):<14s} {'no':>28s}")
for name, open_axes in (("documentation open", FULL), ("documentation closed",
                                                        FULL - {"documentation"})):
    b = review(d, open_axes)
    missed = [k for k in d["defects"] if k["no"] not in b]
    print(f"{name:<22s} documentation missed "
          f"{sum(1 for k in missed if k['class'] == 'documentation')}, "
          f"unwritten missed "
          f"{sum(1 for k in missed if k['class'] == UNWRITTEN)}")

closed axis      open   found missed  loss missed class breakdown                  
-                   5      22      2     0 unwritten requirement 2                 
interface           4      19      5     3 interface 3, unwritten requirement 2    
implementation      4      16      8     6 implementation 6, unwritten requirement 2
test                4      19      5     3 test 3, unwritten requirement 2         
documentation       4      19      5     3 documentation 3, unwritten requirement 2
style               4      15      9     7 style 7, unwritten requirement 2        

class                defects chunks          found by documentation axis
documentation              3 [2, 2, 3]                               yes
unwritten requirement       2 [2, 10]                                  no
documentation open     documentation missed 0, unwritten missed 2
documentation closed   documentation missed 3, unwritten missed 2

Closing the Five Axes One at a Time

The top table is this topic’s closing table: same change, same single reviewer, the only difference is which axis is closed.

The loss column confirms a single rule in all five rows: 3, 6, 3, 3, 7. These numbers are the class distribution itself. The number of defects lost when an axis is closed equals the number of defects in that axis’s class — neither more nor less. The third lesson’s order-independence measurement showed this going forward; this table shows it going backward.

The size of the loss varies threefold from axis to axis. The most expensive closure is the style axis: 7 defects, found drops to 15. Next comes implementation: 6 defects, found 16. The interface, test, and documentation axes each cost 3 defects, and all three leave found sitting at 19.

The documentation axis looks like one of the cheapest closures in this table, and that appearance is the lesson’s warning. The table counts how many defects are missed; it does not count when the cost of a missed defect is paid. An implementation defect is paid once, when run. A documentation defect is paid every time it is read, and every reader gets the same wrong information. Two numbers land in the same column but are paid at two separate times. This distinction is outside the setup and cannot be extracted from the table; the table only gives found and missed counts.

One row of the table overlaps exactly with the first lesson’s table. When the style axis is closed, found is 15; in the first lesson, the “three reviewers, separate axes” configuration also found 15, and that panel’s covered axes were interface, implementation, test, and documentation — the four axes other than style. Two separate measurements, two separate paths, arrive at the same number. This is not a coincidence, it is the result of additivity: whatever number of people the same set of axes is spread across, it finds the same set.

The table’s practical use is a price list. An axis closing is often not a decision explicitly made; it happens on its own, from no one being assigned to that axis or that axis’s finding not being treated as a blocker. The table gives that decision’s price in advance, and the prices are not equal: closing the style axis costs 7, implementation 6, the others 3 each. The choice between “open one more axis” and “add one more person” sits inside the same measurement too: the second decision’s return was measured at 0 in the third lesson, the first’s is between 3 and 7 here.

The breakdown column on the right carries the same two words in every row: unwritten requirement 2. It stands there in all six rows; no axis removes it.

A Written Document and an Unwritten Requirement

The bottom table places these two classes side by side and shows where the difference is not.

The documentation class’s three defects sit in chunks 2, 2, and 3. The unwritten requirement class’s two defects sit in chunks 2 and 10. The two classes’ defects meet in the same chunk: chunk two holds both two documentation defects and one unwritten-requirement defect. Same fifty lines, same reading, same reviewer.

When the documentation axis is open, documentation missed is 0; the two documentation defects in the second chunk are found. The unwritten-requirement defect in that same chunk is not found, and unwritten missed stays at 2. When the documentation axis is closed, documentation missed climbs to 3, and unwritten missed is still 2.

The difference is therefore not a difference of location. The two defects sit in the same chunk and the reading passes over both. The difference is whether there is a text to look at. In the documentation defect, a written sentence exists and can be compared against the code; in the unwritten-requirement defect, no sentence to compare ever got written.

This is the documentation axis’s most confused trait. When it is open, a review saying “documentation was looked at too” is correct — but what was looked at is what was written. An unwritten requirement is outside the documentation axis too, because the documentation axis, in the end, also compares two texts, and if one of the two texts does not exist, no comparison can be made. All five axes stay inside the same definition: review looks at what was written.

This gives the documentation axis’s boundary case. If a behavior was never documented at all, is this a documentation defect or an unwritten requirement? The distinction looks at whether the text was expected. If the change introduces a new public name, that name’s documentation is expected, and its absence falls into the documentation class — this is exactly the second finding type, and the documentation axis finds it. In contrast, a case the change never addresses at all produces no text the documentation axis would expect, and stays in the sixth class.

The boundary’s practical counterpart is a single sentence: the documentation axis asks whether what the change introduces is documented; it does not ask what it failed to introduce. The thing that asks the second question is not review, it is the decision that sets the change’s scope.

Summary

• The documentation axis compares not one text but two: what the code does and what the text says it does. Its finding is a contradiction and cannot be found by reading the documentation alone.
• There are three finding types: stale text (invisible in the change because it did not change), missing text (the added name has no counterpart), contradicting text (two places describe two separate behaviors).
• The loss from closing an axis equals its class size: 3, 6, 3, 3, 7. The most expensive closure is the style axis, dropping found to 15.
• Closing the documentation axis drops found from 22 to 19. The table counts missed defects, not when the cost is paid: a documentation defect is repaid every time it is read.
• Documentation defects sit in chunks 2, 2, 3, unwritten-requirement defects in chunks 2 and 10; both are present in the second chunk. With the documentation axis open, documentation missed is 0, unwritten missed stays 2.
• The difference is not location but the presence of a text: a documentation defect has a sentence to compare, an unwritten requirement does not.

Next Step

The table’s most expensive row is the style axis: closing it misses 7 defects and found drops to 15. This is the five axes’ most crowded class and, at the same time, the one requiring the least information to see. The two traits together open a separate question: does this axis require a human’s reading at all. The next lesson measures the style axis handed off to a tool — how the total found changes, what the class distribution of the human’s finding becomes, and how many reviewers it takes to produce the same result.