Docs Alongside Code

Every measurement up to this point took the document as it was the moment it was written. The four types, their surface mixes, heading layout, and whether examples ran — all of it was measured at a single moment, version zero. At that moment a document is a text, and every one of the text’s claims is true.

A document does not stay at that moment. What it describes keeps changing, and if the document does not change with it, the claim and reality drift apart. This lesson’s question is not how the document is written, it is where it stands: whether the documentation lives next to the code or somewhere separate; whether it enters the same record as the change, or gets gone over on a separate calendar. We will build three operating regimes and track the same fifty-six claims through twelve versions in all three.

Three Operating Regimes

Separate calendar. The documentation lives outside the code; it has its own repository, its own publishing schedule. No one opens it together with a change. At fixed intervals — every five versions, in our setup — someone sits down and goes over the whole document. At that pass every claim is refreshed, then the document is left alone again.

Versioned alongside code. The documentation is inside the source tree and enters the same record as the change itself. This is the core of the docs as code approach: the documentation is not a separate entity, it is a file of the repository. The author, while changing the code, can also touch the documentation in the same record. They do not have to.

Versioned alongside code and brought into review. The change is read by someone else, and the reader also sees the documentation. Because the documentation’s lines stand on the same screen as the code’s lines, the sentence “you renamed this field, the reference still shows the old name” becomes possible to construct.

All three regimes spend the same writing effort. What differs between them is when the documentation gets looked at, not how well it is written.

The third regime has two conditions, and both are structural. First, the documentation file has to stand inside the change: a documentation fix filed as a separate record does not appear on the same screen as the code change it fixes. Second, a missing documentation update has to be counted by the reviewer as a defect class: of two people looking at the same list, one reads an untouched file as a gap, the other does not. The measurement assumes the second condition holds; where it does not, the third regime collapses into the second and the table’s gap disappears.

The Surface Review Sees

Review’s own axis was established in the Code Review and Team Process course: review sees only in the axis it looks at and only as far as the chunk it reads. That discussion is not repeated here. What we need is only this boundary: the reviewer reads what is visible inside the change.

This boundary splits the four surfaces in two. Signature and flow are visible inside the change: when a field’s type changes or the order of steps changes, the lines producing that change are inside the record, and the reviewer reads both side by side. Name and concept are invisible: something’s externally given name changes on the product side, a reasoning changes in a discussion; neither may leave a trace in the source line. When the reviewer reads the old name in the documentation, they have no reason to suspect it is wrong.

Where the documentation stands inside the change is visible the same way:

# example change listing, not executed

change: unit field added to the measurement interface
  source/metrics.py                 +14  -3
  source/metrics_validate.py         +6  -0
  docs/reference/metrics.md          +9  -4
  docs/tutorial/first-metric.md       0   0

The fourth line is what this lesson cares about. The reference is updated, the tutorial is untouched. The reviewer sees this list and can notice the tutorial went untouched; but to notice, they need to know which claim of the tutorial is affected by this change. What they can know is signature and flow; what they cannot is name and concept.

The Documentation’s Version

Putting the documentation next to the code has a second consequence independent of review: the documentation enters the repository’s versioning scheme. A documentation change made on a branch belongs to that branch; when a version tag is placed, that tag points at the documentation too. The criterion the Introduction to Version Control course established applies directly here — if the record was written to the object database, it can be brought back. The documentation’s state at version three can be read from the version-three tag.

Documentation run on a separate calendar does not carry this property. It has a single state: the current one. A reader using version three reads a text describing version six and cannot measure the gap. This is a different kind of wrongness from the staleness we measure: the claim is true for today’s code, false for the code the reader has in hand. Our measurement does not count this, because the oracle knows every version individually; in a real setup, which version the reader is on is not known, and this uncertainty cannot be closed by the documentation.

The versioning decision also has a direct cost. If the documentation moves forward on the same branch as the code, a fix reaches a published version only by being carried over to that version’s branch; on a separate calendar, a fix is instantly visible to everyone. The docs-as-code approach binds correctness to the version and, in exchange, multiplies the publishing work. Our setup does not measure this cost; the only thing it measures is whether a claim stays true.

The Measurement’s Assumptions

• DO1 — Fifty-six claims, four types, and four surfaces are taken from the shared setup. The oracle is the setup itself: because we wrote it, we know which claim is bound to which surface and at which version it goes stale.
• DO2 — Surface frequencies do not change: signature changes once every two versions, flow every three, name every five, concept every twelve. A claim goes stale the moment the surface it is bound to changes.
• DO3 — Seventeen of the fifty-six claims are embedded in a runnable example. An embedded claim makes noise when it goes stale: the example fails. A claim that is not embedded makes no sound when it goes stale; this is silent staleness.
• DO4 — In the separate-calendar regime, the document is gone over at versions five and ten, and at that pass every claim is refreshed. In the versions between, no one looks at the document.
• DO5 — In the versioned-alongside-code regime, only embedded claims are repaired: staleness announces itself because the example fails, and it gets fixed in the same record.
• DO6 — In the reviewed regime, claims bound to the signature and flow surface are repaired in addition to embedded claims; those on the name and concept surface are not repaired, because they are not visible inside the change.
• DO7 — The measure’s unit is the stale version-claim count: at the end of every version, the claims standing stale are counted and summed across twelve versions. The upper bound is fifty-six times twelve, that is, 672.
• DO8 — Repair happens within the version the claim goes stale. The difference between regimes is not the speed of repair, it is which claim never gets touched at all.

Measurement

"""The effect of bringing documentation into review on staleness.

Part 1 - three operating regimes, stale version-count over twelve versions.
Part 2 - surfaces' embeddedness and visibility-in-review distribution.
"""
TYPES = {
    "tutorial":    {"flow": 9, "signature": 3, "name": 2, "concept": 1},
    "how-to":      {"flow": 6, "signature": 5, "name": 2, "concept": 1},
    "explanation": {"flow": 1, "signature": 1, "name": 1, "concept": 9},
    "reference":   {"flow": 0, "signature": 13, "name": 2, "concept": 0},
}
FREQUENCY = {"signature": 2, "flow": 3, "name": 5, "concept": 12}
EMBEDDED = {"tutorial": 0.60, "how-to": 0.70, "explanation": 0.10,
            "reference": 0.00}
SEEN = ("signature", "flow")   # surfaces code review notices
OVERHAUL = (5, 10)             # versions when the separate-calendar doc is overhauled
LAST = 12


def claims():
    entries, no = [], 0
    for kind, mix in TYPES.items():
        embedded_target = 0
        for surface, count in mix.items():
            for _ in range(count):
                no += 1
                embedded_target += EMBEDDED[kind]
                embedded = embedded_target >= 1 and surface != "concept"
                if embedded:
                    embedded_target -= 1
                entries.append({"no": no, "type": kind, "surface": surface,
                                 "embedded": embedded})
    return entries


def changed(version):
    return {s for s, f in FREQUENCY.items() if version % f == 0}


def repaired(regime, claim, version):
    if regime == "separate calendar":
        return version in OVERHAUL
    if regime == "versioned together":
        return claim["embedded"]
    return claim["embedded"] or claim["surface"] in SEEN


REGIMES = ("separate calendar", "versioned together", "versioned + reviewed")


def run(regime, entries):
    true = {i["no"]: True for i in entries}
    debt = 0
    for version in range(1, LAST + 1):
        d = changed(version)
        for i in entries:
            if i["surface"] in d:
                true[i["no"]] = False
            if not true[i["no"]] and repaired(regime, i, version):
                true[i["no"]] = True
        debt += sum(1 for i in entries if not true[i["no"]])
    stale = [i for i in entries if not true[i["no"]]]
    silent = [i for i in stale if not i["embedded"]]
    return debt, len(entries) - len(stale), len(stale), len(silent)


L = claims()
print(f"claims {len(L)}, embedded in example {sum(1 for i in L if i['embedded'])}, "
      f"surface review sees {sum(1 for i in L if i['surface'] in SEEN)}")
print()
print(f"{'regime':>20s} {'stale version-claims':>21s} {'true':>6s} {'stale':>6s} "
      f"{'silent':>7s}")
for regime in REGIMES:
    debt, true, stale, silent = run(regime, L)
    print(f"{regime:>20s} {f'{debt}/{len(L) * LAST}':>21s} {true:6d} {stale:6d} "
          f"{silent:7d}")
print()
print(f"{'surface':>10s} {'claims':>6s} {'embedded':>8s} {'visible in review':>19s}")
for y in FREQUENCY:
    g = [i for i in L if i["surface"] == y]
    print(f"{y:>10s} {len(g):6d} {sum(1 for i in g if i['embedded']):8d} "
          f"{'yes' if y in SEEN else 'no':>19s}")

claims 56, embedded in example 17, surface review sees 38

              regime  stale version-claims   true  stale  silent
   separate calendar               299/672      7     49      35
  versioned together               300/672     17     39      39
versioned + reviewed                43/672     41     15      15

   surface claims embedded   visible in review
 signature     22        5                 yes
      flow     16        9                 yes
      name      7        3                  no
   concept     11        0                  no

The right three columns give the state at the end of the twelfth version: how many claims stayed true, how many are stale, how many of the stale ones are silent.

Same Debt, Different Distribution

The first two rows’ totals are nearly identical: 299 and 300. A document run on a separate calendar and a document versioned alongside code but not brought into review spend almost the same amount of time wrong across twelve versions. Moving it next to the code alone produced no gain.

The totals are equal; the distributions are entirely different. In the separate-calendar regime, no claim is permanently stale — the passes at versions five and ten reset everything — but between two passes, every kind of claim stands stale. Only 7 claims stay true at version twelve, and these are the ones whose surface has not changed since the last pass: the name claims refreshed at version ten, because name changes once every five versions and its next change is at version fifteen. The number seven is the full count of claims bound to the name surface.

This also explains where the separate-calendar regime’s debt comes from. If the overhaul interval is longer than the fastest surface’s frequency, that surface is already stale between two passes. A five-version interval does nothing against a signature that changes every two versions: the twenty-two signature claims fall wrong again at the second version after every pass and stay that way until the next pass. Shortening the interval lowers the debt but grows the reading work linearly; every pass means fifty-six claims verified by hand.

In the versioned-alongside-code regime, the table flips: 17 claims are always true and 39 claims are stale at version twelve. Seventeen is familiar — it is exactly the count of embedded claims. Thirty-nine is familiar too: all of the stale ones are silent. This regime repairs the one that makes noise immediately and never touches the one that does not. The documentation is next to the code, it passes before the eye at every change, and thirty-nine claims stay wrong for years.

Versioning alongside code is a relocation decision, not a maintenance decision. Moving a file’s location places no obligation on anyone to read it.

What Review Saves and What It Cannot

The third row drops the debt from 300 to 43 — sevenfold. The gap comes from a single thing: the reviewer also reads the documentation claims bound to the surface the change touches. The lower table gives the scope: 38 claims are bound to the two surfaces review sees, more than two-thirds of the fifty-six.

15 claims remain, and all of them are silent in the table. These are claims bound to the name and concept surface that are not embedded in an example: 7 − 3 = 4 on the name surface, 11 on the concept surface. The embedded count being zero in the concept column is a rule of the setup — a reasoning cannot be embedded in a runnable example.

These fifteen claims produce no warning in any of the three regimes. The example runs, review passes, no one complains. At version twelve the document carries 41 true claims and looks quite good to the reader; none of the 15 wrong sentences it carries shows itself. The course’s central problem shows up exactly here, and the next two lessons build on top of it: the visible portion of staleness and the existing portion of staleness are not the same thing.

The regimes’ cost is not equal either. Separate calendar demands two full overhauls; all fifty-six claims are verified by hand twice. The reviewed regime demands no extra pass at all, because verification is folded into work already being done — reading the change. The gap between them is the decision of whether documentation maintenance is a separate job or a field of an existing job.

What the Measure Does Not Say

The third regime’s 43 is a lower bound, and an optimistic one. Assumption DO6 says review catches every claim it can see; the Code Review and Team Process course’s result showed this is not true — review sees only as far as the chunk it reads even on the axis it looks at, and it does miss defects. Documentation lines are the most readily skipped part of that chunk, because they do not block the build from working. In a real setup, the third row’s debt sits somewhere above 43 and below the second row’s 300.

Two numbers are unaffected by this optimism. The 15 claims on the name and concept surface can never be caught in this regime no matter how careful review is: they are not visible inside the change, so there is no place to look. The 17 embedded claims are equally certain in the opposite direction: they need no review at all, because the example announces itself the moment it fails to run. The choice of regime only changes the fate of the 24 claims in between.

Summary

• Documentation can run in three regimes: reviewed on a separate calendar, versioned alongside code, or versioned alongside code and brought into review.
• Separate calendar produces 299/672, versioned-alongside-code produces 300/672 stale version-claims; equal debt, entirely different distribution.
• Versioned-alongside-code leaves 17 true claims at version twelve, and 39 of the 39 stale claims are silent; relocation alone is not maintenance.
• Review drops the debt to 43/672 and raises true claims to 41, because it sees the 38 claims on the signature and flow surface.
• The remaining 15 claims are on the name and concept surface, invisible to review, and all silent; this number is the starting point of the next lessons.

Next Step

Review works because it is a person reading the claim, and a reading person is expensive. There is a cheaper path for one of the surfaces: instead of writing the claim by hand, generate it from the source it is bound to. A generated claim changes along with its source, and its going stale becomes unthinkable. The next lesson separates production approaches by class name and measures which surface generation actually saves, and which two surfaces it never saves at all.