Staleness and Audit

The previous two lessons built two defenses, and both stayed incomplete. Review cannot see claims on the name and concept surface; generation cannot generate claims on the name and flow surface. What is left over from both defenses is a remainder that announces its wrongness in no way at all.

This lesson counts that remainder. The course’s central question is asked exactly here: how many of a document’s claims go wrong over time, and how many of the wrong ones say so on their own? The second number matters more than the first, because a claim that announces itself is not a problem, it is a notification. A claim that does not announce itself is found only when someone goes looking for it — and the name of that search is documentation audit.

Two Kinds of Staleness

The shared setup’s distinction is simple and does not change through the course. 17 of fifty-six claims are embedded in a runnable example. When an embedded claim goes stale, the example fails: the reader copying it sees an error, and if a continuous integration pipeline runs the example, the pipeline itself breaks. This claim makes noise.

The remaining 39 claims are plain sentences. A field’s type, a step’s order, something’s name, a decision’s reasoning — all of them stand inside the text, and when they go wrong, nothing happens. The page still opens, the pipeline still passes, the search still finds it. This is silent staleness, and it stands at the center of the course’s measure.

The distinction’s operational consequence is this: you have two kinds of evidence about a document’s health, and one is measurable, the other is not. A broken example can be counted. A silently wrong sentence cannot be counted — if it could, it would already have been fixed. The definition of silent staleness is that it does not get itself counted. The measurement can only do this with an oracle built into the setup: we know which claim goes stale at which version, because we wrote it.

What a Documentation Audit Does

A documentation audit is the job of comparing claims against their source one by one. A person reads the text, verifies whether every sentence is still true by checking the source, and flags the wrong ones. This job’s cost is measured by the number of claims read; its gain is the number of silent stale claims found.

An audit’s scope is a choice. The whole document can be read, or just one type. The choice of scope is directly a yield question: how many silent stale claims are found per claim read? Claims that make noise are outside this computation — they are found regardless of scope, because they have already announced themselves.

An audit’s output is a list, not a fix. The setup does not measure whether a found stale claim gets corrected; the only thing it measures is whether it gets found. There is no way to fix a stale claim that was never found.

# example audit output, not executed

reference / submit_metric
  [stale]  the source field is required        -> the definition has a default value
  [stale]  returns: record number (integer)     -> the definition returns text
  [true]   the unit field is required

tutorial / first measurement
  [stale]  the session opens first, then the measurement is submitted -> order changed

The list’s format does not matter; what matters is that every line is a claim. A documentation audit does not count pages, does not count paragraphs, it counts claims. This is the measure’s unit.

The Measurement’s Assumptions

• DO21 — Fifty-six claims, four types, and four surfaces are taken from the shared setup. The oracle is the setup itself; which claim goes stale at which version is known.
• DO22 — Surface frequencies do not change: signature once every two versions, flow every three, name every five, concept every twelve. A claim is stale if the surface it is bound to has changed at least once by this version.
• DO23 — Seventeen of fifty-six claims are embedded in an example; an embedded stale claim makes noise, a non-embedded one is silent. Embeddedness is determined together with the surface, and the concept surface has no embedded claims at all.
• DO24 — A documentation audit reads every claim in its scope and compares it against the oracle; it does not miss a stale claim it reads. This is an optimistic assumption and gives the upper bound of what an audit can find.
• DO25 — A stale claim that makes noise is found regardless of audit scope; the example is already failing. Audit scope only determines whether silent staleness is found.
• DO26 — Audit’s cost is the number of claims read. Yield is the ratio of silent staleness found per claim read, and it is undefined when no claim is read.
• DO27 — Audit runs separately at two versions: version three and version twelve. The gap between the two shows that the audit’s target changes with version.
• DO28 — The set’s resolution at fifty-six claims is 1/56; a difference smaller than this cannot be defended with this setup.

Measurement

"""Staleness and documentation audit: the cost of finding the silent one.

Part 1 - four types' true/stale/silent triples across six versions.
Part 2 - audit scopes: silent staleness found per claim read.
"""
TYPES = {
    "tutorial":    {"flow": 9, "signature": 3, "name": 2, "concept": 1},
    "how-to":      {"flow": 6, "signature": 5, "name": 2, "concept": 1},
    "explanation": {"flow": 1, "signature": 1, "name": 1, "concept": 9},
    "reference":   {"flow": 0, "signature": 13, "name": 2, "concept": 0},
}
FREQUENCY = {"signature": 2, "flow": 3, "name": 5, "concept": 12}
EMBEDDED = {"tutorial": 0.60, "how-to": 0.70, "explanation": 0.10,
            "reference": 0.00}
VERSIONS = (0, 1, 2, 3, 6, 12)


def claims():
    entries, no = [], 0
    for kind, mix in TYPES.items():
        embedded_target = 0
        for surface, count in mix.items():
            for _ in range(count):
                no += 1
                embedded_target += EMBEDDED[kind]
                embedded = embedded_target >= 1 and surface != "concept"
                if embedded:
                    embedded_target -= 1
                entries.append({"no": no, "type": kind, "surface": surface,
                                 "embedded": embedded})
    return entries


def stale(claim, version):
    return version // FREQUENCY[claim["surface"]] >= 1


def measure(entries, version, kind=None):
    group = [i for i in entries if kind is None or i["type"] == kind]
    stale_list = [i for i in group if stale(i, version)]
    silent = [i for i in stale_list if not i["embedded"]]
    return len(group), len(group) - len(stale_list), len(stale_list), len(silent)


def audit(entries, version, scope):
    """Scope: types read by hand. Embedded staleness is found outside scope too."""
    read = [i for i in entries if i["type"] in scope]
    stale_list = [i for i in entries if stale(i, version)]
    found = [i for i in stale_list if i["embedded"] or i in read]
    found_silent = [i for i in found if not i["embedded"]]
    missed = [i for i in stale_list if not i["embedded"] and i not in read]
    return len(read), len(found), len(found_silent), len(missed)


L = claims()
print(f"claims {len(L)}, types {len(TYPES)}, embedded in example "
      f"{sum(1 for i in L if i['embedded'])}")
print("surface distribution: " + ", ".join(
    f"{y} {sum(1 for i in L if i['surface'] == y)}" for y in FREQUENCY))
print()
print(f"{'version':>7s}" + "".join(f" {t:>18s}" for t in TYPES))
print(f"{'':7s}" + "".join(f" {'true/stale/silent':>18s}" for _ in TYPES))
for v in VERSIONS:
    line = f"{v:7d}"
    for kind in TYPES:
        _, true, stale_n, silent = measure(L, v, kind)
        line += f" {f'{true}/{stale_n}/{silent}':>18s}"
    print(line)
print()
for v in (3, 12):
    _, _, stale_n, silent = measure(L, v)
    print(f"version {v} stale {stale_n}/{len(L)}, silent {silent} "
          f"({silent / stale_n:.4f})")
SCOPES = [("no audit", ()), ("reference only", ("reference",)),
          ("explanation only", ("explanation",)),
          ("tutorial only", ("tutorial",)),
          ("how-to only", ("how-to",)),
          ("all four types", tuple(TYPES))]
for v in (3, 12):
    print(f"\nversion {v}")
    print(f"{'audit scope':>21s} {'read':>7s} {'stale found':>14s} "
          f"{'silent found':>15s} {'silent missed':>13s} {'yield':>7s}")
    for name, scope in SCOPES:
        read, found, found_s, missed = audit(L, v, scope)
        yield_ = f"{found_s / read:.4f}" if read else "—"
        print(f"{name:>21s} {read:7d} {found:14d} {found_s:15d} "
              f"{missed:13d} {yield_:>7s}")

claims 56, types 4, embedded in example 17
surface distribution: signature 22, flow 16, name 7, concept 11

version           tutorial             how-to        explanation          reference
         true/stale/silent  true/stale/silent  true/stale/silent  true/stale/silent
      0             15/0/0             14/0/0             12/0/0             15/0/0
      1             15/0/0             14/0/0             12/0/0             15/0/0
      2             12/3/1              9/5/2             11/1/1            2/13/13
      3             3/12/5             3/11/4             10/2/2            2/13/13
      6             1/14/6             1/13/4              9/3/3            0/15/15
     12             0/15/7             0/14/5            0/12/12            0/15/15

version 3 stale 38/56, silent 24 (0.6316)
version 12 stale 56/56, silent 39 (0.6964)

version 3
          audit scope    read    stale found    silent found silent missed   yield
             no audit       0             14               0            24       —
       reference only      15             27              13            11  0.8667
     explanation only      12             16               2            22  0.1667
        tutorial only      15             19               5            19  0.3333
          how-to only      14             18               4            20  0.2857
       all four types      56             38              24             0  0.4286

version 12
          audit scope    read    stale found    silent found silent missed   yield
             no audit       0             17               0            39       —
       reference only      15             32              15            24  1.0000
     explanation only      12             29              12            27  1.0000
        tutorial only      15             24               7            32  0.4667
          how-to only      14             22               5            34  0.3571
       all four types      56             56              39             0  0.6964

Top Table: Four Types, Six Versions

Versions zero and one are empty; no surface has changed yet and all four types keep every claim. At version two, signature changes and the table splits immediately: reference gives 2/13/13 while explanation gives 11/1/1. Two texts, same version, same repository, written with the same care; one kept eighty-seven percent of its claims, the other kept thirteen percent. The difference does not come from the writing, it comes from the surface bound to.

At version three, the flow surface changes, and this time tutorial and how-to collapse: 3/12/5 and 3/11/4. Explanation survives with 10/2/2, because nine of its claims are bound to concept, and concept changes once every twelve. By version six, the name surface has also turned over; by version twelve, concept turns over too and all four types reset: 56/56.

Not all stale counts carry the same weight. At version three, stale is 38, silent is 24; ratio 0.6316. At version twelve, stale is 56, silent is 39; ratio 0.6964. The second number is the course’s binding result: roughly seventy percent of staleness is silent.

The ratio’s climbing with version is not a coincidence. Embedded claims concentrate on the signature and flow surfaces, and these two surfaces change early; their staleness happens in the early versions and makes noise. Name and concept change late, carry no embedded claims, and their staleness is silent. As time passes, the silent share grows. A document does not just get more wrong as it ages, it also gets more silently wrong.

Whether this rise is measurable also has to be asked. The set’s resolution at fifty-six claims is 1/56, that is, 0.0179; a smaller difference cannot be defended with this setup. The gap between the two ratios is 0.6964 − 0.6316 = 0.0648, more than three times the resolution. The rise is inside the measurement band. By contrast, observing a half-claim shift between two versions is impossible with this set; the setup is not built for finer distinctions.

Lower Tables: Audit Yield

The no audit row is the baseline. At version three, without anyone reading anything, 14 stale claims are found and 24 silent ones slip through; at version twelve, 17 are found and 39 slip through. The found count is exactly the embedded claims’ broken examples. This is the only feedback the document gives on its own, and it is limited to roughly a third of fifty-six claims.

The scope rows show where an audit should be aimed. At version three, reading reference only finds 13 silent stale claims for fifteen claims read; yield 0.8667. In the same version, reading explanation only finds 2 for twelve claims read; yield 0.1667. The gap is more than fivefold, and the effort spent by the person doing the audit is nearly the same.

The table flattens at version twelve. Reference and explanation both give 1.0000: every claim read is a silent stale one, because neither carries an embedded claim and everything has gone stale. Tutorial gives 0.4667, how-to 0.3571; in these two types, part of the claims read are embedded and were already found — the audit reads them for nothing.

Two operating rules follow from this. First: the audit’s target is not the type, it is the surface. Reference being an early high yield does not come from its writing style, it comes from thirteen of its claims being bound to signature. Second: a type with many embedded claims is a poor audit target. Staleness there already announces itself; the audit’s reading effort should go to what does not announce itself.

The full-scope row closes with two numbers. At version three, reading all fifty-six claims finds 24 silent stale ones and yield is 0.4286; at version twelve it finds 39 and yield is 0.6964. This second number is identical to the top table’s silence ratio, and by definition it must be: a fully scoped documentation audit’s yield is exactly that document’s silent-staleness ratio. However well the audit is run, the limit of what it can find is how many of the document’s claims are silently wrong.

What Triggers an Audit

If scope determines audit yield, something has to determine scope. A calendar is a poor determinant: an audit done every three months reads the same claims regardless of which surface changed, and part of the effort goes to sentences that never changed at all.

A better determinant is the repository’s own record. That a surface changed is written inside the record that made that change: when a field is added, the signature has changed; when a step is added, the flow has changed. The Introduction to Version Control course established how to read this record, and the Branching and Collaboration course measured which questions history can answer. The use here is narrow: history tells the audit which surface to aim at.

This method has a limit, and it shows up for the second time in the course. History carries a trace when code changes; the name and concept surfaces change outside the code. There may be no record in history at all that a product’s name changed or a reasoning became invalid. At version twelve, 18 claims stand on these two surfaces, and all of them are stale. An audit triggered by history never looks for them; they need another source to pull the trigger.

Summary

• Staleness has two kinds: an embedded claim makes noise when it goes stale, a non-embedded claim goes silently stale and gives no sign.
• At version two, reference gives 2/13/13, explanation 11/1/1; what determines two equally carefully written texts’ lifespan is the surface they are bound to.
• At version twelve, 56 of fifty-six claims are stale and 39 are silent; the ratio is 0.6964, higher than version three’s 0.6316.
• Without a documentation audit, only embedded staleness is found: 17 are found at version twelve, 39 slip through.
• Audit yield varies with scope: at version three, reading reference only gives 0.8667, reading explanation only gives 0.1667; the target is the surface, not the type.
• A fully scoped audit’s yield is exactly the document’s silence ratio — 0.6964 at version twelve.

Next Step

A documentation audit finds silent staleness, but it requires someone to sit down and read fifty-six claims, and that job takes no place on anyone’s calendar by itself. As long as the audit is not done, those thirty-nine claims stand in place and someone reads them every day. The next lesson looks at those readers: what does a reader who hits a wrong sentence do, how many report it, and after how many visits does how much of the silent staleness become visible. The course’s final lesson measures the feedback loop, the third claim’s only antidote.