Documentation Production Approaches

The previous lesson showed that review drops the debt sevenfold, but this gain costs a person also reading the documentation lines at every change. A reading person is expensive, and their attention is not fixed. There is a cheaper path for one surface: instead of writing the claim by hand, generate it from the source it is bound to.

A generated claim changes on its own when its source changes. No one has to remember it, no one has to review it; its going stale becomes unthinkable. This lesson’s question is how far this goes. How many of the fifty-six claims can be read from a source, how many cannot, and which surface are the ones that cannot on? The measurement places three production approaches side by side with the hand-written document and tracks all four across the same twelve versions.

A Claim’s Source

A claim’s source is the place that determines its correctness. The sentence “this field’s type is an integer” has its source in the line where the field is defined. The sentence “the measurement is created first, then submitted” has its source in the mechanism that requires this order. The sentence “this structure is called a unit” has its source in the decision that gave it the name. The sentence “the unit field is required, because measurements cannot be compared without a unit” has its source in the reasoning itself.

The difference is this: the first source is machine-readable, the other three are not. A field’s type is written inside the source and can be pulled from there. The order of steps, a name, and a decision’s reasoning do not stand anywhere inside the source; only a person knows them. Production is built on top of this distinction, and this distinction is production’s boundary.

Three Approach Classes

Approaches are referred to not by product name but by class name; products change, classes stay.

Generation from a markup language. The document is written in a markup language and a converter turns it into a publishable form. Links are checked, a table of contents is extracted, page layout is applied. What is produced is format. The text’s claims stand as written and the source is never looked at.

Generation from a doc comment. Claims are written inside the source, right next to the item they describe; a tool collects them into a document. Part of the collected text is hand-written, part is read from the definition itself: the parameter’s name, its type, the return value. Its scope is narrow — only text describing the source can stand next to the source. A tutorial or a piece of reasoning does not fit there.

Generation from source. The interface’s definition is read directly and the reference text is generated from it. Its scope is wide: whatever the document type, the signature claims inside it can come from the source. Text obtained this way is called generated reference.

What a generated fragment looks like is below. The setup belongs to a product and is not executed:

# generated reference fragment, example dump, not executed

submit_metric(value, unit, source=None)

  value    decimal        required
  unit     text           required
  source   text or empty  default: empty

  returns: record id (text)

Every line in this fragment is read from the definition. If a field is added, a line is added; if a type changes, a column changes. What the fragment does not say is equally clear: why the unit field is required, in what order it should be called, and where the name given to this function comes from cannot be generated.

Generated text has one more property, and it does not enter the measurement: its order is the source’s order, not the reader’s. Fields are listed in the order they are defined, functions are arranged by their place in the source. The scannable layout measured in the Writing Discipline topic is not built here, because the tool doing the generation does not know which question the reader arrived with. This is not a shortcoming of generated reference, it is its definition: its completeness comes from the source, its layout does not. A document’s generated sections and its hand-written sections therefore do not substitute for each other; one is always correct and disordered, the other ordered and capable of going stale.

The Measurement’s Assumptions

• DO11 — Fifty-six claims, four types, and four surfaces are taken from the shared setup; surface frequencies do not change. The oracle is the setup itself.
• DO12 — An approach is defined by two things: the types it covers and the surface it reads from source. A claim of a covered type bound to the read surface never goes stale at all.
• DO13 — Generation from a markup language covers all four types and reads no surface from source; what it produces is format.
• DO14 — Generation from a doc comment covers only the reference type and reads the signature surface from source; tutorial, explanation, or how-to text is not placed next to the source.
• DO15 — Generation from source covers all four types and reads the signature surface from source; a signature claim can be generated from the definition regardless of type.
• DO16 — No approach can read the name, flow, or concept surface; these three surfaces’ sources do not stand anywhere machine-readable.
• DO17 — Production does not change example-embeddedness. An embedded claim makes noise when it goes stale, a non-embedded one does not; the definition of silent staleness stays as in the shared setup.
• DO18 — The measurement does not count production’s setup and operating cost; it only gives the number of claims staying true, going stale, and going silently stale per version.

Measurement

"""The surface production approaches save and cannot save.

Part 1 - four approaches, true/stale/silent over six versions.
Part 2 - source-based generation's per-type gain and per-surface limit.
"""
TYPES = {
    "tutorial":    {"flow": 9, "signature": 3, "name": 2, "concept": 1},
    "how-to":      {"flow": 6, "signature": 5, "name": 2, "concept": 1},
    "explanation": {"flow": 1, "signature": 1, "name": 1, "concept": 9},
    "reference":   {"flow": 0, "signature": 13, "name": 2, "concept": 0},
}
FREQUENCY = {"signature": 2, "flow": 3, "name": 5, "concept": 12}
EMBEDDED = {"tutorial": 0.60, "how-to": 0.70, "explanation": 0.10,
            "reference": 0.00}
VERSIONS = (0, 1, 2, 3, 6, 12)
ALL_FOUR = ("tutorial", "how-to", "explanation", "reference")
# approach: (types covered, surface read from source)
APPROACH = {"hand-written": ((), None),
            "from markup": (ALL_FOUR, None),
            "from doc comment": (("reference",), "signature"),
            "from source": (ALL_FOUR, "signature")}


def claims():
    entries, no = [], 0
    for kind, mix in TYPES.items():
        embedded_target = 0
        for surface, count in mix.items():
            for _ in range(count):
                no += 1
                embedded_target += EMBEDDED[kind]
                embedded = embedded_target >= 1 and surface != "concept"
                if embedded:
                    embedded_target -= 1
                entries.append({"no": no, "type": kind, "surface": surface,
                                 "embedded": embedded})
    return entries


def stale(claim, version, approach):
    scope, read = APPROACH[approach]
    if claim["type"] in scope and claim["surface"] == read:
        return False
    return version // FREQUENCY[claim["surface"]] >= 1


def measure(entries, version, approach, kind=None):
    group = [i for i in entries if kind is None or i["type"] == kind]
    stale_list = [i for i in group if stale(i, version, approach)]
    silent = [i for i in stale_list if not i["embedded"]]
    return len(group), len(group) - len(stale_list), len(stale_list), len(silent)


L = claims()
print(f"claims {len(L)}, bound to signature "
      f"{sum(1 for i in L if i['surface'] == 'signature')}, remaining "
      f"{sum(1 for i in L if i['surface'] != 'signature')}")
print()
print(f"{'version':>7s}" + "".join(f" {a:>18s}" for a in APPROACH))
print(f"{'':7s}" + "".join(f" {'true/stale/silent':>18s}" for _ in APPROACH))
for v in VERSIONS:
    line = f"{v:7d}"
    for approach in APPROACH:
        _, true, stale_n, silent = measure(L, v, approach)
        line += f" {f'{true}/{stale_n}/{silent}':>18s}"
    print(line)
print()
print(f"{'type':>15s} {'claims':>6s} {'sig':>4s} {'hand 6':>7s} "
      f"{'source 6':>9s} {'hand 12':>8s} {'source 12':>10s}")
for kind in TYPES:
    sig = sum(1 for i in L if i["type"] == kind and i["surface"] == "signature")
    t, h6, _, _ = measure(L, 6, "hand-written", kind)
    _, s6, _, _ = measure(L, 6, "from source", kind)
    _, h12, _, _ = measure(L, 12, "hand-written", kind)
    _, s12, _, _ = measure(L, 12, "from source", kind)
    print(f"{kind:>15s} {t:6d} {sig:4d} {f'{h6}/{t}':>7s} {f'{s6}/{t}':>9s} "
          f"{f'{h12}/{t}':>8s} {f'{s12}/{t}':>10s}")
print()
print(f"{'surface':>10s} {'claims':>6s} {'read from source':>17s} "
      f"{'true at version 12':>19s}")
for y in FREQUENCY:
    g = [i for i in L if i["surface"] == y]
    true = sum(1 for i in g if not stale(i, 12, "from source"))
    print(f"{y:>10s} {len(g):6d} {'yes' if y == 'signature' else 'no':>17s} "
          f"{true:19d}")

claims 56, bound to signature 22, remaining 34

version       hand-written        from markup   from doc comment        from source
         true/stale/silent  true/stale/silent  true/stale/silent  true/stale/silent
      0             56/0/0             56/0/0             56/0/0             56/0/0
      1             56/0/0             56/0/0             56/0/0             56/0/0
      2           34/22/17           34/22/17             47/9/4             56/0/0
      3           18/38/24           18/38/24           31/25/11            40/16/7
      6           11/45/28           11/45/28           24/32/15           33/23/11
     12            0/56/39            0/56/39           13/43/26           22/34/22

           type claims  sig  hand 6  source 6  hand 12  source 12
       tutorial     15    3    1/15      4/15     0/15       3/15
         how-to     14    5    1/14      6/14     0/14       5/14
    explanation     12    1    9/12     10/12     0/12       1/12
      reference     15   13    0/15     13/15     0/15      13/15

   surface claims  read from source  true at version 12
 signature     22               yes                  22
      flow     16                no                   0
      name      7                no                   0
   concept     11                no                   0

Column headers abbreviate the approach class names: generation from a markup language, from a doc comment, and from source.

The First Two Rows Are Identical

The table’s fastest-read finding is that the first two rows are exactly the same. The hand-written document and the document generated from a markup language give the same triple in all six of six versions: at version twelve, both are 0/56/39.

The reason is in the definition. Generation from a markup language never looks at the claims’ source; it just shows what the text says in a neater form. Automating publishing is a real gain — links get checked, versions get separated, format gets unified — but it has nothing to do with correctness. A tool “producing documentation” does not mean what it produces is claims. The word production covers both jobs with the same name, and the table shows a fifty-six-claim gap between them. The question to ask when evaluating a setup is not what the tool produces, it is which surface it reads from which source; if the answer to this question is “none,” what is produced is format alone.

What Generation from Source Saves

The fourth column gives 56/0/0 at version two; the hand-written column gave 34/22/17 at the same version. The first crack opens only at version three, when the flow surface changes. At version twelve, generation from source leaves 22 true claims — against the hand-written document’s 0.

The number twenty-two reads directly: it is the count of claims bound to the signature surface. The lower table shows this one by one. In the signature row, 22 stay true at version twelve — all of them; in the other three rows, 0. Generation from source removes one surface entirely and touches none of the other three. The ratio is the measure’s shortest summary: 22 of fifty-six claims can be generated, 34 cannot.

Generation from a doc comment reads the same surface but covers only one type; at version twelve it leaves 13 true claims. Thirteen is the count of signature claims in the reference type. The nine-claim gap between the two is the signature claims inside the other three types, and they do not fit next to the source — a call example in the middle of a tutorial cannot stand in the source’s doc comment.

The Two Surfaces That Cannot Be Saved

Generation from source’s 34 stale claims at version twelve break down as: flow 16, concept 11, name 7. The course’s fifth reading brings two of this trio forward, because both also appear inside the reference type: name and flow.

Why can these two surfaces not be generated? Name is something’s externally given name, and it does not have to match the identifier in the source. The source saying submit_metric does not say whether the document should call it “submitting a measurement” or “creating a record”; the naming decision sits outside the source and leaves no trace there when it changes. Flow is the order of steps, and it cannot be derived from individual signatures. All three functions’ definitions can be generated correctly; which one gets called first is written in none of their definitions.

The lower table’s second column counts 23 claims on these two surfaces. They are unaffected by whether generation is set up: they go stale at the same rate whether in a document with generation set up or without. Generation does not save these claims, and it even makes them more dangerous: the document looks current, the signature tables are fresh, and the flow sentence between them is silently wrong.

Gain by Type

The middle table separates who generation benefits by type. At version six, reference gives 0/15 hand-written and 13/15 generated from source — this is the same comparison measured in the Document Types topic. Tutorial climbs from 1/15 to 4/15; the gain is three claims, and it stays three at version twelve too.

The gap comes from the types’ surface mix. Of reference’s fifteen claims, 13 are bound to signature; of tutorial’s fifteen, only 3 are. Tutorial primarily narrates flow, and flow cannot be generated. In the explanation type the table is even sharper: only 1 of twelve claims is bound to signature, generation from source gives 1/12 at version twelve, and the gain is a single claim.

The operating decision that follows is correct and narrow in scope: generation targets the reference type. Setting up generation for the other three types is an investment the surface mix does not repay in setup cost. Spending the same effort on review and documentation audit for those types keeps more claims true.

Generation’s Effect on Silence

The last column’s triple at version twelve is 22/34/22: 22 of thirty-four stale claims are silent. The same row in the hand-written document gives 0/56/39. Generation eliminated twenty-two stale claims, and while the silent count drops from thirty-nine to twenty-two, the count of things making noise drops too.

The arithmetic is short. 5 of the twenty-two signature claims are embedded in a runnable example. Because generation keeps these claims true, those five examples never break again. Of the remaining thirty-four claims, the embedded count is 17 − 5 = 12. So generation also narrows the channel that announces staleness: 5 of seventeen warning points close.

This is not a defect, it is a displacement. Before generation, the document produced its own signal for seventeen of fifty-six claims; afterward, it does so for twelve of thirty-four. In both cases, the only way to find the rest is for someone to sit down and compare the claim against its source. Generation shrinks this job; it does not eliminate it.

Summary

• A claim can be generated if its source is machine-readable; signature is such a surface, name, flow, and concept are not.
• Generation from a markup language produces format, not claims; in the table it gives triples exactly identical to the hand-written document, and both are 0/56/39 at version twelve.
• Generation from source keeps 22 claims true at version twelve; this number is the full count of claims bound to the signature surface.
• Generation from a doc comment covers only the reference type and stays at 13 claims; the scope gap is nine claims.
• 23 of the 34 claims generation cannot save are on the name and flow surface; the document can look current while these claims are silently wrong.
• Generation targets the reference type by surface mix: reference gains 13/15, tutorial 3/15, explanation 1/12.

Next Step

We now have two defenses in hand, and both are partial: review cannot see name and concept, generation cannot generate name and flow. What remains from both is claims that announce their wrongness in no way at all. The next lesson goes to the course’s center and counts that remainder: the four types’ true, stale, and silent triples across six versions, the silence ratio at version twelve, and the per-claim cost of the documentation audit needed to find the silent ones.