Documentation

The previous lesson bound a published version’s content to its name: in an immutable registry, an (name, version) pair names a single distribution. The package carries code to the installing side.

It does not carry how to use that code. That information comes from documentation, and documentation can be produced two ways: written by hand, or generated from the source. This course’s last lesson measures both with the same question: which outcome does generation unify, which does it not?

Generation’s Input and Output

Approaches to generating documentation from source were established in the Technical Writing and Documentation course; staleness and review were addressed there too. That discussion is not repeated here. The only thing measured is how many distinct outcomes generation drops to one.

This course does not write a tool’s name. The documentation generator is modeled here with a single function: it parses the source, selects functions whose name does not start with an underscore, reads each one’s signature and its summary text’s first line, and sorts the result by name. A real generator does far more — it builds links, processes type information, produces page layout; the only property the model keeps is that its output is entirely derived from the source.

The measurement defines four source versions: the canonical form, a form with function definitions reordered, a form with a private helper function added, and a form with two signatures changed. Alongside these stands a handwritten document; it was written with the old signatures and has not been updated by hand since that day.

The measurement’s assumptions:

• QP29 — Three functions are taken from the shared setup; the discount function’s body is not changed and the flawed ceiling stays in place.
• QP30 — The discount function’s summary text states that the ceiling is 25 percent; the body applies 20. This gap between the claim and the behavior is the setup itself.
• QP31 — The documentation generator only reads from the source and never compares any claim against behavior; the number of checks it performs is 0.
• QP32 — The handwritten document is the same across all four source versions; it does not track the source at all.
• QP33 — Runnable examples are run with the standard library’s doctest fixture. What a doctest is was established in this course’s doctests lesson; here it is used as a verification tool, not rebuilt.
• QP34 — No duration is measured, no file is written. What is counted is distinct document, stale signature, and failing example count.

Measurement

"""Documentation: what generation unifies, what it does not."""

import ast
import doctest
import hashlib
import types

BODY = {
    "discount": (
        "(amount, member, coupon)",
        '"""Applies the member and coupon discount; the rate ceiling is 25 percent.\n\n'
        "    >>> discount(100, True, True)\n    75\n    \"\"\"\n"
        "    rate = 0\n    if member:\n        rate += 10\n"
        "    if coupon:\n        rate += 15\n"
        "    if rate > 20:\n        rate = 20\n"
        "    return amount - amount * rate // 100\n"),
    "eligible": (
        "(package)",
        '"""Returns the versions falling within the declaration\'s range.\n\n'
        '    >>> eligible("report")\n    [(1, 0), (1, 1)]\n    """\n'
        "    lo, hi = DECLARATION[package]\n"
        "    return [s for s in REGISTRY[package] if lo <= s < hi]\n"),
    "lock": (
        "(chosen)",
        '"""Fixes resolution\'s result.\n\n'
        "    >>> lock({\"report\": (1, 1)})\n    {'report': (1, 1)}\n    \"\"\"\n"
        "    return dict(chosen)\n"),
}
HEADER = ('REGISTRY = {"report": [(0, 9), (1, 0), (1, 1)]}\n'
          'DECLARATION = {"report": ((1, 0), (2, 0))}\n\n\n')
HELPER = 'def _normalize(s):\n    """Internal use."""\n    return tuple(s)\n\n\n'
NEW_SIGNATURE = {"discount": "(amount, member, coupon, ceiling)",
                  "lock": "(chosen, verify)"}


def build(names, helper=False, signature_changed=False):
    p = HEADER + (HELPER if helper else "")
    for name in names:
        sig, body = BODY[name]
        if signature_changed and name in NEW_SIGNATURE:
            sig = NEW_SIGNATURE[name]
        p += f"def {name}{sig}:\n    {body}\n\n"
    return p


VERSIONS = {
    "canonical": build(["discount", "eligible", "lock"]),
    "order changed": build(["lock", "eligible", "discount"]),
    "helper added": build(["discount", "eligible", "lock"], helper=True),
    "signature changed": build(["discount", "eligible", "lock"],
                                signature_changed=True),
}
HANDWRITTEN = {"discount": "discount(amount, member)",
               "eligible": "eligible(package, declaration)",
               "lock": "lock(chosen)"}


def generate(source):
    """Documentation generator model: public names' signature and summary are read from source."""
    record = {}
    for d in ast.parse(source).body:
        if isinstance(d, ast.FunctionDef) and not d.name.startswith("_"):
            sig = ", ".join(a.arg for a in d.args.args)
            first = (ast.get_docstring(d) or "").split("\n")[0]
            record[d.name] = f"{d.name}({sig}) — {first}"
    return "\n".join(record[a] for a in sorted(record))


def signature(source, name):
    for d in ast.parse(source).body:
        if isinstance(d, ast.FunctionDef) and d.name == name:
            return f"{name}({', '.join(a.arg for a in d.args.args)})"
    return ""


def doc_signature(doc, name):
    for s in doc.splitlines():
        if s.startswith(name + "("):
            return s.split(" — ")[0]
    return ""


def digest(text):
    return hashlib.sha256(text.encode()).hexdigest()[:8]


print(f"{'source version':<24s} {'public names':>12s} {'doc digest':>12s} "
      f"{'stale in generated':>19s} {'stale in handwritten':>21s}")
docs = []
for name, k in VERSIONS.items():
    b = generate(k)
    docs.append(digest(b))
    stale_gen = sum(1 for a in HANDWRITTEN if doc_signature(b, a) != signature(k, a))
    stale_hw = sum(1 for a in HANDWRITTEN if HANDWRITTEN[a] != signature(k, a))
    print(f"{name:<24s} {len(b.splitlines()):12d} {digest(b):>12s} "
          f"{stale_gen:>16d}/{len(HANDWRITTEN)} {stale_hw:>18d}/{len(HANDWRITTEN)}")
print(f"four source versions, {len(set(docs))} distinct generated docs; "
      f"handwritten doc is the same across all four")

print()
module = types.ModuleType("metrics")
exec(compile(VERSIONS["canonical"], "<metrics>", "exec"), module.__dict__)
tried, failed = 0, []
for t in doctest.DocTestFinder().find(module):
    runner = doctest.DocTestRunner(verbose=False)
    runner.run(t, out=lambda s: None)
    tried += runner.tries
    if runner.failures:
        failed.append(t.name.split(".")[-1])
print(f"doc test tried {tried} examples, {len(failed)} failed: "
      f"{', '.join(failed)}")
print("checks the documentation generator does for the claim: 0")

source version           public names   doc digest  stale in generated  stale in handwritten
canonical                           3     644f3575                0/3                  2/3
order changed                       3     644f3575                0/3                  2/3
helper added                        3     644f3575                0/3                  2/3
signature changed                   3     ec5c99c8                0/3                  3/3
four source versions, 2 distinct generated docs; handwritten doc is the same across all four

doc test tried 3 examples, 1 failed: discount
checks the documentation generator does for the claim: 0

What Unifies: The Signature

The table’s first three rows give the same document digest: 644f3575. Three source versions are distinct texts — one writes function definitions in a different order, one carries a private helper function — but the document they produce is single. Because the generator sorts its output by name, definition order drops out; because it skips a name starting with an underscore, the private helper drops out.

The fourth row gives a separate digest: ec5c99c8. Two signatures changed, the document changed. This is not a flaw, it is generation’s definition: documentation tracks the source’s public interface, and only that.

The stale-signature columns give the distinction as a number. In the generated document, stale signatures are 0/3 across all four versions. In the handwritten document, it is 2/3 in the first three versions, 3/3 in the fourth — even though the handwritten document never changed at all. Staleness grows on its own as the source moves forward, and nothing announces it.

The difference between them is not a difference of effort, it is a difference of binding. In the generated document there is no copy between the signature and the source; the signature is reread on every generation. In the handwritten document there is a copy, and the copy staying equal to the original depends on a person remembering. This is exactly the outcome generation unifies: a single document from the same source, and zero stale signatures in that document.

What Does Not Unify: The Claim

The bottom rows give the measurement’s second half and show where unification stops.

The discount function’s summary text states that the ceiling is 25 percent; the body applies 20. The generator carries this line into the document as is. The number of checks it performs is 0, because the generator is a copier: whatever the source says, it writes that into the document. If the source itself says something wrong, generation spreads that wrongness further.

The doc test, by contrast, tries 3 examples and 1 fails: discount. It fails not because the example is wrong, but because the example is runnable. Had the same claim been written as prose, nothing would have failed.

From here comes the lesson’s result. Generation unifies two outcomes: that the document shows the same interface as the source, and that the same document comes out of the same source. It does not unify one outcome: that the claim in the document matches the behavior. That match is tested only when the claim is written in a runnable form, and when it is tested, what checks it is not the documentation generator, it is the test.

This is the same shape as the course’s coverage measurement. Coverage counts where the code ran, not what the test looked at. Generation determines what the document shows, not whether it is correct. Every tool looks at its own input; the only thing that has an oracle is the expectation.

Summary

• A documentation generator derives its output entirely from the source; definition order and private names do not enter the result.
• Four source versions give 2 distinct generated documents; the first three merge into a single digest, the fourth separates because the signature changed.
• Stale signatures in the generated document are 0/3 across all four versions; in the handwritten document they are 2/3 and 3/3, and the document never changed at all.
• Generation does not unify the claim: the 25-percent claim in the discount function’s summary is carried into the document as is, and the checks the generator performs are 0.
• When the same claim is written as a runnable example, 1 of 3 examples fails; what tests the claim is not the generator, it is the test.

Course Wrap-Up

This course asked a single question across fourteen lessons: how many distinct outcomes did the same input give? The question never turned into “does it work” in any lesson, because how many separate ways a thing that works actually works is a separate question, and reproducibility is measured with that question.

Lesson	Measured	Distinct outcome count
Virtual Environments	six install orders, shared and isolated site	5 in shared, 1 in isolated
Dependency Declaration	same declaration, two strategies and two registry views	2, 1 with lock; 3 across four installs, 1 with lock
Package Managers	four resolution strategies, loose and strict constraint	4 sets (3 consistent); 2 sets under strict (0 consistent); 1 with lock
Version Management	same declaration across three fictional interpreters	2; 6 accepted, 3 rejected across nine installs
Test Frameworks	two forms of writing, three discovery rules	1 in form; 3 in tests found, 1 in report
Fixtures and Parametrization	same five-test team across three orders	2, 1 with fixture
Mock Objects	same test in three modes, mock and real source	3 with real, 1 with mock; 2 with a drifted mock
Doctests	twelve claims growing stale with a form change	stale turning audible 4 and 6: 2
Coverage Measurement	coverage of a three-test set and the call deviating from the oracle	line and branch 1.0000, passed 3/3; deviating call 1; with a fourth test, coverage 1, passed ratio 2
Formatters and Linters	the same function’s six separate forms	text 6 → 1; finding set 5 → 1
Multi-Environment Testing	same team in a three-axis matrix, 18 runs	4; 3 with lock, 2 with fixture
Packaging	nine builds from the same source, three inclusion rules	5; 1 with explicit list; 3 and 1 at install
Publishing	two installs with the same lock file, two registry policies	2, 1 in the immutable registry
Documentation	documentation generated from four source versions	2; stale signature 0 against 2 and 3

The table’s reading gathers into one sentence: in nearly every row, the first number is greater than one, the second is one. The difference between them is not a tool difference, it is a decision difference. The lock file did not remove resolution; it made the decision once and put it in writing. Isolation did not make install easier; it stopped install order from changing the result. The fixture did not make testing faster; it stopped order from deciding the outcome. The explicit list did not choose files; it stopped the choice from depending on the directory’s state. In every case, what unified was not the work itself, it was the source of the decision.

A second pattern sits in the table too: a unified number does not give correctness. Coverage did not see the flawed expectation even where it dropped to one; the formatter dropped six forms to one but none of the remaining six findings reported the flaw; documentation generation unified the document but carried the wrong claim in it. Unification is the condition for comparability, not for correctness. The only thing that gives correctness is the oracle, and the oracle always comes from outside the measurement.

Course	Measurement axis
Python Fundamentals	the protocol syntax calls
Data Structures and Functional Tools	the object created, held, and shared
Object-Oriented Python and Types	who answered the call, who checked the claim
Concurrency and Performance	overlapping step
Python Projects: Packaging and Testing	how many distinct outcomes did the same input give

Five axes form a sequence. The first course looked beneath syntax and counted which special method each form calls. The second measured how an object is created, where it is held, and who it is shared with. The third separated which class answers a call and when a given claim gets checked. The fourth counted how many of a job’s steps overlap and built the case for counting steps instead of time. The fifth put all of these into a project and asked how many distinct outcomes the same input gives.

Together the five build a language’s own mechanism: what the language calls, what it holds, what it checks, how it splits work, and how it becomes repeatable as a project. This mechanism is specific to the language and does not carry over to another language as is.

What carries over is the questions. The language curricula starting with M08 ask the same five questions of another language, and the answers come out different: an answer given by a special method in one language is given by an interface in another; a claim checked at run time in one is checked at compile time in another; overlap bounded by an interpreter lock in one is met by another mechanism in another. A language’s own mechanism has been built; next is how another language answers the same questions.