Doctests

The previous lesson measured that a mock object dropped distinct outcome count from three to one, but said nothing about agreement with the real source. The claims there — the expected version set, the expected call count — were written in a test file and ran on every run. They could be wrong, but they could not stay silent.

A project’s correctness claims do not sit only in test files. A function’s docstring saying “this call returns that value” is a claim too. The difference: no run tests that sentence. When the code changes, the sentence stays where it is, nothing breaks, and the claim falsifies silently. This lesson’s question: after the same change, how many stale claims does the same document turn audible?

Silent Staleness and Audible Staleness

When the surface a documentation claim is tied to changes, the claim goes stale. The Technical Writing and Documentation course of the Software Development Practice curriculum measured this over fifty-six claims and split them into two classes: a claim embedded in a runnable example makes noise when it goes stale — the example fails to run; a claim that is not embedded produces nothing when it goes stale, and this is silent staleness. In that measurement, seventeen of the fifty-six claims were embedded, and that number was a given, not a choice.

In this lesson, embedding is a choice. The measurement does not repeat those numbers; it asks a different question: if the same claim can be written as prose or as a runnable example, what does changing the writing do to the count of stale claims that go audible? Embedding was that measurement’s input; here, it is this measurement’s variable.

The mechanism that makes this measurable is called a doctest, and it is provided by the standard library’s doctest module. The mechanism is simple: lines in the document text starting with the interactive shell prompt count as expressions, and the lines beneath them count as expected output. The module runs the expressions, compares the output against expected, and counts mismatches as failing examples. The document, that is, becomes a test suite itself.

It needs stating from the start that a doctest does not substitute for a unit test suite. What it tests is the example written in the document; an edge case, an error path, or a failure scenario not written there never enters a doctest at all. Multiplying examples to make them comprehensive also makes the document unreadable — a document is written to be read, not to test. A doctest’s job is not to grow coverage, it is to keep the claims already written standing.

Two Writings of a Claim

The same fact can be written two ways. “The version text is made of numbers separated by periods” is a prose claim; it gets read, understood, and tested by nothing. The same fact, written as a call and its expected output, becomes testable — and its readability usually improves too, because an example puts a concrete value in place of an abstract description.

The choice between the two looks like a style preference, yet it carries a measurable difference. A prose claim going stale only gets noticed if a human reads it; an example going stale turns the run red. The measurement counts exactly this difference.

A Doctest Bounds What Can Be Written

A doctest’s comparison rule is strict: expected output and actual output have to be equal as text. This rule also decides which example can go into the document — an example whose output is not the same on every run cannot go into the document.

In Python, this constraint has a few well-known sources. An object’s default representation carries a number tied to its identity that changes from run to run; a floating-point number’s representation is sensitive to operation order; the print order of a set or dictionary keys is a separate topic worth its own attention. Document examples, for this reason, either get written to return a deterministic value, or get made deterministic by sorting, formatting, or reducing the output to a comparison.

The measurement’s examples show this directly. The diff function’s result is a list sorted with sorted; without the sort, the example could give a different order each run. The function producing the lock text also writes packages in name order. Neither decision was made to make the document testable, both were made to make the result readable — but the doctest rests on top of these decisions.

The consequence looks backward at first glance: adding a doctest affects the code, not the document. If a function’s output is not deterministic, that function’s documentation cannot become testable; if that is wanted, what has to change is the function itself.

The Claim That Cannot Convert

Not every prose claim can convert to an example, and this is the doctest’s limit. A doctest compares an expression’s value; it cannot compare a sentence that has no value.

Five of the measurement’s six prose claims depend on a call’s result and can convert to an example. The sixth is a rationale: “no prefix is written, because the text sorts like a number.” This sentence’s testable half — no prefix being written — is already tested in another example; its untestable half is what the sentence actually carries, the reason for the decision. A reason has no returned value.

In practice: a doctest can protect sentences that state what is the case, it cannot protect sentences that state why it is the case. The second class is the most expensive one to go stale, because a wrong rationale leaves the reader alone with a wrong decision. When an example fails, the reader stops trusting the document; when a rationale goes stale, the reader keeps trusting it, and gets steered the wrong way.

The measurement’s assumptions:

The Measurement’s Assumptions

• TE29 — The four documented functions are this lesson’s own: version text, lock text, and the function giving the diff between two dictionaries. The shared setup’s REGISTRY, MANIFEST, and discount definitions are not touched.
• TE30 — Total claim count is 12. Six are runnable examples in the document text, six are prose sentences. The total is the same in both writings; only the distribution changes.
• TE31 — The change is a format change: a prefix gets added to the version text, and the lock text’s field separator changes. The diff function stays untouched.
• TE32 — The oracle is the setup itself: we know which claim goes stale because we wrote both implementations. Prose claims’ staleness gets flagged from this knowledge.
• TE33 — Runnable examples’ staleness is not flagged, it is measured: the doctest gets run, and the count of failing examples gets read.
• TE34 — In the second writing, the convertible stale prose claims get converted to examples. The non-stale ones stay prose; the measurement counts only staleness’s noise.
• TE35 — The one claim that cannot convert to an example is the rationale, and it stays prose in both writings.
• TE36 — The doctest takes the document text as a string separate from the source and runs it with two separate implementation sets; the run’s output is swallowed, only the numbers get read.
• TE37 — An example fails if its expected output is not equal to the actual output; there is no partial match.
• TE38 — Before the change, both writings give 0 failures; the document is consistent with the implementation it was written against.
• TE39 — Duration is never measured, no file is left in the repository. What gets counted is claims, stale claims, failing examples, and claims that stay silent.
• TE40 — Distinct outcome count is the number of mutually different audible-stale-claim counts the two document writings give.

The Measurement

"""Doctests: same document, two writing styles, same staleness, two outcomes.

Part 1 - the examples in the document get run; prose claims are flagged from
the oracle.
Part 2 - two prose claims get turned into examples and the measurement is
repeated.
"""
import doctest

CHOICE = {"metrics": (1, 2), "common": (3, 2), "report": (1, 1)}
PREVIOUS = {"metrics": (1, 1), "common": (3, 2), "report": (1, 1)}


def old_version(v):
    return ".".join(map(str, v))


def new_version(v):
    return "v" + ".".join(map(str, v))


def old_lock(choice):
    return " ".join(f"{p}={old_version(choice[p])}" for p in sorted(choice))


def new_lock(choice):
    return ", ".join(f"{p}={new_version(choice[p])}" for p in sorted(choice))


def diff(a, b):
    return sorted(p for p in a if a[p] != b[p])


OLD = {"version": old_version, "lock": old_lock, "diff": diff,
       "CHOICE": CHOICE, "PREVIOUS": PREVIOUS}
NEW = {"version": new_version, "lock": new_lock, "diff": diff,
       "CHOICE": CHOICE, "PREVIOUS": PREVIOUS}

EXAMPLES = """
>>> version((1, 2))
'1.2'
>>> version((3, 0))
'3.0'
>>> lock(CHOICE)
'common=3.2 metrics=1.2 report=1.1'
>>> lock(PREVIOUS)
'common=3.2 metrics=1.1 report=1.1'
>>> diff(CHOICE, PREVIOUS)
['metrics']
>>> diff(CHOICE, CHOICE)
[]
"""

EXTRA_EXAMPLES = """
>>> set(version((1, 2))) <= set('0123456789.')
True
>>> lock(CHOICE).count(',')
0
"""

# Prose claims: text, stale after the change?, convertible to an example?
PROSE = (("version text carries only digits and periods", True, True),
         ("the lower bound is inclusive, the upper bound is exclusive", False, True),
         ("packages are written in name order", False, True),
         ("fields in the lock text are separated without commas", True, True),
         ("the diff list is given in name order", False, True),
         ("no prefix is written, because the text sorts like a number", True, False))


def run(text, namespace):
    test = doctest.DocTestParser().get_doctest(text, dict(namespace), "doc", None, 0)
    result = doctest.DocTestRunner(verbose=False).run(test, out=lambda s: None)
    return result.attempted, result.failed


STALE_PROSE = sum(1 for _, s, _ in PROSE if s)
CONVERTED = sum(1 for _, s, c in PROSE if s and c)
STYLES = (("prose-heavy", EXAMPLES, 0),
          ("two claims as examples", EXAMPLES + EXTRA_EXAMPLES, CONVERTED))

print(f"{'doc style':<22s} {'examples':>8s} {'prose':>7s} {'audible stale':>13s} "
      f"{'silent stale':>13s}")
audible, silents = [], []
for label, text, converted in STYLES:
    attempted, failed = run(text, NEW)
    silent = STALE_PROSE - converted
    audible.append(failed)
    silents.append(silent)
    print(f"{label:<22s} {attempted:8d} {len(PROSE) - converted:7d} {failed:13d} "
          f"{silent:13d}")
print(f"claim total in both styles is {len(PROSE) + run(EXAMPLES, NEW)[0]}, "
      f"stale count {audible[0] + STALE_PROSE}")
print(f"audible stale claims {audible[0]} and {audible[1]}: "
      f"{len(set(audible))} distinct outcomes")
print(f"converting to examples turned {silents[0] - silents[1]} silent staleness "
      f"audible; {silents[1]} could not convert because it is a rationale")
print(f"the same document gives {run(EXAMPLES, OLD)[1]} and "
      f"{run(EXAMPLES + EXTRA_EXAMPLES, OLD)[1]} failures before the change")

doc style              examples   prose audible stale  silent stale
prose-heavy                   6       6             4             3
two claims as examples        8       4             6             1
claim total in both styles is 12, stale count 7
audible stale claims 4 and 6: 2 distinct outcomes
converting to examples turned 2 silent staleness audible; 1 could not convert because it is a rationale
the same document gives 0 and 0 failures before the change

The Same Staleness, Two Distinct Outcomes

The last line gives the measurement’s starting point: before the change, both writings produce 0 failures. The document is consistent with the implementation it was written against, and every claim is correct. Staleness gets born when the format changes. Without this line, the rest of the measurement would be unreadable: to say the failing examples come from staleness, it had to be shown that none of them failed before the change.

After the change, 7 of twelve claims go stale: four runnable examples, three prose. Stale claim count is the same in both writings — how the document is written does not change what falsifies. The only thing it changes is whether the falsification gets heard.

In the prose-heavy writing, the doctest gives 4 failures, and 3 stale claims stay silent. Once the two stale prose claims get converted to examples, the audible count rises to 6, and what stays silent drops to 1. The same staleness, 2 distinct outcomes across the two document writings.

The difference between these two numbers is exactly the writing itself. The code is the same, the change is the same, the set of claims that go stale is the same. Only two sentences moved from prose to example, and staleness’s visibility spread across more than two. This is what a doctest adds to the measurement: writing a claim so that it speaks when it falsifies.

It is worth noting where the audible count comes from. The four runnable examples failed because they went stale, and we did not flag their failure — the doctest ran, the count got read. Prose claims’ staleness, though, could not be measured, it got flagged from the oracle: since we wrote both implementations, we know which sentence is no longer correct. The measurement’s asymmetry is not an accident, it is the topic itself. An audible claim can be counted; a silent claim can only be known — and in a real project, that knowledge is not there either.

Another reading: the difference between the two writings was 2, while stale prose claim count was 3. Converting to examples, that is, did not save all the stale prose claims, it saved two-thirds. This ratio is not the setup’s choice, it is an example of claim classes’ natural distribution: a document always contains sentences with no value.

What Is Left

The one claim staying silent is the rationale, and it staying there is not a gap, it is a limit. A doctest runs an expression and compares its value; a sentence starting with “because” has no expression to run. This claim can only be checked by a human reading it, and where that check belongs is the subject of the Software Development Practice curriculum.

What the measurement says here is that the silent count cannot be dropped to zero. A doctest reduces silent staleness, it does not remove it; if it could, the whole document would become runnable and stop being a document at all. A document’s value, in any case, is not only in its testable sentences; it is in the sentences that carry the reader to a decision.

There is a limit in the reverse direction too. Every claim converted to an example makes the document more fragile. The two new examples in the second writing test the version text’s character set and the comma count in the lock text; these are not the behavior itself, they are details of its format. When the format changes deliberately, these examples fail too, and what they say is not “the code broke,” it is “the document needs updating.” This is exactly a doctest’s benefit: it gives both messages through the same channel, at the same time.

One last thing needs writing: what a number does not mean. 6/12 does not mean half the document is tested; it means half the claims in the document make noise when they falsify. A claim not written in the document never enters this measurement, and a function’s undocumented behavior sits entirely outside this number.

Summary

• A sentence in a docstring is a correctness claim too; its difference is that no run tests it. It falsifies silently when the surface it is tied to changes.
• A doctest turns a document into a test suite by running the calls in the document text and comparing them against expected output; a claim being embedded is not a given, it is a writing choice.
• 7 of 12 claims go stale from a single format change, and this number is independent of writing; writing only decides whether the staleness gets heard.
• The prose-heavy writing turns 4 stale claims audible and leaves 3 silent; once two claims convert to examples, the audible count rises to 6 — 2 distinct outcomes.
• The one claim left silent is a rationale and cannot convert to an example: a doctest compares an expression’s value, and a reason has no value.

Next Step

Every measurement up to here counted what the test sees: which tests got discovered, in what order they ran, which source they looked at, which claim turned audible. What all of them share is that what got counted was the test’s point of view. The next lesson turns the counter to the other side and looks at the code itself: when a three-test suite runs, which lines of the function executed, which branches got taken? This number is misleading to a degree, and the lesson’s job is drawing exactly the boundary of that mistake — even when eight of eight lines execute and three of three tests pass, a call can still be found where the function does not honor its contract.