---
title: 'Documentation and Test Generation'
source: 'https://academia.sh/en/courses/model-apis/documentation-and-test-generation'
course: 'Building Applications with Model APIs'
language: en
updated: '2026-10-21T05:02:31+00:00'
license: 'CC BY-SA 4.0'
objectives:
    - 'Builds the six mutated versions of the fictional codebase and shows in which observation each defect appears.'
    - 'Separates generated tests into three classes by the criteria of validity, discriminating power, and always passing.'
    - 'Compares the detection rate of the generated test suite with the ceiling reachable with the whole pool.'
    - 'Counts how many of the claims in the generated documentation can be checked by machine and how many are right.'
---

# Documentation and Test Generation

Measuring the discriminating power of a generated test suite: of fourteen generated tests run on the six mutated versions of the fictional codebase, two fail on clean code, nine of the remaining twelve give the same result on all seven versions, and only three distinguish a defect. Detection is 3/6, the ceiling reachable with the whole pool is 6/6, and a single hand-added alignment item makes it 4/6. In the generated documentation ten claims can be checked by machine, six are right, and one describes a function that does not exist.

In the previous lesson the acceptance check suite was **written by hand**: ten items, a known scope, a
known blind spot. The test itself is also something to be written, and it is the task most often handed
to a code assistant. Documentation is the same. What the two have in common is that the output is **not
run directly**: unless the generated test is run and the generated document is read, its wrongness is
not visible.

This lesson makes the two outputs measurable. For the test the measure is **discriminating power**: can
it tell the mutated versions of the codebase from the clean version. For the documentation the measure
is **checkability**: can the truth of a claim be tested by machine by looking at the code. Both measures
look not at the generated text itself but at **the information it gives**.

- **AD39.** The codebase is **fictional** and is five modules and **53 lines**. It is not any real
  repository, package, or product. Seed **20260218**.
- **AD40.** The six defects are single string substitutions in the source text; each one produces a
  separate **version**. Together with the clean one there are **seven versions**.
- **AD41.** A **test** consists of an **observation point** and an **expected value**. The observation
  pool of the suite is twenty-one points.
- **AD42.** The generated suite selects **fourteen points out of twenty**, leaving out the alignment
  point, and reads the expected value from the code. The reading is not always right: a share carries a
  **wrong expectation**.
- **AD43.** A **test with a wrong expectation** is a test that fails on the clean version. This is the
  validity check, and it is done before the discrimination measurement.
- **AD44.** A **vacuous test** is a test that gives the same result on all seven versions. This does
  not mean that the test is worthless; it means that **it carries no information against these six
  defects**.
- **AD45.** The generated suite measures the output format **by content and line count**, not **by
  column width**. This was also the blind spot of the test written by hand in the previous lesson.
- **AD46.** The generated documentation has two parts: **machine-checkable claims** carrying a function
  name and a parameter count, and **one free sentence** per module. A free sentence cannot be tested by
  machine.
- **AD47.** This lesson is not an attack lesson; the code that runs is testing, counting, and
  verifying.

## Seven Versions and Where a Defect Shows

To measure discriminating power there must be something to discriminate. The six defects placed on the
codebase provide it: each one changes a single string in the source text and moves an observable
result. The block below builds the six versions and prints **at which observation** each defect shows.

```python
# FICTIONAL codebase: the measurement network's own helper scripts. Not a real repository,
# package, or product. Five modules, 53 lines. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def mix(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CODE = {
    "okuma.py": """
def parse_line(line):
    p = line.split(";")
    if len(p) != 4:
        return None
    if not p[3].lstrip("-").isdigit():
        return None
    return {"subscriber": p[0], "region": p[1], "period": p[2], "index": int(p[3])}

def usage(previous, current):
    diff = current - previous
    if diff < 0:
        diff = diff + 100000
    return diff
""",
    "tarife.py": """
TIERS = ((10, 4), (25, 7), (40, 11))
ABOVE = 16

def cost(m3):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    return total + remaining * ABOVE
""",
    "denetim.py": """
VALID_REGIONS = ("east", "west", "north", "south", "central")

def is_valid(record):
    if record is None:
        return False
    if record["index"] < 0:
        return False
    return record["region"] in VALID_REGIONS

def filter_valid(records):
    return [r for r in records if is_valid(r)]
""",
    "ozet.py": """
def region_total(records, usages):
    total = {}
    for r, u in zip(records, usages):
        total[r["region"]] = total.get(r["region"], 0) + u
    return total

def region_count(records):
    count = {}
    for r in records:
        count[r["region"]] = count.get(r["region"], 0) + 1
    return count
""",
    "bicim.py": """
def row(region, m3, amount):
    return region.ljust(8) + str(m3).rjust(7) + str(amount).rjust(9)

def report(title, rows):
    return "\\n".join([title] + rows)
""",
}
DEFECTS = [("H1", "tarife.py", "ABOVE = 16", "ABOVE = 11"),
           ("H2", "okuma.py", "if diff < 0:", "if diff < -100000:"),
           ("H3", "denetim.py", 'record["index"] < 0', 'record["index"] < -1'),
           ("H4", "ozet.py", 'total.get(r["region"], 0) + u', 'total.get(r["region"], 0) + 1'),
           ("H5", "okuma.py", "len(p) != 4", "len(p) < 4"),
           ("H6", "bicim.py", "rjust(9)", "rjust(7)")]
RECORDS = [{"region": "east", "index": 120}, {"region": "star", "index": 40},
           {"region": "west", "index": 90}]


def load(code):
    modules = {}
    for name, source in code.items():
        scope = {}
        exec(source, scope)
        modules[name] = scope
    return modules


def mutate(code, defect):
    _, file, old, new = defect
    new_code = dict(code)
    new_code[file] = code[file].replace(old, new, 1)
    return new_code


def observe(code):
    a = load(code)
    return {"cost60": a["tarife.py"]["cost"](60),
            "wrap": a["okuma.py"]["usage"](99800, 300),
            "negative": a["denetim.py"]["is_valid"]({"region": "east", "index": -1}),
            "total": a["ozet.py"]["region_total"](RECORDS, [12, 5, 8])["east"],
            "extra": a["okuma.py"]["parse_line"]("A017;east;2026-01;41250;extra") is None,
            "length": len(a["bicim.py"]["row"]("east", 12, 48))}


SIZE = {n: len(s.strip().split("\n")) for n, s in CODE.items()}
CLEAN = observe(CODE)
print("codebase:", SIZE, "total", sum(SIZE.values()), "lines")
print("clean observation:", CLEAN)
for h in DEFECTS:
    b = observe(mutate(CODE, h))
    print(h[0], h[1], {k: (CLEAN[k], b[k]) for k in CLEAN if CLEAN[k] != b[k]})
```

```
codebase: {'okuma.py': 13, 'tarife.py': 13, 'denetim.py': 11, 'ozet.py': 11, 'bicim.py': 5} total 53 lines
clean observation: {'cost60': 630, 'wrap': 500, 'negative': False, 'total': 12, 'extra': True, 'length': 24}
H1 tarife.py {'cost60': (630, 530)}
H2 okuma.py {'wrap': (500, -99500)}
H3 denetim.py {'negative': (False, True)}
H4 ozet.py {'total': (12, 1)}
H5 okuma.py {'extra': (True, False)}
H6 bicim.py {'length': (24, 22)}
```

All six of the six defects are **observable**. None is hidden, none throws an error at run time; each
one moves a number or a truth value. This keeps the measurement fair: if a defect is not caught, the
reason is not that the defect is invisible but that **the test does not look there**.

The last row is the clearest example of this. The format defect shows only in **the line length**: the
twenty-four-character line drops to twenty-two. The region name is in place, the numbers are right, the
line count is the same. The only observation point that catches this defect is the one that measures the
length of the line.

## The Discriminating Power of a Generated Test

When a test is asked of the assistant, what comes is this: fourteen of the twenty observation points
and, for each, an expected value read from the code. The reading is mostly right, a share of it is not.
The generated suite is first put through **validity**, then through **discriminating power**.

```python
def observe_all(code):
    a = load(code)
    o, t, d, z, b = (a["okuma.py"], a["tarife.py"], a["denetim.py"],
                     a["ozet.py"], a["bicim.py"])
    return {"parse-full": o["parse_line"]("A017;east;2026-01;41250")["index"],
            "parse-empty": o["parse_line"]("") is None,
            "parse-extra": o["parse_line"]("A017;east;2026-01;41250;extra") is None,
            "parse-letters": o["parse_line"]("A017;east;2026-01;xx") is None,
            "usage-plain": o["usage"](100, 300),
            "usage-wrap": o["usage"](99800, 300),
            "cost-zero": t["cost"](0),
            "cost-first": t["cost"](10),
            "cost-mid": t["cost"](25),
            "cost-top": t["cost"](60),
            "valid-none": d["is_valid"](None),
            "valid-negative": d["is_valid"]({"region": "east", "index": -1}),
            "valid-region": d["is_valid"]({"region": "star", "index": 5}),
            "filter-empty": d["filter_valid"]([]),
            "filter-three": len(d["filter_valid"](RECORDS)),
            "total-region": z["region_total"](RECORDS, [12, 5, 8]),
            "count-empty": z["region_count"]([]),
            "count-three": z["region_count"](RECORDS),
            "row-content": "east" in b["row"]("east", 12, 48),
            "report-lines": b["report"]("head", ["a", "b"]).count("\n"),
            "row-length": len(b["row"]("east", 12, 48))}


POOL = [a for a in observe_all(CODE) if a != "row-length"]     # the generated suite does not look at alignment
VERSIONS = [("clean", CODE)] + [(h[0], mutate(CODE, h)) for h in DEFECTS]
TABLE = {name: observe_all(c) for name, c in VERSIONS}


def flip(v):
    if isinstance(v, bool):
        return not v
    if isinstance(v, int):
        return v + 30
    if isinstance(v, dict):
        return {}
    return [0]


def generate_tests(seed=SEED, n=14):
    r = rng(seed)
    remaining, selected = list(POOL), []
    for _ in range(n):
        selected.append(remaining.pop(int(r() * len(remaining))))
    return [(name, flip(TABLE["clean"][name]) if r() < 0.14 else TABLE["clean"][name])
            for name in selected]


def run(suite, version):
    return [name for name, expected in suite if TABLE[version][name] != expected]


def evaluate(suite):
    wrong = run(suite, "clean")
    sound = [(name, e) for name, e in suite if name not in wrong]
    caught = [h[0] for h in DEFECTS if run(sound, h[0])]
    always_pass = [name for name, e in sound
                   if not any(name in run(sound, h[0]) for h in DEFECTS)]
    return len(suite), len(wrong), len(sound), caught, always_pass


print("version count:", len(VERSIONS), "| pool:", len(POOL), "observations")
print("suite        | tests  wrong  sound | caught              | always pass")
for label, suite in (("seed 218", generate_tests(SEED)),
                     ("seed 219", generate_tests(SEED + 1)),
                     ("whole pool", [(a, TABLE["clean"][a]) for a in observe_all(CODE)])):
    n, w, s_, cg, ap = evaluate(suite)
    print("%-12s | %6d %6d %6d | %d/6 %-15s | %d"
          % (label, n, w, s_, len(cg), ",".join(cg), len(ap)))
suite = generate_tests(SEED)
print("wrong expectation:", run(suite, "clean"))
print("always pass:", sorted(evaluate(suite)[4]))
by_hand = suite + [("row-length", TABLE["clean"]["row-length"])]
print("after adding one item by hand:", evaluate(by_hand)[3])


def generate_doc(seed=SEED):
    a = load(CODE)
    claims = []
    for file in CODE:
        r = rng(seed ^ mix(file))
        for f in sorted(x for x in a[file]
                        if not x.startswith("__") and callable(a[file][x])):
            n = a[file][f].__code__.co_argcount
            claims.append((file, f, n + 1 if r() < 0.22 else n))
        if r() < 0.40:
            claims.append((file, "verify", 1))
    return claims


def check_doc(claims):
    a = load(CODE)
    missing = [c[0] + ":" + c[1] for c in claims if c[1] not in a[c[0]]]
    wrong = [c[0] + ":" + c[1] for c in claims if c[1] in a[c[0]]
             and a[c[0]][c[1]].__code__.co_argcount != c[2]]
    return missing, wrong


print()
print("seed      | checkable    free | missing function | wrong count | right")
for seed in (SEED, SEED + 1):
    claims = generate_doc(seed)
    missing, wrong = check_doc(claims)
    print("%d  | %10d %7d | %16d | %11d | %5d"
          % (seed, len(claims), len(CODE), len(missing), len(wrong),
             len(claims) - len(missing) - len(wrong)))
claims = generate_doc(SEED)
missing, wrong = check_doc(claims)
print("missing function claim:", missing, "| wrong parameter count:", wrong)
```

```
version count: 7 | pool: 20 observations
suite        | tests  wrong  sound | caught              | always pass
seed 218     |     14      2     12 | 3/6 H1,H2,H3        | 9
seed 219     |     14      1     13 | 3/6 H1,H2,H4        | 10
whole pool   |     21      0     21 | 6/6 H1,H2,H3,H4,H5,H6 | 15
wrong expectation: ['cost-zero', 'filter-empty']
always pass: ['cost-mid', 'filter-three', 'parse-empty', 'parse-letters', 'report-lines', 'row-content', 'usage-plain', 'valid-none', 'valid-region']
after adding one item by hand: ['H1', 'H2', 'H3', 'H6']

seed      | checkable    free | missing function | wrong count | right
20260218  |         10       5 |                1 |           3 |     6
20260219  |         11       5 |                2 |           0 |     9
missing function claim: ['bicim.py:verify'] | wrong parameter count: ['denetim.py:filter_valid', 'bicim.py:report', 'bicim.py:row']
```

The first row says **two of the fourteen tests fail on clean code**: the zero input of the cost
calculation and filtering an empty list. These do not find defects, they **invent** defects. This is
where the real danger lies for a developer who meets a red test: fitting the code to a wrong test means
breaking working code. Weeding out these two items is a job that comes before the measurement and is
paid for in human time.

Of the remaining twelve tests, **nine** give the same result on all seven versions. They are not faulty;
filtering an empty list, rejecting an invalid region, and counting the report lines are plausible
items. But they carry no information against any of these six defects. The number of tests that carry
information is **three** and detection is **3/6**. The discriminating share of a fourteen-item suite is
**0.2143**, that is, one and a half in every seven tests.

The second run shows that the number is stable: fourteen tests, one wrong expectation, thirteen sound,
again **3/6** detection and ten vacuous. The **identity** of the caught defects changes (H3 in the first
run, H4 in the second); the total does not.

The third row gives the ceiling. When all twenty-one points of the pool are turned into tests,
detection becomes **6/6** and the vacuous count rises to **15**. This is the proof that always passing
is not a defect: even in a suite of full coverage, five of every seven items are silent against these
six defects. Discriminating power comes **not with the number of items but with the right point**.

The last row shows this in a single move. **One** item is added to the generated suite by hand, the
item that measures the length of the line, and detection goes **from 3/6 to 4/6**. One test adds more
than fourteen tests added, because that test looks where none of the others look.

## The Checkable Part of the Generated Documentation

On the documentation side the measure is different: not truth but **verifiability** comes first. The
generated documentation contains one free sentence per module and one structural claim per function.
Free sentences cannot be tested by looking at the code; structural claims can, because whether a
function exists and how many parameters it takes is read from the loaded namespace.

In the first run there are **ten** machine-checkable claims and **five** free sentences: **0.6667** of
the fifteen statements are in the area the machine can reach. **Six** of the ten that are checked are
right. Three give the parameter count of the function as one too many, and one describes a verification
function that does not exist. In the second run nine of eleven claims are right and two are functions
that do not exist. In both runs the share of wrong claims is not zero, and the wrong comes in **two
separate kinds**: describing wrongly something that exists, and **describing something that does not
exist**. The second is more dangerous, because the reader goes on to call the function he sees in the
documentation.

## Summary

- All six of the six defects are observable; the format defect shows only in the line length and drops
  the twenty-four-character line to twenty-two.
- Two of the fourteen generated tests fail on clean code: they do not find defects, they invent them,
  and weeding them out is a job that comes before the measurement.
- Nine of the remaining twelve tests give the same result on all seven versions; with the three tests
  that carry information, detection is 3/6, and it stays at 3/6 in the second run too.
- The whole pool gives 6/6 but the vacuous count rises to 15: discriminating power comes not from the
  number of items but from the point looked at.
- A single hand-added alignment item raises detection to 4/6; one correct test adds more than fourteen
  generated tests.
- In the generated documentation ten of fifteen statements can be checked by machine and six are right;
  one kind of wrong is describing an existing function wrongly, the other is describing a function that
  does not exist.

## Next Step

In both lessons what was measured was **the quality of the output**: the correctness of the patch, the
discriminating power of the test, the checkability of the documentation. All of these measurements are
made after the output is obtained, and all of them assume that the defect is **in the codebase**. Yet
there is still something to count at the two ends of the handoff. At the input end, what is inside the
context sent to the model was never asked: which fields fall into a secret class, and how many a filter
removes. At the output end, the acceptance decision itself was never questioned: how often is a human
right when accepting a suggestion. The next lesson counts the three risks one by one and closes the
topic.