Documentation and Test Generation

In the previous lesson the acceptance check suite was written by hand: ten items, a known scope, a known blind spot. The test itself is also something to be written, and it is the task most often handed to a code assistant. Documentation is the same. What the two have in common is that the output is not run directly: unless the generated test is run and the generated document is read, its wrongness is not visible.

This lesson makes the two outputs measurable. For the test the measure is discriminating power: can it tell the mutated versions of the codebase from the clean version. For the documentation the measure is checkability: can the truth of a claim be tested by machine by looking at the code. Both measures look not at the generated text itself but at the information it gives.

• AD39. The codebase is fictional and is five modules and 53 lines. It is not any real repository, package, or product. Seed 20260218.
• AD40. The six defects are single string substitutions in the source text; each one produces a separate version. Together with the clean one there are seven versions.
• AD41. A test consists of an observation point and an expected value. The observation pool of the suite is twenty-one points.
• AD42. The generated suite selects fourteen points out of twenty, leaving out the alignment point, and reads the expected value from the code. The reading is not always right: a share carries a wrong expectation.
• AD43. A test with a wrong expectation is a test that fails on the clean version. This is the validity check, and it is done before the discrimination measurement.
• AD44. A vacuous test is a test that gives the same result on all seven versions. This does not mean that the test is worthless; it means that it carries no information against these six defects.
• AD45. The generated suite measures the output format by content and line count, not by column width. This was also the blind spot of the test written by hand in the previous lesson.
• AD46. The generated documentation has two parts: machine-checkable claims carrying a function name and a parameter count, and one free sentence per module. A free sentence cannot be tested by machine.
• AD47. This lesson is not an attack lesson; the code that runs is testing, counting, and verifying.

Seven Versions and Where a Defect Shows

To measure discriminating power there must be something to discriminate. The six defects placed on the codebase provide it: each one changes a single string in the source text and moves an observable result. The block below builds the six versions and prints at which observation each defect shows.

# FICTIONAL codebase: the measurement network's own helper scripts. Not a real repository,
# package, or product. Five modules, 53 lines. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def mix(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CODE = {
    "okuma.py": """
def parse_line(line):
    p = line.split(";")
    if len(p) != 4:
        return None
    if not p[3].lstrip("-").isdigit():
        return None
    return {"subscriber": p[0], "region": p[1], "period": p[2], "index": int(p[3])}

def usage(previous, current):
    diff = current - previous
    if diff < 0:
        diff = diff + 100000
    return diff
""",
    "tarife.py": """
TIERS = ((10, 4), (25, 7), (40, 11))
ABOVE = 16

def cost(m3):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    return total + remaining * ABOVE
""",
    "denetim.py": """
VALID_REGIONS = ("east", "west", "north", "south", "central")

def is_valid(record):
    if record is None:
        return False
    if record["index"] < 0:
        return False
    return record["region"] in VALID_REGIONS

def filter_valid(records):
    return [r for r in records if is_valid(r)]
""",
    "ozet.py": """
def region_total(records, usages):
    total = {}
    for r, u in zip(records, usages):
        total[r["region"]] = total.get(r["region"], 0) + u
    return total

def region_count(records):
    count = {}
    for r in records:
        count[r["region"]] = count.get(r["region"], 0) + 1
    return count
""",
    "bicim.py": """
def row(region, m3, amount):
    return region.ljust(8) + str(m3).rjust(7) + str(amount).rjust(9)

def report(title, rows):
    return "\\n".join([title] + rows)
""",
}
DEFECTS = [("H1", "tarife.py", "ABOVE = 16", "ABOVE = 11"),
           ("H2", "okuma.py", "if diff < 0:", "if diff < -100000:"),
           ("H3", "denetim.py", 'record["index"] < 0', 'record["index"] < -1'),
           ("H4", "ozet.py", 'total.get(r["region"], 0) + u', 'total.get(r["region"], 0) + 1'),
           ("H5", "okuma.py", "len(p) != 4", "len(p) < 4"),
           ("H6", "bicim.py", "rjust(9)", "rjust(7)")]
RECORDS = [{"region": "east", "index": 120}, {"region": "star", "index": 40},
           {"region": "west", "index": 90}]


def load(code):
    modules = {}
    for name, source in code.items():
        scope = {}
        exec(source, scope)
        modules[name] = scope
    return modules


def mutate(code, defect):
    _, file, old, new = defect
    new_code = dict(code)
    new_code[file] = code[file].replace(old, new, 1)
    return new_code


def observe(code):
    a = load(code)
    return {"cost60": a["tarife.py"]["cost"](60),
            "wrap": a["okuma.py"]["usage"](99800, 300),
            "negative": a["denetim.py"]["is_valid"]({"region": "east", "index": -1}),
            "total": a["ozet.py"]["region_total"](RECORDS, [12, 5, 8])["east"],
            "extra": a["okuma.py"]["parse_line"]("A017;east;2026-01;41250;extra") is None,
            "length": len(a["bicim.py"]["row"]("east", 12, 48))}


SIZE = {n: len(s.strip().split("\n")) for n, s in CODE.items()}
CLEAN = observe(CODE)
print("codebase:", SIZE, "total", sum(SIZE.values()), "lines")
print("clean observation:", CLEAN)
for h in DEFECTS:
    b = observe(mutate(CODE, h))
    print(h[0], h[1], {k: (CLEAN[k], b[k]) for k in CLEAN if CLEAN[k] != b[k]})

codebase: {'okuma.py': 13, 'tarife.py': 13, 'denetim.py': 11, 'ozet.py': 11, 'bicim.py': 5} total 53 lines
clean observation: {'cost60': 630, 'wrap': 500, 'negative': False, 'total': 12, 'extra': True, 'length': 24}
H1 tarife.py {'cost60': (630, 530)}
H2 okuma.py {'wrap': (500, -99500)}
H3 denetim.py {'negative': (False, True)}
H4 ozet.py {'total': (12, 1)}
H5 okuma.py {'extra': (True, False)}
H6 bicim.py {'length': (24, 22)}

All six of the six defects are observable. None is hidden, none throws an error at run time; each one moves a number or a truth value. This keeps the measurement fair: if a defect is not caught, the reason is not that the defect is invisible but that the test does not look there.

The last row is the clearest example of this. The format defect shows only in the line length: the twenty-four-character line drops to twenty-two. The region name is in place, the numbers are right, the line count is the same. The only observation point that catches this defect is the one that measures the length of the line.

The Discriminating Power of a Generated Test

When a test is asked of the assistant, what comes is this: fourteen of the twenty observation points and, for each, an expected value read from the code. The reading is mostly right, a share of it is not. The generated suite is first put through validity, then through discriminating power.

def observe_all(code):
    a = load(code)
    o, t, d, z, b = (a["okuma.py"], a["tarife.py"], a["denetim.py"],
                     a["ozet.py"], a["bicim.py"])
    return {"parse-full": o["parse_line"]("A017;east;2026-01;41250")["index"],
            "parse-empty": o["parse_line"]("") is None,
            "parse-extra": o["parse_line"]("A017;east;2026-01;41250;extra") is None,
            "parse-letters": o["parse_line"]("A017;east;2026-01;xx") is None,
            "usage-plain": o["usage"](100, 300),
            "usage-wrap": o["usage"](99800, 300),
            "cost-zero": t["cost"](0),
            "cost-first": t["cost"](10),
            "cost-mid": t["cost"](25),
            "cost-top": t["cost"](60),
            "valid-none": d["is_valid"](None),
            "valid-negative": d["is_valid"]({"region": "east", "index": -1}),
            "valid-region": d["is_valid"]({"region": "star", "index": 5}),
            "filter-empty": d["filter_valid"]([]),
            "filter-three": len(d["filter_valid"](RECORDS)),
            "total-region": z["region_total"](RECORDS, [12, 5, 8]),
            "count-empty": z["region_count"]([]),
            "count-three": z["region_count"](RECORDS),
            "row-content": "east" in b["row"]("east", 12, 48),
            "report-lines": b["report"]("head", ["a", "b"]).count("\n"),
            "row-length": len(b["row"]("east", 12, 48))}


POOL = [a for a in observe_all(CODE) if a != "row-length"]     # the generated suite does not look at alignment
VERSIONS = [("clean", CODE)] + [(h[0], mutate(CODE, h)) for h in DEFECTS]
TABLE = {name: observe_all(c) for name, c in VERSIONS}


def flip(v):
    if isinstance(v, bool):
        return not v
    if isinstance(v, int):
        return v + 30
    if isinstance(v, dict):
        return {}
    return [0]


def generate_tests(seed=SEED, n=14):
    r = rng(seed)
    remaining, selected = list(POOL), []
    for _ in range(n):
        selected.append(remaining.pop(int(r() * len(remaining))))
    return [(name, flip(TABLE["clean"][name]) if r() < 0.14 else TABLE["clean"][name])
            for name in selected]


def run(suite, version):
    return [name for name, expected in suite if TABLE[version][name] != expected]


def evaluate(suite):
    wrong = run(suite, "clean")
    sound = [(name, e) for name, e in suite if name not in wrong]
    caught = [h[0] for h in DEFECTS if run(sound, h[0])]
    always_pass = [name for name, e in sound
                   if not any(name in run(sound, h[0]) for h in DEFECTS)]
    return len(suite), len(wrong), len(sound), caught, always_pass


print("version count:", len(VERSIONS), "| pool:", len(POOL), "observations")
print("suite        | tests  wrong  sound | caught              | always pass")
for label, suite in (("seed 218", generate_tests(SEED)),
                     ("seed 219", generate_tests(SEED + 1)),
                     ("whole pool", [(a, TABLE["clean"][a]) for a in observe_all(CODE)])):
    n, w, s_, cg, ap = evaluate(suite)
    print("%-12s | %6d %6d %6d | %d/6 %-15s | %d"
          % (label, n, w, s_, len(cg), ",".join(cg), len(ap)))
suite = generate_tests(SEED)
print("wrong expectation:", run(suite, "clean"))
print("always pass:", sorted(evaluate(suite)[4]))
by_hand = suite + [("row-length", TABLE["clean"]["row-length"])]
print("after adding one item by hand:", evaluate(by_hand)[3])


def generate_doc(seed=SEED):
    a = load(CODE)
    claims = []
    for file in CODE:
        r = rng(seed ^ mix(file))
        for f in sorted(x for x in a[file]
                        if not x.startswith("__") and callable(a[file][x])):
            n = a[file][f].__code__.co_argcount
            claims.append((file, f, n + 1 if r() < 0.22 else n))
        if r() < 0.40:
            claims.append((file, "verify", 1))
    return claims


def check_doc(claims):
    a = load(CODE)
    missing = [c[0] + ":" + c[1] for c in claims if c[1] not in a[c[0]]]
    wrong = [c[0] + ":" + c[1] for c in claims if c[1] in a[c[0]]
             and a[c[0]][c[1]].__code__.co_argcount != c[2]]
    return missing, wrong


print()
print("seed      | checkable    free | missing function | wrong count | right")
for seed in (SEED, SEED + 1):
    claims = generate_doc(seed)
    missing, wrong = check_doc(claims)
    print("%d  | %10d %7d | %16d | %11d | %5d"
          % (seed, len(claims), len(CODE), len(missing), len(wrong),
             len(claims) - len(missing) - len(wrong)))
claims = generate_doc(SEED)
missing, wrong = check_doc(claims)
print("missing function claim:", missing, "| wrong parameter count:", wrong)

version count: 7 | pool: 20 observations
suite        | tests  wrong  sound | caught              | always pass
seed 218     |     14      2     12 | 3/6 H1,H2,H3        | 9
seed 219     |     14      1     13 | 3/6 H1,H2,H4        | 10
whole pool   |     21      0     21 | 6/6 H1,H2,H3,H4,H5,H6 | 15
wrong expectation: ['cost-zero', 'filter-empty']
always pass: ['cost-mid', 'filter-three', 'parse-empty', 'parse-letters', 'report-lines', 'row-content', 'usage-plain', 'valid-none', 'valid-region']
after adding one item by hand: ['H1', 'H2', 'H3', 'H6']

seed      | checkable    free | missing function | wrong count | right
20260218  |         10       5 |                1 |           3 |     6
20260219  |         11       5 |                2 |           0 |     9
missing function claim: ['bicim.py:verify'] | wrong parameter count: ['denetim.py:filter_valid', 'bicim.py:report', 'bicim.py:row']

The first row says two of the fourteen tests fail on clean code: the zero input of the cost calculation and filtering an empty list. These do not find defects, they invent defects. This is where the real danger lies for a developer who meets a red test: fitting the code to a wrong test means breaking working code. Weeding out these two items is a job that comes before the measurement and is paid for in human time.

Of the remaining twelve tests, nine give the same result on all seven versions. They are not faulty; filtering an empty list, rejecting an invalid region, and counting the report lines are plausible items. But they carry no information against any of these six defects. The number of tests that carry information is three and detection is 3/6. The discriminating share of a fourteen-item suite is 0.2143, that is, one and a half in every seven tests.

The second run shows that the number is stable: fourteen tests, one wrong expectation, thirteen sound, again 3/6 detection and ten vacuous. The identity of the caught defects changes (H3 in the first run, H4 in the second); the total does not.

The third row gives the ceiling. When all twenty-one points of the pool are turned into tests, detection becomes 6/6 and the vacuous count rises to 15. This is the proof that always passing is not a defect: even in a suite of full coverage, five of every seven items are silent against these six defects. Discriminating power comes not with the number of items but with the right point.

The last row shows this in a single move. One item is added to the generated suite by hand, the item that measures the length of the line, and detection goes from 3/6 to 4/6. One test adds more than fourteen tests added, because that test looks where none of the others look.

The Checkable Part of the Generated Documentation

On the documentation side the measure is different: not truth but verifiability comes first. The generated documentation contains one free sentence per module and one structural claim per function. Free sentences cannot be tested by looking at the code; structural claims can, because whether a function exists and how many parameters it takes is read from the loaded namespace.

In the first run there are ten machine-checkable claims and five free sentences: 0.6667 of the fifteen statements are in the area the machine can reach. Six of the ten that are checked are right. Three give the parameter count of the function as one too many, and one describes a verification function that does not exist. In the second run nine of eleven claims are right and two are functions that do not exist. In both runs the share of wrong claims is not zero, and the wrong comes in two separate kinds: describing wrongly something that exists, and describing something that does not exist. The second is more dangerous, because the reader goes on to call the function he sees in the documentation.

Summary

• All six of the six defects are observable; the format defect shows only in the line length and drops the twenty-four-character line to twenty-two.
• Two of the fourteen generated tests fail on clean code: they do not find defects, they invent them, and weeding them out is a job that comes before the measurement.
• Nine of the remaining twelve tests give the same result on all seven versions; with the three tests that carry information, detection is 3/6, and it stays at 3/6 in the second run too.
• The whole pool gives 6/6 but the vacuous count rises to 15: discriminating power comes not from the number of items but from the point looked at.
• A single hand-added alignment item raises detection to 4/6; one correct test adds more than fourteen generated tests.
• In the generated documentation ten of fifteen statements can be checked by machine and six are right; one kind of wrong is describing an existing function wrongly, the other is describing a function that does not exist.

Next Step

In both lessons what was measured was the quality of the output: the correctness of the patch, the discriminating power of the test, the checkability of the documentation. All of these measurements are made after the output is obtained, and all of them assume that the defect is in the codebase. Yet there is still something to count at the two ends of the handoff. At the input end, what is inside the context sent to the model was never asked: which fields fall into a secret class, and how many a filter removes. At the output end, the acceptance decision itself was never questioned: how often is a human right when accepting a suggestion. The next lesson counts the three risks one by one and closes the topic.