Reviewing Generated Code

The previous lesson measured where to put the checkpoint in a multi-step handoff: how many steps apart the stops must be for the error to be caught before it spreads, and what stopping costs. A checkpoint is a place. What to do there was not said. The person who stops has a patch in front of them, and looking at that patch has its own scope, its own cost, and its own blind spot.

This lesson splits that look into three separate checks and counts the defect class each one reaches: the codebase’s own check run, the pattern scan of the generated text, and the existence check of the dependencies taken in. The general procedure of code review was measured in the Software Quality and Testing curriculum, and the procedure of security review in the Cybersecurity and the Application Security and DevSecOps curricula; it is not repeated here. The only thing added is the identity of the producer: the patch under review comes not from a human but from an uncertain component, so the distribution of defects is different, and the boundary of the area the checks cover has to be counted.

• AD31. The codebase is fictional: the measurement network’s own helper scripts. It is not any real repository, package, or product. Five modules, 53 lines. Seed 20260218.
• AD32. This lesson is not an attack lesson. The code that runs is always on the defense side: checking, scanning, counting, verifying. No exploit code, payload, or evasion procedure is written.
• AD33. The generated patch has two parts: a changed module and a new helper file. No check calls the helper file; it has not yet been wired into the codebase.
• AD34. The six behavioral defects placed on the codebase are single string substitutions in the source text; the sixth is a format defect and breaks the alignment of the output.
• AD35. The acceptance check suite has ten items and is written inside the lesson. It is the codebase’s own check suite, not a generated one.
• AD36. The pattern scan looks for four strings. A finding is not proof but a flag; it points to the line a human should look at.
• AD37. The existence check: if the name on an import line is neither among the codebase’s modules nor among the allowed standard library names, it is fabricated. The fabricated names are fictional.
• AD38. The counted unit is the defect, not the candidate. A candidate can carry more than one defect; a flagged candidate is one that receives a warning from at least one check.

The Fictional Codebase and the Subject of Review

A review needs a body to stand on. The body is this: five small modules of the measurement network, written for reading, tariff calculation, record validation, region summary, and output formatting. The modules are held as text and loaded into a separate namespace when they are to be run; this makes it cheap to run a corrupted version of the same codebase side by side.

# FICTIONAL codebase: the measurement network's own helper scripts. Not a real repository,
# package, or product. Five modules, 53 lines. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def mix(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CODE = {
    "okuma.py": """
def parse_line(line):
    p = line.split(";")
    if len(p) != 4:
        return None
    if not p[3].lstrip("-").isdigit():
        return None
    return {"subscriber": p[0], "region": p[1], "period": p[2], "index": int(p[3])}

def usage(previous, current):
    diff = current - previous
    if diff < 0:
        diff = diff + 100000
    return diff
""",
    "tarife.py": """
TIERS = ((10, 4), (25, 7), (40, 11))
ABOVE = 16

def cost(m3):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    return total + remaining * ABOVE
""",
    "denetim.py": """
VALID_REGIONS = ("east", "west", "north", "south", "central")

def is_valid(record):
    if record is None:
        return False
    if record["index"] < 0:
        return False
    return record["region"] in VALID_REGIONS

def filter_valid(records):
    return [r for r in records if is_valid(r)]
""",
    "ozet.py": """
def region_total(records, usages):
    total = {}
    for r, u in zip(records, usages):
        total[r["region"]] = total.get(r["region"], 0) + u
    return total

def region_count(records):
    count = {}
    for r in records:
        count[r["region"]] = count.get(r["region"], 0) + 1
    return count
""",
    "bicim.py": """
def row(region, m3, amount):
    return region.ljust(8) + str(m3).rjust(7) + str(amount).rjust(9)

def report(title, rows):
    return "\\n".join([title] + rows)
""",
}


def load(code):
    modules = {}
    for name, source in code.items():
        scope = {}
        exec(source, scope)
        modules[name] = scope
    return modules


CLASSES = ("tamamlama", "aciklama", "donusturme")
FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py")
REQUIRED = {0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"),
            3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"),
            5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"),
            7: ("denetim.py", "bicim.py")}
TASKS = [{"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i],
          "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8)]

SIZE = {n: len(s.strip().split("\n")) for n, s in CODE.items()}
MODULES = load(CODE)
print("codebase:", SIZE, "total", sum(SIZE.values()), "lines")
print("task count:", len(TASKS), "-", " , ".join(CLASSES))
print("cost(0,10,25,40,60):", [MODULES["tarife.py"]["cost"](m) for m in (0, 10, 25, 40, 60)])
print("usage(99800,300):", MODULES["okuma.py"]["usage"](99800, 300))

codebase: {'okuma.py': 13, 'tarife.py': 13, 'denetim.py': 11, 'ozet.py': 11, 'bicim.py': 5} total 53 lines
task count: 24 - tamamlama , aciklama , donusturme
cost(0,10,25,40,60): [0, 40, 145, 310, 630]
usage(99800,300): 500

The last two lines fix the correct behavior of the codebase. In the tiered tariff, sixty cubic meters comes to 630; when the meter wraps from ninety-nine thousand to three hundred, the usage is 500. These two numbers will be the yardstick from now on: if a patch changes them, the patch is wrong. This is the first rule of the review procedure: correctness is not an impression but an expectation written in advance.

The twenty-four fictional tasks are divided into three classes, and each task concerns certain modules. The patch that arrives as the answer to a task changes one of those modules and leaves a helper file next to it. The reach of the checks will be divided between these two parts.

The Reach of Three Checks

Three checks look at three separate places. The check run sees only the changed module and only the behavior that the checks measure. The pattern scan looks at the text; it does not run it. The existence check looks only at import lines. The union of the three is not the whole codebase, and what this lesson measures is exactly the gap in between.

DEFECTS = [("H1", "tarife.py", "ABOVE = 16", "ABOVE = 11"),
           ("H2", "okuma.py", "if diff < 0:", "if diff < -100000:"),
           ("H3", "denetim.py", 'record["index"] < 0', 'record["index"] < -1'),
           ("H4", "ozet.py", 'total.get(r["region"], 0) + u', 'total.get(r["region"], 0) + 1'),
           ("H5", "okuma.py", "len(p) != 4", "len(p) < 4"),
           ("H6", "bicim.py", "rjust(9)", "rjust(7)")]
FABRICATED = ("measurement_tools", "tariff_helper", "region_summary", "meter_kit")
PATTERNS = ("eval(", "except:", "== None", "open(")
EXTRA = {"eval(": "def evaluate(expression):\n    return eval(expression)\n",
         "except:": "def to_int(s):\n    try:\n        return int(s)\n    except:\n        return 0\n",
         "== None": "def is_empty(record):\n    return record == None\n",
         "open(": "def backup(path):\n    return open(path).read()\n"}
STANDARD = ("math", "json")


def mutate(code, defect):
    _, file, old, new = defect
    new_code = dict(code)
    new_code[file] = code[file].replace(old, new, 1)
    return new_code


def generate(task, seed=SEED):
    r = rng(seed ^ mix(task["name"]))
    u = [r() for _ in range(6)]
    defect = DEFECTS[int(u[1] * 6)] if u[0] < 0.46 else None
    imported = FABRICATED[int(u[3] * 4)] if u[2] < 0.30 else "math"
    pattern = PATTERNS[int(u[5] * 4)] if u[4] < 0.34 else None
    return {"code": mutate(CODE, defect) if defect else dict(CODE),
            "defect": defect[0] if defect else None, "imported": imported, "pattern": pattern,
            "helper": "import " + imported + "\n\n" + (EXTRA[pattern] if pattern else "")}


RECORDS = [{"region": "east", "index": 120}, {"region": "star", "index": 40},
           {"region": "west", "index": 90}]


def run_checks(code):
    a = load(code)
    o, t, d, z, b = (a["okuma.py"], a["tarife.py"], a["denetim.py"],
                     a["ozet.py"], a["bicim.py"])
    failed = []
    for name, result in (
            ("parse-fields", o["parse_line"]("A017;east;2026-01;41250") ==
             {"subscriber": "A017", "region": "east", "period": "2026-01", "index": 41250}),
            ("parse-extra", o["parse_line"]("A017;east;2026-01;41250;extra") is None),
            ("usage-wrap", o["usage"](99800, 300) == 500),
            ("cost-tiers", [t["cost"](m) for m in (0, 10, 25, 40, 60)] ==
             [0, 40, 145, 310, 630]),
            ("valid-negative", d["is_valid"]({"region": "east", "index": -1}) is False),
            ("filter-region", len(d["filter_valid"](RECORDS)) == 2),
            ("total-region", z["region_total"](RECORDS, [12, 5, 8]) ==
             {"east": 12, "star": 5, "west": 8}),
            ("count-region", z["region_count"](RECORDS) ==
             {"east": 1, "star": 1, "west": 1}),
            ("format-row", "east" in b["row"]("east", 12, 48)),
            ("format-report", b["report"]("head", ["a", "b"]).count("\n") == 2)):
        if not result:
            failed.append(name)
    return failed


def scan_patterns(text):
    return [p for p in PATTERNS if p in text]


def check_dependencies(text, code):
    missing = []
    for line in text.split("\n"):
        if line.startswith("import "):
            a = line[7:].strip()
            if a + ".py" not in code and a not in STANDARD:
                missing.append(a)
    return missing


def measure(seed=SEED):
    defects = {"behavior": 0, "pattern": 0, "fabricated": 0}
    caught = {"checks": 0, "pattern": 0, "dependency": 0}
    missed, silent, flagged = [], 0, 0
    for task in TASKS:
        a = generate(task, seed)
        flag = 0
        if a["pattern"]:
            defects["pattern"] += 1
            if scan_patterns(a["helper"]):
                caught["pattern"] += 1
                flag += 1
        if a["imported"] in FABRICATED:
            defects["fabricated"] += 1
            if check_dependencies(a["helper"], a["code"]):
                caught["dependency"] += 1
                flag += 1
        if a["defect"]:
            defects["behavior"] += 1
            if run_checks(a["code"]):
                caught["checks"] += 1
                flag += 1
            else:
                missed.append((task["name"], a["defect"],
                               "flagged" if flag else "clean"))
                silent += flag == 0
        flagged += flag > 0
    return defects, caught, missed, silent, flagged


print("baseline pattern findings:", scan_patterns("".join(CODE.values())) or "none")
print("defect file         failed checks")
for h in DEFECTS:
    print("%-6s %-12s %s" % (h[0], h[1], run_checks(mutate(CODE, h)) or "none"))
print()
print("seed      | behavior pattern fabricated | checks pattern dependency"
      " | flagged missed silent")
result = {}
for seed in (SEED, SEED + 1):
    d, c, ms, sl, fl = measure(seed)
    result[seed] = ms
    print("%d  | %8d %7d %10d | %6d %7d %10d | %7d %6d %6d"
          % (seed, d["behavior"], d["pattern"], d["fabricated"], c["checks"],
             c["pattern"], c["dependency"], fl, len(ms), sl))
print("missed defects, seed 20260218:", result[SEED])
print("missed defects, seed 20260219:", result[SEED + 1] or "none")

baseline pattern findings: none
defect file         failed checks
H1     tarife.py    ['cost-tiers']
H2     okuma.py     ['usage-wrap']
H3     denetim.py   ['valid-negative']
H4     ozet.py      ['total-region']
H5     okuma.py     ['parse-extra']
H6     bicim.py     none

seed      | behavior pattern fabricated | checks pattern dependency | flagged missed silent
20260218  |       11       8          9 |      9       8          9 |      16      2      1
20260219  |        6       9         12 |      6       9         12 |      17      0      0
missed defects, seed 20260218: [('tamamlama-4', 'H6', 'flagged'), ('aciklama-7', 'H6', 'clean')]
missed defects, seed 20260219: none

The first line says the scan ran over a clean baseline: none of the four searched strings is in the codebase itself. If a finding comes out, it means it came from the generated code; this is the precondition that makes the scan results interpretable.

The second table is this lesson’s run-independent measure. The six behavioral defects are placed on the codebase one by one and the acceptance check suite is run. Five of them each fail a single check: the tiered cost, the counter wrap, the negative index check, the region total, and the row with extra fields. The sixth fails no check at all. The reason is in the check suite itself: the two items that measure the format module look at the region name appearing in the output and at the line count, and do not look at the column width. The defect is there and the check does not look there. This result is independent of the seed, the task distribution, and the generation.

The Residual and How a Flag Misleads

The third table puts the two runs side by side. In the first run, the twenty-four patches contain 11 behavioral defects, 8 pattern findings, and 9 fabricated dependencies; 28 defects in total. The pattern scan catches eight of eight, the existence check nine of nine. The check run catches nine of eleven. Of the 28 defects, 26 are seen, 2 slip through, and both are format defects.

In the second run nothing slips through, and this does not mean the check has improved. In that run no format defect was generated; all six of the behavioral defects that were generated are in places the check looks. The number missed depends on the distribution of defects, not on the coverage of the check. The number that reads the coverage is in the second table, and there one defect in six stands uncovered. The strength of a check is measured not by the zero it gives on a good run but by the name of the class it cannot reach.

The last line questions the flag itself. One of the two missed patches (tamamlama-4) is in fact flagged: it took a fabricated dependency and its helper file has a pattern finding. A human who looks at this patch sees two warnings, fixes both, and accepts the patch. What is fixed is a defect, but not the whole of the defects; the format error enters the codebase together with the acceptance. The second (aciklama-7) receives no warning at all and enters directly. The cost column is here too: in the first run 16 candidates are flagged, that is, the number of patches a human has to look at is sixteen of twenty-four; one of the eight patches that pass unflagged is defective. In the second run the flagged count rises to 17 and the gain is zero, because nothing slipped through in that run anyway. The review load grows not with the defect count but with the flag count.

Summary

• The fictional codebase is five modules and 53 lines; its correct behavior is fixed by two numbers, 630 in the tiered cost and 500 in the counter wrap.
• Three checks look at three separate places: the check run at the behavior of the changed module, the pattern scan at the text, the existence check only at the import lines.
• The run-independent measure: five of the six behavioral defects fail the acceptance check suite, the format defect fails none, because the check suite does not look at the column width.
• In the first run 26 of 28 defects are caught and 2 are missed; in the second run nothing is missed, but the reason is that the distribution of defects changed, not the coverage.
• The number of flagged candidates is not the number of caught defects: one of the two missed patches is flagged because of another defect, and it is accepted once its warnings are fixed.
• The review cost is measured by the flag count; in the first run 16 of 24 patches go to a human, and one of the eight that pass unflagged carries a defect.

Next Step

In this lesson the acceptance check suite was written by hand and its coverage was a known quantity: ten items, five modules, one known blind spot. Writing the check suite is also a job, and it is the most often suggested of the tasks that can be handed to the same helper. The next lesson moves the check suite to the generated side and asks the same question of the check suite itself: how many of the six defects placed on the codebase do the generated checks catch, how many checks give the same result on all versions and distinguish nothing, and how many claims in the generated documentation can be verified by machine.