Limits and Risks

The previous two lessons measured the quality of the generated output: the correctness of the patch, the discriminating power of the test, the checkability of the documentation. All three assume that the defect is in the codebase and look for it after the output is obtained. At the two ends of the handoff, though, there is something that has not been counted. At the input end it is what was sent to the model, and at the output end it is the acceptance decision itself.

This lesson counts the three risks one by one: how many fields in the text that enters the context fall into a secret class and how many a filter removes, how many dependency names in the generated code point to something that does not exist, and what share of the accepted suggestions is wrong. The procedure of secret management was covered in the Application Security and DevSecOps curriculum, and dependency policy and component analysis were measured in the same curriculum; neither is repeated here. The only thing added is the count of these three risks.

• AD48. The codebase is fictional and is five modules and 53 lines. The subscriber records, the working settings, and the support requests are fictional too. Seed 20260218.
• AD49. This is not an attack lesson. The code that runs is always on the defending side: counting, classification, filter, existence check. No exploit code, payload, evasion, or publishing procedure is written.
• AD50. A request bundle is the whole of the context sent with a task: the task line, the file list, one to three subscriber records, the working settings, and a support request.
• AD51. Five field names map to four secret classes: identity, contact, location, and access. The values are fictional and none of them is real data.
• AD52. The filter looks at the field name: if the key of the line appears in the secret-class table, the line is removed. That is the whole procedure; key management, masking, and storage are not covered.
• AD53. A trace word is the first part of a secret-class field name. A trace that appears in free text is a leak the filter cannot see, and it is counted separately.
• AD54. A fabricated dependency is a name in an import line that is found neither among the modules of the codebase nor among the allowed standard library names. The fabricated names are fictional and no real package, repository, or publishing path is mentioned.
• AD55. A defective suggestion is a suggestion that changes the behavior of the codebase. The defect is the change of a single comparison boundary or a single constant.
• AD56. Acceptance is the developer’s decision to take the suggestion as it is. The decision depends on context overlap, task class, and difficulty; it does not look at whether the suggestion has a defect, because these defects are single-character changes. The truth of this assumption is tested separately in the measurement.
• AD57. The numbers are printed on two seeds; the second run’s seed is 20260219.

Secret-Class Fields That Enter the Context

What is sent when a task is given to a code assistant is not the task sentence. Along with it go the file list, sample records, working settings, and often a user text that is the source of the problem. This bundle leaves over a network, and what is in it is not known until it is counted.

# FICTIONAL: the helper codebase of a measurement network, five modules and 53 lines, with
# subscriber records, working settings, and support requests. None of it is real.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def mix(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CLASSES = ("tamamlama", "aciklama", "donusturme")
FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py")
REQUIRED = {0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"),
            3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"),
            5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"),
            7: ("denetim.py", "bicim.py")}
TASKS = [{"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i],
          "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8)]

SECRET_FIELDS = {"subscriber_id": "identity", "contact": "contact", "address": "location",
                 "access_string": "access", "session_key": "access"}
TRACE_WORDS = tuple(sorted({a.split("_")[0] for a in SECRET_FIELDS}))
SUBSCRIBERS = [{"subscriber_id": "A0417", "contact": "line-3", "address": "east-12",
                "region": "east", "period": "2026-01", "index": "41250"},
               {"subscriber_id": "A0912", "address": "west-4",
                "region": "west", "period": "2026-01", "index": "38104"},
               {"subscriber_id": "A1330", "contact": "line-1",
                "region": "north", "period": "2026-01", "index": "50077"}]
SETTINGS = {"measurement_network": "east", "timeout": "20", "log_path": "data/measurement",
            "access_string": "SAMPLE", "session_key": "SAMPLE"}
REQUESTS = ["my address changed please register me for the new period",
            "my contact information looks wrong in the system",
            "the meter reading was entered incomplete please correct it",
            "my tariff tier was calculated wrongly",
            "my subscriber record was opened twice please merge them",
            "the reading was not taken after the outage"]


def build_context(task, i):
    s = ["task: " + task["name"], "file: " + ",".join(task["required"])]
    for rec in SUBSCRIBERS[:1 + i % 3]:
        s += [a + ": " + d for a, d in rec.items()]
    s += [a + ": " + d for a, d in SETTINGS.items()]
    s.append("request: " + REQUESTS[i % len(REQUESTS)])
    return s


def count_secrets(lines):
    count = {}
    for s in lines:
        label = SECRET_FIELDS.get(s.split(":")[0])
        if label:
            count[label] = count.get(label, 0) + 1
    return count


def apply_filter(lines):
    return [s for s in lines if s.split(":")[0] not in SECRET_FIELDS]


def free_text_traces(lines):
    return sum(1 for s in lines if s.startswith("request:")
               for w in TRACE_WORDS if w in s)


totals = {"lines": 0, "secret": 0, "kept": 0, "trace": 0}
class_count = {}
for i, task in enumerate(TASKS):
    bundle = build_context(task, i)
    kept = apply_filter(bundle)
    totals["lines"] += len(bundle)
    totals["secret"] += len(bundle) - len(kept)
    totals["kept"] += len(kept)
    totals["trace"] += free_text_traces(kept)
    for label, n in count_secrets(bundle).items():
        class_count[label] = class_count.get(label, 0) + n
print("trace words:", TRACE_WORDS)
print("context lines:", totals["lines"], "| after filtering:", totals["kept"])
print("secret class fields:", dict(sorted(class_count.items())), "total", totals["secret"])
exposed = totals["secret"] + totals["trace"]
print("removed by filter:", totals["secret"], "| traces left in free text:", totals["trace"],
      "| coverage: %.4f" % (totals["secret"] / exposed))
print("sample context (task 1), filtered:")
for s in apply_filter(build_context(TASKS[0], 0)):
    print("  " + s)

trace words: ('access', 'address', 'contact', 'session', 'subscriber')
context lines: 456 | after filtering: 288
secret class fields: {'access': 48, 'contact': 32, 'identity': 48, 'location': 40} total 168
removed by filter: 168 | traces left in free text: 12 | coverage: 0.9333
sample context (task 1), filtered:
  task: tamamlama-1
  file: okuma.py
  region: east
  period: 2026-01
  index: 41250
  measurement_network: east
  timeout: 20
  log_path: data/measurement
  request: my address changed please register me for the new period

In the twenty-four request bundles there are 456 context lines, and 168 of them fall into one of the four secret classes: identity 48, access 48, location 40, contact 32. That is, a little more than one in every three lines sent is a field that contributes nothing to the task but still leaves. This ratio is 0.3684 and it is written nowhere; it is not known without counting.

The filter removes all of the structured fields: 168 of 168. By its own criterion this is a perfect result. But the criterion is wrong. What the filter looks at is the field name, and secret-class information does not live only in the field name. The 12 traces left in free text are the proof; the real coverage of the filter is 0.9333. In the sample bundle this is plain to see: after the subscriber_id, contact, and address lines have been cleaned, the request line at the bottom still talks about the subscriber’s address. Deleting a structured field does not delete the sentence that carries the same information.

The cost of the filter can be read in the same output. In the remaining record there are region, period, and index and no subscriber field. That is the first of the four columns of the parser, and the task concerns the reading module. Removing the secret-class fields also takes away one column of the example the task needs.

Fabricated Dependency and the Acceptance Decision

The second and third risks are on the output side. Generated code can carry a dependency name, and that name may correspond to nothing. The way to see this is not to try the name but to look at whether the name exists: the modules of the codebase and the allowed standard library names are a known set, and every name outside it is suspect. The third risk is the hardest to measure, because what is measured is not the model’s output but the human’s decision.

FABRICATED = ("measurement_tools", "tariff_helper", "region_summary", "meter_kit")
PATTERNS = ("eval(", "except:", "== None", "open(")
STANDARD = ("math", "json")
BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18}


def generate_code(task, seed=SEED):
    r = rng(seed ^ mix(task["name"]))
    u = [r() for _ in range(6)]
    return {"defective": u[0] < 0.46,
            "imported": FABRICATED[int(u[3] * 4)] if u[2] < 0.30 else "math",
            "pattern": PATTERNS[int(u[5] * 4)] if u[4] < 0.34 else None}


def existence_check(name):
    return name + ".py" not in FILES and name not in STANDARD


def assistant(task, selection, seed=SEED):
    r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(selection))))
    overlap = sum(1 for f in task["required"] if f in selection) / len(task["required"])
    irrelevant = sum(1 for f in selection if f not in task["required"]) / len(FILES)
    score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"]
    score = min(max(score, 0.0), 1.0)
    return {"accepted": r() < score, "score": score,
            "tokens": 40 + 26 * len(selection) + (18 if task["label"] == "donusturme" else 0)}


def measure(seed=SEED):
    d = {"fabricated": 0, "distinct": set(), "pattern": 0, "defective": 0, "accepted": 0,
         "accepted_wrong": 0, "rejected_right": 0, "clean_accepted": 0, "tokens": 0}
    for task in TASKS:
        code = generate_code(task, seed)
        y = assistant(task, task["required"], seed)
        if existence_check(code["imported"]):
            d["fabricated"] += 1
            d["distinct"].add(code["imported"])
        d["pattern"] += code["pattern"] is not None
        d["defective"] += code["defective"]
        d["accepted"] += y["accepted"]
        d["tokens"] += y["tokens"]
        d["accepted_wrong"] += y["accepted"] and code["defective"]
        d["rejected_right"] += (not y["accepted"]) and (not code["defective"])
        d["clean_accepted"] += y["accepted"] and not code["defective"]
    return d


print("seed      | fabricated distinct pattern | defective accepted accepted-wrong"
      " | baseline share accepted share | rejected-right")
for seed in (SEED, SEED + 1):
    d = measure(seed)
    print("%d  | %10d %8d %7d | %9d %8d %14d | %14.4f %14.4f | %14d"
          % (seed, d["fabricated"], len(d["distinct"]), d["pattern"], d["defective"],
             d["accepted"], d["accepted_wrong"], d["defective"] / 24,
             d["accepted_wrong"] / d["accepted"], d["rejected_right"]))
print("acceptance rate | on defective suggestions | on clean suggestions")
for seed in (SEED, SEED + 1):
    d = measure(seed)
    de, cl = d["defective"], 24 - d["defective"]
    print("%d        | %-24s | %s"
          % (seed, "%d/%d = %.4f" % (d["accepted_wrong"], de, d["accepted_wrong"] / de),
             "%d/%d = %.4f" % (d["clean_accepted"], cl, d["clean_accepted"] / cl)))
d = measure(SEED)
print("fabricated names:", sorted(d["distinct"]))
print("context tokens per request:", d["tokens"] // 24)

seed      | fabricated distinct pattern | defective accepted accepted-wrong | baseline share accepted share | rejected-right
20260218  |          9        4       8 |        11       19             10 |         0.4583         0.5263 |              4
20260219  |         12        4       9 |         6       19              5 |         0.2500         0.2632 |              4
acceptance rate | on defective suggestions | on clean suggestions
20260218        | 10/11 = 0.9091           | 9/13 = 0.6923
20260219        | 5/6 = 0.8333             | 14/18 = 0.7778
fabricated names: ['measurement_tools', 'meter_kit', 'region_summary', 'tariff_helper']
context tokens per request: 94

The number for the second risk is in the first columns. Nine of the twenty-four generated pieces refer to a name that does not exist, and they are spread over only four distinct names. In the second run the number is 12, and the distinct-name count is again four. A fabricated dependency is not scattered noise: the same small set of names is produced again and again, because the name is derived from the codebase’s own naming habit and looks plausible. This is also why the existence check is cheap: the set to compare against is known, and the check is nothing more than a string test.

The Number for Overconfidence

The third risk is read in the last columns, and it is the most uncomfortable number of this lesson. In the first run 11 of the twenty-four suggestions are defective; the baseline share is 0.4583. The developer accepts 19 of these suggestions. 10 of the accepted ones are defective, that is, the share of wrong among the accepted is 0.5263. The acceptance decision did not lower the share of wrong relative to the baseline share; in this run it even raised it slightly.

The second run repeats the same structure with a lower baseline share: defective 6, baseline share 0.2500, acceptance again 19, the share of wrong among the accepted 0.2632. In both runs the acceptance share is above the baseline and the two numbers are close to each other. What is to be read from this is the following: the acceptance decision carries no information about defects. The proportion of wrong among the accepted falls in the same place as the proportion of wrong in the pool of suggestions.

The last table tests this directly. The acceptance rate on defective suggestions is 0.9091 in the first run and on clean suggestions 0.6923; in the second run 0.8333 and 0.7778. All four numbers are close to each other, and the defective side is not lower in either run. Assumption AD56 is thus tested: a single-character change is not seen in a quick reading. Four of the five rejected suggestions, meanwhile, are clean; the rejection decision carries no information either, it only produces work.

The cost is collected in three places. On the context side 94 tokens go out per request, and part of that is secret-class fields. On the dependency side nine of the twenty-four pieces fall to review. On the acceptance side the cost is written directly into the codebase: in the first run 10 defects were accepted, that is, they entered the patches that go into review through the developer’s own decision. The three checks built in the previous lesson are therefore not a decoration; they are the second line behind the acceptance decision.

Summary

• In the twenty-four request bundles, 168 of the 456 context lines fall into four secret classes: identity 48, access 48, location 40, contact 32.
• A filter that looks at field names removes all of the structured fields but cannot reach the 12 traces in free text; the real coverage is 0.9333.
• The cost of the filter is visible: when the subscriber field is removed, one of the parser’s four columns drops out of the example.
• Nine of the twenty-four generated pieces carry a dependency name that does not exist, and they are spread over only four distinct names; in the second run the number is 12 and the distinct-name count does not change.
• Ten of the nineteen accepted suggestions are defective; the share of wrong among the accepted is 0.5263 and the baseline share is 0.4583, and in the second run 0.2632 and 0.2500.
• The acceptance rates on defective and clean suggestions are close to each other and the defective side is not lower in either run: the acceptance decision carries no information about defects.

Next Step

Over six lessons the assistant itself was measured: in which task class it is accepted, what context adds and where it drags down, where the checkpoint stops the error, what the three checks catch, what the generated test discriminates, and that the acceptance decision discriminates nothing. This topic closes here.

One thing stayed constant throughout the course’s twenty lessons and was never questioned: the model was always an endpoint that stood far away. The call went out over the network, queued, hit the rate limit, was cut off midway, fell into the cache, and was paid for in tokens. The context bundle left by the same road, and the fields in it had to be counted. Taking the endpoint onto your own hardware moves all these numbers to another plane: network latency and rate limit go away, and in their place come a memory budget, the number of concurrent requests, and the hardware itself. The next topic measures this trade-off and asks the first question like this: when you host a model yourself, what exactly is held in memory, and how many concurrent requests does that budget cover.