--- title: 'Limits and Risks' source: 'https://academia.sh/en/courses/model-apis/limits-and-risks' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' objectives: - 'Counts the fields that enter the context in a request bundle by secret class and computes the coverage of a filter that looks at field names.' - 'Counts the fabricated dependency names in generated code with an existence check.' - 'Compares the share of wrong among the accepted suggestions with the baseline share and shows that the acceptance decision did not reduce defects.' - 'Writes the cost of the three risks as context lines, dependency names, and the number of accepted defects.' --- # Limits and Risks Counting the three risks of AI-assisted development: of the 456 context lines in twenty-four request bundles, 168 fall into a secret class; a filter that looks at field names removes all of them but cannot reach the 12 traces left in free text, so its coverage stays at 0.9333; nine of the generated pieces of code carry a dependency name that does not exist; and ten of the nineteen accepted suggestions are defective, so the share of wrong among the accepted is 0.5263, above the baseline share of 0.4583. The previous two lessons measured the quality of the generated output: the correctness of the patch, the discriminating power of the test, the checkability of the documentation. All three assume that the defect is **in the codebase** and look for it after the output is obtained. At the two ends of the handoff, though, there is something that has not been counted. At the input end it is **what was sent** to the model, and at the output end it is **the acceptance decision itself**. This lesson counts the three risks one by one: how many fields in the text that enters the context fall into a secret class and how many a filter removes, how many dependency names in the generated code point to something that does not exist, and what share of the accepted suggestions is wrong. The procedure of secret management was covered in the Application Security and DevSecOps curriculum, and dependency policy and component analysis were measured in the same curriculum; neither is repeated here. The only thing added is **the count of these three risks**. - **AD48.** The codebase is **fictional** and is five modules and **53 lines**. The subscriber records, the working settings, and the support requests are fictional too. Seed **20260218**. - **AD49.** This is not an attack lesson. **The code that runs is always on the defending side**: counting, classification, filter, existence check. No exploit code, payload, evasion, or publishing procedure is written. - **AD50.** A **request bundle** is the whole of the context sent with a task: the task line, the file list, one to three subscriber records, the working settings, and a support request. - **AD51.** Five field names map to four **secret classes**: identity, contact, location, and access. The values are fictional and none of them is real data. - **AD52.** **The filter looks at the field name**: if the key of the line appears in the secret-class table, the line is removed. That is the whole procedure; key management, masking, and storage are not covered. - **AD53.** A **trace word** is the first part of a secret-class field name. A trace that appears in free text is a leak the filter cannot see, and it is counted separately. - **AD54.** A **fabricated dependency** is a name in an `import` line that is found neither among the modules of the codebase nor among the allowed standard library names. The fabricated names are fictional and no real package, repository, or publishing path is mentioned. - **AD55.** A **defective suggestion** is a suggestion that changes the behavior of the codebase. The defect is the change of a single comparison boundary or a single constant. - **AD56.** **Acceptance** is the developer's decision to take the suggestion as it is. The decision depends on context overlap, task class, and difficulty; it **does not look** at whether the suggestion has a defect, because these defects are single-character changes. The truth of this assumption is tested separately in the measurement. - **AD57.** The numbers are printed on two seeds; the second run's seed is **20260219**. ## Secret-Class Fields That Enter the Context What is sent when a task is given to a code assistant is not the task sentence. Along with it go the file list, sample records, working settings, and often a user text that is the source of the problem. This bundle leaves over a network, and what is in it is not known **until it is counted**. ```python # FICTIONAL: the helper codebase of a measurement network, five modules and 53 lines, with # subscriber records, working settings, and support requests. None of it is real. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ("tamamlama", "aciklama", "donusturme") FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py") REQUIRED = {0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py")} TASKS = [{"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8)] SECRET_FIELDS = {"subscriber_id": "identity", "contact": "contact", "address": "location", "access_string": "access", "session_key": "access"} TRACE_WORDS = tuple(sorted({a.split("_")[0] for a in SECRET_FIELDS})) SUBSCRIBERS = [{"subscriber_id": "A0417", "contact": "line-3", "address": "east-12", "region": "east", "period": "2026-01", "index": "41250"}, {"subscriber_id": "A0912", "address": "west-4", "region": "west", "period": "2026-01", "index": "38104"}, {"subscriber_id": "A1330", "contact": "line-1", "region": "north", "period": "2026-01", "index": "50077"}] SETTINGS = {"measurement_network": "east", "timeout": "20", "log_path": "data/measurement", "access_string": "SAMPLE", "session_key": "SAMPLE"} REQUESTS = ["my address changed please register me for the new period", "my contact information looks wrong in the system", "the meter reading was entered incomplete please correct it", "my tariff tier was calculated wrongly", "my subscriber record was opened twice please merge them", "the reading was not taken after the outage"] def build_context(task, i): s = ["task: " + task["name"], "file: " + ",".join(task["required"])] for rec in SUBSCRIBERS[:1 + i % 3]: s += [a + ": " + d for a, d in rec.items()] s += [a + ": " + d for a, d in SETTINGS.items()] s.append("request: " + REQUESTS[i % len(REQUESTS)]) return s def count_secrets(lines): count = {} for s in lines: label = SECRET_FIELDS.get(s.split(":")[0]) if label: count[label] = count.get(label, 0) + 1 return count def apply_filter(lines): return [s for s in lines if s.split(":")[0] not in SECRET_FIELDS] def free_text_traces(lines): return sum(1 for s in lines if s.startswith("request:") for w in TRACE_WORDS if w in s) totals = {"lines": 0, "secret": 0, "kept": 0, "trace": 0} class_count = {} for i, task in enumerate(TASKS): bundle = build_context(task, i) kept = apply_filter(bundle) totals["lines"] += len(bundle) totals["secret"] += len(bundle) - len(kept) totals["kept"] += len(kept) totals["trace"] += free_text_traces(kept) for label, n in count_secrets(bundle).items(): class_count[label] = class_count.get(label, 0) + n print("trace words:", TRACE_WORDS) print("context lines:", totals["lines"], "| after filtering:", totals["kept"]) print("secret class fields:", dict(sorted(class_count.items())), "total", totals["secret"]) exposed = totals["secret"] + totals["trace"] print("removed by filter:", totals["secret"], "| traces left in free text:", totals["trace"], "| coverage: %.4f" % (totals["secret"] / exposed)) print("sample context (task 1), filtered:") for s in apply_filter(build_context(TASKS[0], 0)): print(" " + s) ``` ``` trace words: ('access', 'address', 'contact', 'session', 'subscriber') context lines: 456 | after filtering: 288 secret class fields: {'access': 48, 'contact': 32, 'identity': 48, 'location': 40} total 168 removed by filter: 168 | traces left in free text: 12 | coverage: 0.9333 sample context (task 1), filtered: task: tamamlama-1 file: okuma.py region: east period: 2026-01 index: 41250 measurement_network: east timeout: 20 log_path: data/measurement request: my address changed please register me for the new period ``` In the twenty-four request bundles there are **456 context lines**, and **168** of them fall into one of the four secret classes: identity 48, access 48, location 40, contact 32. That is, a little more than one in every three lines sent is a field that contributes nothing to the task but still leaves. This ratio is **0.3684** and it is written nowhere; it is not known without counting. The filter removes **all** of the structured fields: 168 of 168. By its own criterion this is a perfect result. But the criterion is wrong. What the filter looks at is the field name, and secret-class information does not live only in the field name. The **12 traces** left in free text are the proof; the real coverage of the filter is **0.9333**. In the sample bundle this is plain to see: after the `subscriber_id`, `contact`, and `address` lines have been cleaned, the request line at the bottom still talks about the subscriber's address. Deleting a structured field does not delete the sentence that carries the same information. The cost of the filter can be read in the same output. In the remaining record there are region, period, and index and **no subscriber field**. That is the first of the four columns of the parser, and the task concerns the reading module. Removing the secret-class fields also takes away one column of the example the task needs. ## Fabricated Dependency and the Acceptance Decision The second and third risks are on the output side. Generated code can carry a dependency name, and that name may correspond to nothing. The way to see this is not to **try** the name but to look at whether the name exists: the modules of the codebase and the allowed standard library names are a known set, and every name outside it is suspect. The third risk is the hardest to measure, because what is measured is not the model's output but **the human's decision**. ```python FABRICATED = ("measurement_tools", "tariff_helper", "region_summary", "meter_kit") PATTERNS = ("eval(", "except:", "== None", "open(") STANDARD = ("math", "json") BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18} def generate_code(task, seed=SEED): r = rng(seed ^ mix(task["name"])) u = [r() for _ in range(6)] return {"defective": u[0] < 0.46, "imported": FABRICATED[int(u[3] * 4)] if u[2] < 0.30 else "math", "pattern": PATTERNS[int(u[5] * 4)] if u[4] < 0.34 else None} def existence_check(name): return name + ".py" not in FILES and name not in STANDARD def assistant(task, selection, seed=SEED): r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(selection)))) overlap = sum(1 for f in task["required"] if f in selection) / len(task["required"]) irrelevant = sum(1 for f in selection if f not in task["required"]) / len(FILES) score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"] score = min(max(score, 0.0), 1.0) return {"accepted": r() < score, "score": score, "tokens": 40 + 26 * len(selection) + (18 if task["label"] == "donusturme" else 0)} def measure(seed=SEED): d = {"fabricated": 0, "distinct": set(), "pattern": 0, "defective": 0, "accepted": 0, "accepted_wrong": 0, "rejected_right": 0, "clean_accepted": 0, "tokens": 0} for task in TASKS: code = generate_code(task, seed) y = assistant(task, task["required"], seed) if existence_check(code["imported"]): d["fabricated"] += 1 d["distinct"].add(code["imported"]) d["pattern"] += code["pattern"] is not None d["defective"] += code["defective"] d["accepted"] += y["accepted"] d["tokens"] += y["tokens"] d["accepted_wrong"] += y["accepted"] and code["defective"] d["rejected_right"] += (not y["accepted"]) and (not code["defective"]) d["clean_accepted"] += y["accepted"] and not code["defective"] return d print("seed | fabricated distinct pattern | defective accepted accepted-wrong" " | baseline share accepted share | rejected-right") for seed in (SEED, SEED + 1): d = measure(seed) print("%d | %10d %8d %7d | %9d %8d %14d | %14.4f %14.4f | %14d" % (seed, d["fabricated"], len(d["distinct"]), d["pattern"], d["defective"], d["accepted"], d["accepted_wrong"], d["defective"] / 24, d["accepted_wrong"] / d["accepted"], d["rejected_right"])) print("acceptance rate | on defective suggestions | on clean suggestions") for seed in (SEED, SEED + 1): d = measure(seed) de, cl = d["defective"], 24 - d["defective"] print("%d | %-24s | %s" % (seed, "%d/%d = %.4f" % (d["accepted_wrong"], de, d["accepted_wrong"] / de), "%d/%d = %.4f" % (d["clean_accepted"], cl, d["clean_accepted"] / cl))) d = measure(SEED) print("fabricated names:", sorted(d["distinct"])) print("context tokens per request:", d["tokens"] // 24) ``` ``` seed | fabricated distinct pattern | defective accepted accepted-wrong | baseline share accepted share | rejected-right 20260218 | 9 4 8 | 11 19 10 | 0.4583 0.5263 | 4 20260219 | 12 4 9 | 6 19 5 | 0.2500 0.2632 | 4 acceptance rate | on defective suggestions | on clean suggestions 20260218 | 10/11 = 0.9091 | 9/13 = 0.6923 20260219 | 5/6 = 0.8333 | 14/18 = 0.7778 fabricated names: ['measurement_tools', 'meter_kit', 'region_summary', 'tariff_helper'] context tokens per request: 94 ``` The number for the second risk is in the first columns. **Nine** of the twenty-four generated pieces refer to a name that does not exist, and they are spread over only **four distinct names**. In the second run the number is **12**, and the distinct-name count is again four. A fabricated dependency is not scattered noise: the same small set of names is produced again and again, because the name is derived from the codebase's own naming habit and looks plausible. This is also why the existence check is cheap: the set to compare against is known, and the check is nothing more than a string test. ## The Number for Overconfidence The third risk is read in the last columns, and it is the most uncomfortable number of this lesson. In the first run **11** of the twenty-four suggestions are defective; the baseline share is **0.4583**. The developer accepts **19** of these suggestions. **10 of the accepted ones are defective**, that is, the share of wrong among the accepted is **0.5263**. The acceptance decision **did not lower** the share of wrong relative to the baseline share; in this run it even raised it slightly. The second run repeats the same structure with a lower baseline share: defective **6**, baseline share **0.2500**, acceptance again **19**, the share of wrong among the accepted **0.2632**. In both runs the acceptance share is above the baseline and the two numbers are close to each other. What is to be read from this is the following: **the acceptance decision carries no information about defects.** The proportion of wrong among the accepted falls in the same place as the proportion of wrong in the pool of suggestions. The last table tests this directly. The acceptance rate on defective suggestions is **0.9091** in the first run and on clean suggestions **0.6923**; in the second run **0.8333** and **0.7778**. All four numbers are close to each other, and the defective side is not lower in either run. Assumption AD56 is thus tested: a single-character change is not seen in a quick reading. **Four** of the five rejected suggestions, meanwhile, are clean; the rejection decision carries no information either, it only produces work. The cost is collected in three places. On the context side **94 tokens** go out per request, and part of that is secret-class fields. On the dependency side nine of the twenty-four pieces fall to review. On the acceptance side the cost is written directly into the codebase: in the first run **10 defects** were accepted, that is, they entered the patches that go into review through the developer's own decision. The three checks built in the previous lesson are therefore not a decoration; they are the second line **behind** the acceptance decision. ## Summary - In the twenty-four request bundles, 168 of the 456 context lines fall into four secret classes: identity 48, access 48, location 40, contact 32. - A filter that looks at field names removes all of the structured fields but cannot reach the 12 traces in free text; the real coverage is 0.9333. - The cost of the filter is visible: when the subscriber field is removed, one of the parser's four columns drops out of the example. - Nine of the twenty-four generated pieces carry a dependency name that does not exist, and they are spread over only four distinct names; in the second run the number is 12 and the distinct-name count does not change. - Ten of the nineteen accepted suggestions are defective; the share of wrong among the accepted is 0.5263 and the baseline share is 0.4583, and in the second run 0.2632 and 0.2500. - The acceptance rates on defective and clean suggestions are close to each other and the defective side is not lower in either run: the acceptance decision carries no information about defects. ## Next Step Over six lessons the assistant itself was measured: in which task class it is accepted, what context adds and where it drags down, where the checkpoint stops the error, what the three checks catch, what the generated test discriminates, and that the acceptance decision discriminates nothing. This topic closes here. One thing stayed constant throughout the course's twenty lessons and was never questioned: the model was always an endpoint that stood **far away**. The call went out over the network, queued, hit the rate limit, was cut off midway, fell into the cache, and was paid for in tokens. The context bundle left by the same road, and the fields in it had to be counted. Taking the endpoint onto **your own hardware** moves all these numbers to another plane: network latency and rate limit go away, and in their place come a memory budget, the number of concurrent requests, and the hardware itself. The next topic measures this trade-off and asks the first question like this: when you host a model yourself, what exactly is held in memory, and how many concurrent requests does that budget cover.