Code Assistants

The lessons of this course so far have always treated the call inside a product: a subscriber writes a short support ticket, a client sends it to a model endpoint, and what was measured was the tokens, retries, cache hits, and outage share of that pipeline. That the same call is made in one more place was never counted: at the developer’s own desk, on code that has not yet reached any subscriber.

The measure there is different. The one who reads the response is not a subscriber but the person who writes the code; and that person either takes the response into use as it is, or throws it away and writes their own. This is the measure of this lesson: acceptance. The lesson builds a fictional codebase, measures three usage classes on the same set of tasks, and counts the class in which the bare call collapses.

• AD1. The codebase is fictional: the measurement network’s own helper scripts. It is not any real repository, package, framework, or product. Five modules, 53 lines.
• AD2. The code assistant is a simulator; no real assistant, editor, or plugin is called. The seed is 20260218, and random is not used.
• AD3. The measure is acceptance: if the developer takes the suggestion into use it is an acceptance, if the developer rewrites it, a rejection. Whether the suggestion is actually correct is a separate question and is not asked in this lesson.
• AD4. Three usage classes (tamamlama for completion, aciklama for explanation, donusturme for transformation in the output): completion (writing a missing body), explanation (saying what an existing piece does), transformation (converting an existing piece into another form). Eight fictional tasks per class, twenty-four in total.
• AD5. The base acceptance probability differs by class: explanation 0.72, completion 0.34, transformation 0.18. The reason is structural: the input of an explanation is inside the prompt; completion and transformation have to fit the rest of the codebase.
• AD6. Context overlap raises acceptance, an irrelevant file lowers it, and task difficulty lowers it. The three are combined into a single score, and the score is clipped between zero and one.
• AD7. The cost is tokens: base 40, 26 per file in the context, 18 extra for transformation. The latency class is the same in all three setups: single call, one per task.
• AD8. The run-independent measure is expected acceptance (the mean of the scores); the counted acceptance is printed separately for two runs (seed 20260218 and seed 20260219).

The Fictional Codebase

The measurement network’s field crew carries a handful of helper scripts that turn meter readings into a report. These scripts are not a product; they are short files with which a team does its own work, and they are fictional. The five modules are held in a dictionary; this way they can be shown in a single block and the context can be selected by file name.

# FICTIONAL CODEBASE -- not a real repository, package, or product. The measurement
# network's own helper scripts: five modules. Seed 20260218.
CODE = {
    "okuma.py": """
def parse_line(line):
    p = line.split(";")
    if len(p) != 4:
        return None
    if not p[3].lstrip("-").isdigit():
        return None
    return {"subscriber": p[0], "region": p[1], "period": p[2], "index": int(p[3])}

def usage(previous, current):
    diff = current - previous
    if diff < 0:
        diff = diff + 100000
    return diff
""",
    "tarife.py": """
TIERS = ((10, 4), (25, 7), (40, 11))
ABOVE = 16

def cost(m3):
    remaining, lower, total = m3, 0, 0
    for upper, unit in TIERS:
        band = min(remaining, upper - lower)
        if band <= 0:
            break
        total += band * unit
        remaining -= band
        lower = upper
    return total + remaining * ABOVE
""",
    "denetim.py": """
VALID_REGIONS = ("east", "west", "north", "south", "central")

def is_valid(record):
    if record is None:
        return False
    if record["index"] < 0:
        return False
    return record["region"] in VALID_REGIONS

def filter_valid(records):
    return [r for r in records if is_valid(r)]
""",
    "ozet.py": """
def region_total(records, usages):
    total = {}
    for r, u in zip(records, usages):
        total[r["region"]] = total.get(r["region"], 0) + u
    return total

def region_count(records):
    count = {}
    for r in records:
        count[r["region"]] = count.get(r["region"], 0) + 1
    return count
""",
    "bicim.py": """
def row(region, m3, amount):
    return region.ljust(8) + str(m3).rjust(7) + str(amount).rjust(9)

def report(title, rows):
    return "\\n".join([title] + rows)
""",
}


def load(code):
    modules = {}
    for name, source in code.items():
        scope = {}
        exec(source, scope)
        modules[name] = scope
    return modules


MODULES = load(CODE)
SIZE = {n: len(s.strip().split("\n")) for n, s in CODE.items()}
print("module      lines")
for n in CODE:
    print(f"  {n:<10}{SIZE[n]:>4}")
print(f"  {'total':<10}{sum(SIZE.values()):>4}")

READINGS = [                                # FICTIONAL meter readings
    "A017;east;2026-01;41250", "A017;east;2026-02;41312",
    "B204;west;2026-02;99800", "B204;west;2026-03;300",
    "C088;north;2026-02;7130", "C088;north;2026-03;7166",
    "D311;outside;2026-03;5000", "E402;south;2026-03;corrupt",
]
records = [MODULES["okuma.py"]["parse_line"](s) for s in READINGS]
clean = MODULES["denetim.py"]["filter_valid"](records)
print("\nread", len(records), "valid", len(clean))

pairs = [(clean[i], clean[i + 1]) for i in range(0, len(clean) - 1, 2)]
usages = [MODULES["okuma.py"]["usage"](a["index"], b["index"]) for a, b in pairs]
latest = [b for _, b in pairs]
totals = MODULES["ozet.py"]["region_total"](latest, usages)
rows = [MODULES["bicim.py"]["row"](r, m, MODULES["tarife.py"]["cost"](m))
        for r, m in sorted(totals.items())]
print(MODULES["bicim.py"]["report"]("region       m3   amount", rows))

module      lines
  okuma.py    13
  tarife.py   13
  denetim.py  11
  ozet.py     11
  bicim.py     5
  total       53

read 8 valid 6
region       m3   amount
east         62      662
north        36      266
west        500     7670

The five modules are interdependent and build a single pipeline: okuma.py (reading) turns a line into a record, denetim.py (checks) discards the invalid ones, ozet.py (summary) totals by region, tarife.py (tariff) computes the tiered cost, bicim.py (render) writes the aligned row. Two of the eight readings are eliminated: one carries an unrecognized region, and one has an index that is not a number. The 500 cubic meters of the west region is also the result of a rule; the meter wrapped from 99800 to 300, and usage compensates for that wrap.

All 53 lines fit inside a single prompt. Keeping the codebase small is deliberate: here “enlarging the context” is really an option and its cost can be counted. In a large repository this option does not exist anyway, but the direction of the measure is the same.

Three Usage Classes and Acceptance

Twenty-four fictional tasks are divided into three classes. Completion asks for writing a body: a new function that takes a region count, a missing branch, an edge case. Explanation asks what an existing piece does: why usage adds 100000, what the TIERS tuple encodes. Transformation converts an existing piece into another form: returning a tuple instead of a dictionary, turning a loop into a comprehension, merging two modules under a common signature.

The acceptance rate of these three classes is not the same, and the reason is written into the simulator as the base probability. The input of an explanation is inside the prompt: the piece to be explained has already been sent, and nothing needs to be known from outside. Completion and transformation, on the other hand, have to fit the unseen part of the codebase: which key names are used, what each function returns, where which invalidity rule is applied.

# ASSISTANT SIMULATOR -- FICTIONAL. No real code assistant is called.
# Acceptance is computed from the task class and the context overlap; random is not used.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def mix(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CLASSES = ("tamamlama", "aciklama", "donusturme")
FILES = ("okuma.py", "tarife.py", "denetim.py", "ozet.py", "bicim.py")
REQUIRED = {                                # task number -> required files
    0: ("okuma.py",),
    1: ("tarife.py",),
    2: ("denetim.py", "okuma.py"),
    3: ("ozet.py", "denetim.py"),
    4: ("bicim.py", "ozet.py"),
    5: ("tarife.py", "okuma.py"),
    6: ("ozet.py", "okuma.py", "tarife.py"),
    7: ("denetim.py", "bicim.py"),
}
TASKS = [                                   # 24 FICTIONAL tasks: 3 classes x 8
    {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)}
    for c in CLASSES
    for i in range(8)
]
BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18}


def assistant(task, context, seed=SEED):
    r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context))))
    overlap = sum(1 for f in task["required"] if f in context) / len(task["required"])
    irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES)
    score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"]
    score = min(max(score, 0.0), 1.0)
    accepted = r() < score
    tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0)
    return {"accepted": accepted, "score": score, "tokens": tokens}


SETUPS = (("none", ()), ("open file", ("okuma.py",)), ("whole codebase", FILES))


def measure(context, label, seed=SEED):
    t = [x for x in TASKS if x["label"] == label]
    s = [assistant(x, context, seed) for x in t]
    return (sum(x["accepted"] for x in s), sum(x["score"] for x in s) / len(s),
            sum(x["tokens"] for x in s))


print("task count", len(TASKS), " example:",
      ", ".join(t["name"] + " -> " + "+".join(t["required"]) for t in TASKS[:2]))
print("\ncontext           class       expected accept A accept B")
for name, ctx in SETUPS:
    for label in CLASSES:
        acc_a, p, _ = measure(ctx, label)
        acc_b, _, _ = measure(ctx, label, SEED + 1)
        print(f"{name:<18}{label:<12}{p:>8.4f}{acc_a:>7}/8{acc_b:>7}/8")

print("\ncontext           total A   total B   tokens    accepted/1000 tokens")
for name, ctx in SETUPS:
    total_a = sum(measure(ctx, label)[0] for label in CLASSES)
    total_b = sum(measure(ctx, label, SEED + 1)[0] for label in CLASSES)
    tokens = sum(measure(ctx, label)[2] for label in CLASSES)
    print(f"{name:<18}{total_a:>7}/24{total_b:>8}/24{tokens:>10}{1000 * total_a / tokens:>17.2f}")

task count 24  example: tamamlama-1 -> okuma.py, tamamlama-2 -> tarife.py

context           class       expected accept A accept B
none              tamamlama     0.2650      1/8      1/8
none              aciklama      0.6450      4/8      6/8
none              donusturme    0.1050      2/8      0/8
open file         tamamlama     0.3954      3/8      1/8
open file         aciklama      0.7417      5/8      6/8
open file         donusturme    0.2429      2/8      1/8
whole codebase    tamamlama     0.6275      6/8      7/8
whole codebase    aciklama      0.9813      8/8      8/8
whole codebase    donusturme    0.4675      4/8      5/8

context           total A   total B   tokens    accepted/1000 tokens
none                    7/24       7/24      1104             6.34
open file              10/24       8/24      1728             5.79
whole codebase         18/24      20/24      4224             4.26

Where the Bare Call Collapses

The first three rows are the table of the bare call: there is no context, and only the task description is sent. Seven of the twenty-four tasks are accepted, and seven in the second run as well. Even though the total looks constant, the distribution is not constant, and the real information is in the distribution.

The collapse is in two classes. Completion 1/8, the same in both runs; expected acceptance 0.2650. Transformation 2/8 and 0/8 in the second run; expected acceptance 0.1050, the lowest number in the table. The only class left standing is explanation: 4/8 and 6/8, expected acceptance 0.6450. A developer can ask an assistant without context “what does this function do” and use more than half of the answers; when asking the same assistant to “complete this function”, the developer will throw away seven of eight attempts.

The reason is structural, and the number confirms it. In an explanation task, everything that needs to be known has been put into the prompt. In a completion task, however, the keys of the dictionary that the body to be written will return, the name of the function it will call, and the invalidity rule it must follow are in other files. Because the assistant cannot see them, it produces code that is plausible but does not fit; plausible code is also rejected code.

Within a class there is also a distribution, and a single number hides it. The eight tasks of each class are spread over four distinct difficulty levels, and difficulty is subtracted directly from the acceptance score; the mean of the class lies somewhere between the easiest task and the hardest one. The run-to-run difference measures a separate thing: the same task, the same context, and the same score can reach two distinct decisions under two seeds. This is why the expected acceptance column is placed next to the counted acceptance in the table; the counted acceptance depends on the run, the expected acceptance on the setup itself.

What Context Adds and What It Costs

The second setup is what a developer does most readily: also sending the file that is open at that moment. Here that file is okuma.py, and it is the same for every task. The total rises from 7/24 to 10/24, a gain of 3/24. The run-to-run difference of the same setup is 2/24 (10 versus 8). The gain is only one and a half times the difference; moreover, in the completion column the direction reverses (3/8 and 1/8). A single fixed file is not scaffolding; when it is relevant to the task it helps, and when it is not it counts as irrelevant context.

The third setup is not ambiguous. When the whole codebase is sent, the total is 18/24, and 20/24 in the second run; the gain over the baseline is 11/24, more than five times the run-to-run difference. All three classes rise: completion from 1/8 to 6/8, explanation from 4/8 to 8/8, transformation from 2/8 to 4/8.

The cost column writes the price of this gain. The number of calls is 24 in all three setups and the latency class stays the same as single call; only the tokens change: from 1104 to 4224, about 3.8 times. Acceptance per thousand tokens falls from 6.34 to 5.79, and from there to 4.26. The scaffolding improved the work and lowered the yield per unit; both are true at the same time, and both have to be written down.

Finally, the run-independent column fixes an ordering. Transformation stays at 0.4675 even when the whole codebase is given; explanation is at 0.6450 with no context at all. Adding context does not change the places of the classes; it moves all of them up together. Describing a code assistant as “good” or “bad” with a single number is therefore misleading: the number changes with the task class, and the hard class that receives the most context can remain behind the easy class that receives the least.

Summary

• The fictional codebase is five modules and 53 lines; reading, validation, summary, tariff, and format build a single report pipeline, and the whole of it fits inside a prompt.
• The measure is acceptance and it is counted separately for the three usage classes; the difference between classes is structural, because the input of an explanation is inside the prompt, while the input of completion and transformation is outside.
• The call without context gives 7/24 acceptance; the collapse is in completion (1/8) and transformation (2/8), and the only class left standing is explanation (4/8, expected 0.6450).
• Adding a single fixed file gives 10/24, but the run-to-run difference of the same setup is 2/24 and the direction reverses in the completion column: fixed context is not scaffolding.
• The whole codebase raises it to 18/24; the cost rises from 1104 tokens to 4224 tokens, and acceptance per thousand tokens falls from 6.34 to 4.26. The number of calls and the latency class do not change.
• Context does not change the order of the classes: transformation with full context (0.4675) stays below explanation without context (0.6450).

Next Step

In this lesson the context was given three times as fixed: none, always the same file, or all of it. None of the three is a choice. At a real desk, the context is rebuilt for every task, and two questions are asked: which files, and how many. The table above leaves both of these questions unanswered; it shows only the ends. The next lesson fills the gap: as the number of files increases one by one, how the tokens grow, where acceptance peaks, and what a file irrelevant to the task does to acceptance. What will be measured there is not the length of the context but its selection.