Providing Context

The previous lesson gave the context three times as fixed: none, always the same file, or all of it. The distance between the ends was large — from seven of twenty-four tasks to eighteen — but none of the three was a choice. At a developer’s desk the context is rebuilt for every task, and two separate questions are asked: which files and how many.

This lesson separates those two questions. First the context is enlarged: the same set of files is given to everyone and its size grows one by one. Then the context is selected: each task receives only the files that task requires. The two curves are placed side by side on the token axis, and a third number is added on top: what a file irrelevant to the task does to acceptance.

• AD9. The codebase is the same fictional codebase built with its bodies in the previous lesson: five modules, 53 lines. Only its structure is needed here, because the context is selected by file name.
• AD10. The set of tasks and the assistant simulator are also the same: twenty-four fictional tasks, three classes, seed 20260218, no random.
• AD11. The effect of context on acceptance has two terms: overlap (how many of the required files are in the context) raises acceptance, irrelevance (how many of the files in the context are unnecessary) lowers it.
• AD12. The cost is a flat 26 tokens per file; file size does not enter the cost. In a real context a large file is expensive, so the cost curve here is optimistic.
• AD13. The number of files sent is counted separately; it is a total for the twenty-four tasks and a volume measure independent of tokens.
• AD14. The context provider is fictional and has a single setting: the hit rate, that is, the share of tasks in which it finds the required set correctly. When it misses, it drops one required file and puts an irrelevant file in its place; the file count does not change, only which file it is.
• AD15. The provider’s own draw is fixed by the seed; in the second run only the uncertainty of the assistant changes. This way the difference between the two runs is attributed only to the model, not to the selection.
• AD16. The run-independent measure is again expected acceptance; the counted acceptance is printed separately for two seeds. The latency class is single call in all setups, one per task.

The Codebase Structure and the Provider

For context selection, what is needed is not the bodies of the codebase but its structure: which modules exist, how long they are, and which names they expose to the outside. This is also the plane a context provider works on.

# THE SAME FICTIONAL CODEBASE -- five modules, 53 lines; the bodies were built in the previous lesson.
# Only its structure is needed here: size and exposed names.
SIZE = {"okuma.py": 13, "tarife.py": 13, "denetim.py": 11, "ozet.py": 11, "bicim.py": 5}
EXPOSED = {
    "okuma.py": ("parse_line", "usage"),
    "tarife.py": ("TIERS", "ABOVE", "cost"),
    "denetim.py": ("VALID_REGIONS", "is_valid", "filter_valid"),
    "ozet.py": ("region_total", "region_count"),
    "bicim.py": ("row", "report"),
}
FILES = tuple(SIZE)

SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def mix(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CLASSES = ("tamamlama", "aciklama", "donusturme")
REQUIRED = {
    0: ("okuma.py",),
    1: ("tarife.py",),
    2: ("denetim.py", "okuma.py"),
    3: ("ozet.py", "denetim.py"),
    4: ("bicim.py", "ozet.py"),
    5: ("tarife.py", "okuma.py"),
    6: ("ozet.py", "okuma.py", "tarife.py"),
    7: ("denetim.py", "bicim.py"),
}
TASKS = [                                   # 24 FICTIONAL tasks: 3 classes x 8
    {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)}
    for c in CLASSES
    for i in range(8)
]
BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18}


def assistant(task, context, seed=SEED):   # SIMULATOR -- FICTIONAL
    r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context))))
    overlap = sum(1 for f in task["required"] if f in context) / len(task["required"])
    irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES)
    score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"]
    score = min(max(score, 0.0), 1.0)
    accepted = r() < score
    tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0)
    return {"accepted": accepted, "score": score, "tokens": tokens}


def provider(task, hit, seed=SEED):
    """FICTIONAL context provider. When it hits, it finds the required set; when it misses,
    it drops one required file and puts an irrelevant file in its place. The file count stays the same."""
    r = rng(seed ^ mix("saglayici|" + task["name"]))
    if r() < hit:
        return tuple(task["required"])
    extra = tuple(f for f in FILES if f not in task["required"])[:1]
    return tuple(task["required"][1:]) + extra


print("module      lines  exposed names")
for f in FILES:
    print(f"  {f:<10}{SIZE[f]:>4}   {', '.join(EXPOSED[f])}")
distribution = {}
for t in TASKS:
    distribution[len(t["required"])] = distribution.get(len(t["required"]), 0) + 1
print(f"\ntasks {len(TASKS)}, required file count distribution {dict(sorted(distribution.items()))}"
      f", total required files {sum(len(t['required']) for t in TASKS)}")
print("example task:", TASKS[6]["name"], "->", "+".join(TASKS[6]["required"]),
      " tokens", assistant(TASKS[6], TASKS[6]["required"])["tokens"],
      "/ with the whole codebase", assistant(TASKS[6], FILES)["tokens"])

module      lines  exposed names
  okuma.py    13   parse_line, usage
  tarife.py   13   TIERS, ABOVE, cost
  denetim.py  11   VALID_REGIONS, is_valid, filter_valid
  ozet.py     11   region_total, region_count
  bicim.py     5   row, report

tasks 24, required file count distribution {1: 6, 2: 15, 3: 3}, total required files 45
example task: tamamlama-7 -> ozet.py+okuma.py+tarife.py  tokens 118 / with the whole codebase 170

More than half of the tasks ask for two files, six ask for one, three ask for three. The total need of the twenty-four tasks is 45 files. Sending the whole codebase to every task, on the other hand, means 120 files, which is more than twice what is needed. In the example task the difference looks small on its own: 118 tokens against 170. Over twenty-four tasks the same difference accumulates.

Enlarging versus Selecting

The two curves are printed in the same table. The batch rows give everyone the same first k files; the selected rows give each task only its own required files, and irrelevant files are added on top in turn.

def measure(select, seed=SEED):
    accepted = tokens = files = 0
    score = 0.0
    for t in TASKS:
        ctx = select(t)
        s = assistant(t, ctx, seed)
        accepted += s["accepted"]
        score += s["score"]
        tokens += s["tokens"]
        files += len(ctx)
    return accepted, score / len(TASKS), tokens, files


def show(name, select):
    a = measure(select)
    acc_b = measure(select, SEED + 1)[0]
    print(f"{name:<26}{a[3]:>7}{a[1]:>11.4f}{a[0]:>8}/24{acc_b:>8}/24{a[2]:>10}")


print("setup                      files   expected accept A accept B    tokens")
for k in range(6):
    show(f"batch {k}", lambda t, b=FILES[:k]: b)
show("selected (required only)", lambda t: t["required"])
for n in (1, 2, 3):
    def select(t, n=n):
        extra = tuple(f for f in FILES if f not in t["required"])[:n]
        return tuple(t["required"]) + extra
    show(f"selected + {n} irrelevant", select)

print("\nprovider hit rate  found required share  expected accept A accept B    tokens")
for h in (0.00, 0.25, 0.50, 0.75, 1.00):
    def select(t, i=h):
        return provider(t, i)
    a = measure(select)
    acc_b = measure(select, SEED + 1)[0]
    found = sum(sum(1 for f in t["required"] if f in select(t)) for t in TASKS)
    print(f"{h:>13.2f}{found / 45:>22.4f}{a[1]:>10.4f}{a[0]:>8}/24{acc_b:>8}/24{a[2]:>10}")

setup                      files   expected accept A accept B    tokens
batch 0                         0     0.3383       7/24       7/24      1104
batch 1                        24     0.4600      10/24       8/24      1728
batch 2                        48     0.5402      12/24      13/24      2352
batch 3                        72     0.6013      15/24      13/24      2976
batch 4                        96     0.6608      15/24      15/24      3600
batch 5                       120     0.6921      18/24      20/24      4224
selected (required only)       45     0.8233      19/24      19/24      2274
selected + 1 irrelevant        69     0.7833      19/24      18/24      2898
selected + 2 irrelevant        93     0.7433      20/24      18/24      3522
selected + 3 irrelevant       114     0.7029      17/24      20/24      4068

provider hit rate  found required share  expected accept A accept B    tokens
         0.00                0.4667    0.4960      13/24      10/24      2274
         0.25                0.6222    0.5796      15/24      12/24      2274
         0.50                0.7333    0.6404      15/24      14/24      2274
         0.75                0.8222    0.7050      17/24      15/24      2274
         1.00                1.0000    0.8233      19/24      19/24      2274

Reading the Curve

The batch rows confirm intuition: adding files raises expected acceptance from 0.3383 to 0.6921 and never lowers it at any step. The cost is also regular — every step is exactly 624 tokens, because one more file enters each of the twenty-four tasks. The curve is the number behind the claim that enlarging the context works, and the claim is true.

The curve has a weakness that does not show in the table: order. The batch rows add the files in the order of the module tuple, so which file the second row carries is not a decision but an accident. With a different order, a different curve would emerge between the same two endpoints. The selected setup has no such dependence, because it never asks the question of order. If the number of a setup is sensitive to an arbitrary decision that is not listed, that number should not be read on its own.

The second weakness is on the cost side and it is in favor of selection. In this measurement every file costs 26 tokens regardless of its size; yet the longest module of the codebase is more than two and a half times longer than the shortest. In a real context, sending in batch also pays for the cost of the long files, while selected sending often does not. So the difference below is a lower bound.

The selected row displaces the same claim. When only the required files are sent, expected acceptance is 0.8233; 0.1312 higher than the 0.6921 of the whole codebase. It does this with 45 files and 2274 tokens: a little more than a third of the file volume, 54% of the tokens. A setup that wins on both axes at once is rarely seen, and the reason here is clear: sending the whole codebase also sends the required files, but alongside them it carries on average three irrelevant files.

The comparison at equal cost makes this sharper. The batch 2 setup spends 2352 tokens and gives 0.5402; the selected setup gives 0.8233 with 2274 tokens, that is, slightly fewer. A difference of 0.2831 at almost the same budget. If the token budget is a design constraint, where you spend that budget determines more than how much you spend.

The counted acceptance columns carry a warning. The selected row is 19/24 and the whole codebase is 18/24; the difference is a single task. But in the second run the ranking reverses: 19/24 against 20/24. By counted acceptance these two setups are indistinguishable, because the difference stays within the difference between two runs. What makes the distinction is the run-independent column. This is a direct application of the course’s measure rule: a number that moves between two runs does not support a decision when it is read from a single run.

An Irrelevant File Has No Threshold

The last three selected rows answer a single question: after the right files have already been sent, what does one more file do. The answer is regular and one-directional. Expected acceptance falls from 0.8233 to 0.7833, to 0.7433, and to 0.7029; every irrelevant file takes exactly 0.0400. At the same time the cost rises from 2274 to 4068; 624, 624, and 546 tokens per step.

The “point where the hit rate falls” being looked for here is not a threshold: it starts with the first irrelevant file. There is no free irrelevant file, because the irrelevance term is linear and the overlap term is already saturated. Adding three irrelevant files increases the tokens by 79% and takes 0.1204 from expected acceptance — this is the plainest row in which the added machinery makes the work worse.

Counted acceptance is misleading here too, and it can be seen why: it goes 19, 19, 20, 17, and in the second run 19, 18, 18, 20. Someone looking at the first run could say “two irrelevant files raised acceptance”. The expected acceptance column closes that reading. In a set of twenty-four tasks the weight of a single acceptance is 0.0417, that is, larger than the real effect of an irrelevant file; counted acceptance does not have the resolution to show this detail and requires a larger set of tasks.

The Provider’s Hit Rate Is Independent of Cost

The second table turns context selection into a capability. At every hit rate the provider sends the same number of files, so the token column is 2274 in all five rows. The only thing that changes is which files are sent, and expected acceptance moves between 0.4960 and 0.8233. Context quality is not bought with tokens here; the measured difference comes entirely from the selection itself.

The last row of this table is the same as the selected row of the previous table, and it has to be: a hit rate of 1.00 is a provider that knows the required set in every task. Such a provider is a measurement fiction, not a setup. Knowing which files are required is information very close to the task having been solved; a real provider guesses it from name similarity, call relations, or past changes. This is why 0.8233 is not a target but an upper bound, and the table shows not the cost of getting close to that bound but the cost of failing to get close.

Even at a hit rate of zero, the share of required files found is 0.4667, because when one of the two files of a two-file task is dropped, the other stays in place. So a bad provider is not zero, only weak — and an expected acceptance of 0.4960 is above sending no context at all (0.3383).

The real decision point is in the middle. Sending the whole codebase gives 0.6921. The provider cannot beat it at a 0.50 hit rate (0.6404) and beats it at a 0.75 hit rate (0.7050). So a context provider that can find the required files in about three of four tasks is better than sending everything; if it cannot, it is worse, and because it is cheap, this is hard to notice. Since the cost column is fixed, the provider’s badness does not show on the invoice, only in acceptance.

Summary

• Enlarging the context raises expected acceptance from 0.3383 to 0.6921 and every step costs exactly 624 tokens; the curve is regular and never falls at any step.
• Selecting the context gives 0.8233 on the same set of tasks; it does this with 45 files and 2274 tokens, that is, 0.1312 higher than the whole codebase and at about half the cost.
• At almost equal budget the difference is even larger: batch of two files 0.5402 with 2352 tokens, selected context 0.8233 with 2274 tokens.
• Every irrelevant file takes exactly 0.0400 from expected acceptance and adds 624 tokens; there is no threshold, it starts with the first file and loses 0.1204 at three files.
• Counted acceptance does not distinguish in either of these decisions (19/24 against 18/24, reversed in the second run); what makes the distinction is the run-independent expected acceptance column.
• The token cost of the context provider is independent of its hit rate and fixed at 2274; for it to beat sending the whole codebase, it has to find the required files in about three of four tasks.

Next Step

In these two lessons every task was a single call: the context went, the suggestion came, the developer accepted it or did not. The decision point was single and right there. A development job often does not stand like that — reading a file, changing a function, updating the callers, and reformatting the report, when delegated with a single request, produce a multi-step job, and the developer no longer looks at every step but at the place where they stop. The next lesson measures this stopping place: when stopping every how many steps catches an error before it propagates, how many steps are wasted, and how many calls the stopping itself costs.