--- title: 'Providing Context' source: 'https://academia.sh/en/courses/model-apis/providing-context' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:31+00:00' license: 'CC BY-SA 4.0' objectives: - 'Compares the difference between enlarging the context and selecting the context, in tokens and acceptance, on the same set of tasks.' - 'Calculates the share an irrelevant file takes from expected acceptance and the tokens it adds.' - 'Shows, by varying the hit rate of a fictional context provider, that selection quality is independent of cost.' - 'Interprets in which case counted acceptance and run-independent expected acceptance come apart.' --- # Providing Context Measuring enlarging the context against selecting the context side by side on the same fictional codebase: sending all the files raises expected acceptance to 0.6921 and spends 4224 tokens and 120 files, while sending only the required files raises it to 0.8233 and spends 2274 tokens and 45 files. Every irrelevant file takes 0.0400 from expected acceptance and adds 624 tokens; there is no threshold, it starts with the first file. The token cost of the context provider is independent of its hit rate and is fixed at 2274, but expected acceptance varies between 0.4960 and 0.8233; for the provider to beat sending the whole codebase, it has to find the required files in about three of four tasks. The previous lesson gave the context three times as **fixed**: none, always the same file, or all of it. The distance between the ends was large — from seven of twenty-four tasks to eighteen — but none of the three was a choice. At a developer's desk the context is rebuilt for every task, and two separate questions are asked: **which files** and **how many**. This lesson separates those two questions. First the context is **enlarged**: the same set of files is given to everyone and its size grows one by one. Then the context is **selected**: each task receives only the files that task requires. The two curves are placed side by side on the token axis, and a third number is added on top: **what a file irrelevant to the task does to acceptance**. - **AD9.** The codebase is the same **fictional** codebase built with its bodies in the previous lesson: five modules, **53 lines**. Only its structure is needed here, because the context is selected **by file name**. - **AD10.** The set of tasks and the assistant simulator are also the same: twenty-four fictional tasks, three classes, seed **20260218**, no `random`. - **AD11.** The effect of context on acceptance has two terms: **overlap** (how many of the required files are in the context) raises acceptance, **irrelevance** (how many of the files in the context are unnecessary) lowers it. - **AD12.** The cost is a flat **26 tokens** per file; file size does not enter the cost. In a real context a large file is expensive, so the cost curve here is **optimistic**. - **AD13.** The number of files sent is counted separately; it is a total for the twenty-four tasks and a volume measure independent of tokens. - **AD14. The context provider is fictional** and has a single setting: the **hit rate**, that is, the share of tasks in which it finds the required set correctly. When it misses, it drops one required file and puts an irrelevant file in its place; **the file count does not change**, only which file it is. - **AD15.** The provider's own draw is **fixed by the seed**; in the second run only the uncertainty of the assistant changes. This way the difference between the two runs is attributed only to the model, not to the selection. - **AD16.** The run-independent measure is again **expected acceptance**; the counted acceptance is printed separately for two seeds. The latency class is **single call** in all setups, one per task. ## The Codebase Structure and the Provider For context selection, what is needed is not the bodies of the codebase but its **structure**: which modules exist, how long they are, and which names they expose to the outside. This is also the plane a context provider works on. ```python # THE SAME FICTIONAL CODEBASE -- five modules, 53 lines; the bodies were built in the previous lesson. # Only its structure is needed here: size and exposed names. SIZE = {"okuma.py": 13, "tarife.py": 13, "denetim.py": 11, "ozet.py": 11, "bicim.py": 5} EXPOSED = { "okuma.py": ("parse_line", "usage"), "tarife.py": ("TIERS", "ABOVE", "cost"), "denetim.py": ("VALID_REGIONS", "is_valid", "filter_valid"), "ozet.py": ("region_total", "region_count"), "bicim.py": ("row", "report"), } FILES = tuple(SIZE) SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def mix(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CLASSES = ("tamamlama", "aciklama", "donusturme") REQUIRED = { 0: ("okuma.py",), 1: ("tarife.py",), 2: ("denetim.py", "okuma.py"), 3: ("ozet.py", "denetim.py"), 4: ("bicim.py", "ozet.py"), 5: ("tarife.py", "okuma.py"), 6: ("ozet.py", "okuma.py", "tarife.py"), 7: ("denetim.py", "bicim.py"), } TASKS = [ # 24 FICTIONAL tasks: 3 classes x 8 {"name": c + "-" + str(i + 1), "label": c, "required": REQUIRED[i], "difficulty": 0.05 * (i % 4)} for c in CLASSES for i in range(8) ] BASE = {"tamamlama": 0.34, "aciklama": 0.72, "donusturme": 0.18} def assistant(task, context, seed=SEED): # SIMULATOR -- FICTIONAL r = rng(seed ^ mix(task["name"] + "|" + ",".join(sorted(context)))) overlap = sum(1 for f in task["required"] if f in context) / len(task["required"]) irrelevant = sum(1 for f in context if f not in task["required"]) / len(FILES) score = BASE[task["label"]] + 0.55 * overlap - 0.30 * irrelevant - task["difficulty"] score = min(max(score, 0.0), 1.0) accepted = r() < score tokens = 40 + 26 * len(context) + (18 if task["label"] == "donusturme" else 0) return {"accepted": accepted, "score": score, "tokens": tokens} def provider(task, hit, seed=SEED): """FICTIONAL context provider. When it hits, it finds the required set; when it misses, it drops one required file and puts an irrelevant file in its place. The file count stays the same.""" r = rng(seed ^ mix("saglayici|" + task["name"])) if r() < hit: return tuple(task["required"]) extra = tuple(f for f in FILES if f not in task["required"])[:1] return tuple(task["required"][1:]) + extra print("module lines exposed names") for f in FILES: print(f" {f:<10}{SIZE[f]:>4} {', '.join(EXPOSED[f])}") distribution = {} for t in TASKS: distribution[len(t["required"])] = distribution.get(len(t["required"]), 0) + 1 print(f"\ntasks {len(TASKS)}, required file count distribution {dict(sorted(distribution.items()))}" f", total required files {sum(len(t['required']) for t in TASKS)}") print("example task:", TASKS[6]["name"], "->", "+".join(TASKS[6]["required"]), " tokens", assistant(TASKS[6], TASKS[6]["required"])["tokens"], "/ with the whole codebase", assistant(TASKS[6], FILES)["tokens"]) ``` ``` module lines exposed names okuma.py 13 parse_line, usage tarife.py 13 TIERS, ABOVE, cost denetim.py 11 VALID_REGIONS, is_valid, filter_valid ozet.py 11 region_total, region_count bicim.py 5 row, report tasks 24, required file count distribution {1: 6, 2: 15, 3: 3}, total required files 45 example task: tamamlama-7 -> ozet.py+okuma.py+tarife.py tokens 118 / with the whole codebase 170 ``` More than half of the tasks ask for **two** files, six ask for one, three ask for three. The total need of the twenty-four tasks is **45 files**. Sending the whole codebase to every task, on the other hand, means 120 files, which is **more than twice** what is needed. In the example task the difference looks small on its own: 118 tokens against 170. Over twenty-four tasks the same difference accumulates. ## Enlarging versus Selecting The two curves are printed in the same table. The **batch** rows give everyone the same first `k` files; the **selected** rows give each task only its own required files, and irrelevant files are added on top in turn. ```python def measure(select, seed=SEED): accepted = tokens = files = 0 score = 0.0 for t in TASKS: ctx = select(t) s = assistant(t, ctx, seed) accepted += s["accepted"] score += s["score"] tokens += s["tokens"] files += len(ctx) return accepted, score / len(TASKS), tokens, files def show(name, select): a = measure(select) acc_b = measure(select, SEED + 1)[0] print(f"{name:<26}{a[3]:>7}{a[1]:>11.4f}{a[0]:>8}/24{acc_b:>8}/24{a[2]:>10}") print("setup files expected accept A accept B tokens") for k in range(6): show(f"batch {k}", lambda t, b=FILES[:k]: b) show("selected (required only)", lambda t: t["required"]) for n in (1, 2, 3): def select(t, n=n): extra = tuple(f for f in FILES if f not in t["required"])[:n] return tuple(t["required"]) + extra show(f"selected + {n} irrelevant", select) print("\nprovider hit rate found required share expected accept A accept B tokens") for h in (0.00, 0.25, 0.50, 0.75, 1.00): def select(t, i=h): return provider(t, i) a = measure(select) acc_b = measure(select, SEED + 1)[0] found = sum(sum(1 for f in t["required"] if f in select(t)) for t in TASKS) print(f"{h:>13.2f}{found / 45:>22.4f}{a[1]:>10.4f}{a[0]:>8}/24{acc_b:>8}/24{a[2]:>10}") ``` ``` setup files expected accept A accept B tokens batch 0 0 0.3383 7/24 7/24 1104 batch 1 24 0.4600 10/24 8/24 1728 batch 2 48 0.5402 12/24 13/24 2352 batch 3 72 0.6013 15/24 13/24 2976 batch 4 96 0.6608 15/24 15/24 3600 batch 5 120 0.6921 18/24 20/24 4224 selected (required only) 45 0.8233 19/24 19/24 2274 selected + 1 irrelevant 69 0.7833 19/24 18/24 2898 selected + 2 irrelevant 93 0.7433 20/24 18/24 3522 selected + 3 irrelevant 114 0.7029 17/24 20/24 4068 provider hit rate found required share expected accept A accept B tokens 0.00 0.4667 0.4960 13/24 10/24 2274 0.25 0.6222 0.5796 15/24 12/24 2274 0.50 0.7333 0.6404 15/24 14/24 2274 0.75 0.8222 0.7050 17/24 15/24 2274 1.00 1.0000 0.8233 19/24 19/24 2274 ``` ## Reading the Curve The batch rows confirm intuition: adding files raises expected acceptance **from 0.3383 to 0.6921** and never lowers it at any step. The cost is also regular — every step is exactly **624 tokens**, because one more file enters each of the twenty-four tasks. The curve is the number behind the claim that enlarging the context works, and the claim is true. The curve has a weakness that does not show in the table: **order**. The batch rows add the files in the order of the module tuple, so which file the second row carries is not a decision but an accident. With a different order, a different curve would emerge between the same two endpoints. The selected setup has no such dependence, because it never asks the question of order. If the number of a setup is sensitive to an arbitrary decision that is not listed, that number should not be read on its own. The second weakness is on the cost side and it is **in favor of selection**. In this measurement every file costs 26 tokens regardless of its size; yet the longest module of the codebase is more than two and a half times longer than the shortest. In a real context, sending in batch also pays for the cost of the long files, while selected sending often does not. So the difference below is a **lower bound**. The selected row displaces the same claim. When only the required files are sent, expected acceptance is **0.8233**; **0.1312** higher than the **0.6921** of the whole codebase. It does this with **45 files** and **2274 tokens**: **a little more than a third** of the file volume, **54%** of the tokens. A setup that wins on both axes at once is rarely seen, and the reason here is clear: sending the whole codebase also sends the required files, but alongside them it carries **on average three irrelevant files**. The comparison at equal cost makes this sharper. The **batch 2** setup spends 2352 tokens and gives **0.5402**; the selected setup gives **0.8233** with 2274 tokens, that is, slightly fewer. A difference of **0.2831** at almost the same budget. If the token budget is a design constraint, where you spend that budget determines more than how much you spend. The counted acceptance columns carry a warning. The selected row is **19/24** and the whole codebase is **18/24**; the difference is a single task. But in the second run the ranking reverses: 19/24 against **20/24**. By counted acceptance these two setups are **indistinguishable**, because the difference stays within the difference between two runs. What makes the distinction is the run-independent column. This is a direct application of the course's measure rule: a number that moves between two runs does not support a decision when it is read from a single run. ## An Irrelevant File Has No Threshold The last three selected rows answer a single question: after the right files have already been sent, what does **one more file** do. The answer is regular and one-directional. Expected acceptance falls **from 0.8233 to 0.7833, to 0.7433, and to 0.7029**; every irrelevant file takes exactly **0.0400**. At the same time the cost rises **from 2274 to 4068**; 624, 624, and 546 tokens per step. The "point where the hit rate falls" being looked for here is not a threshold: it **starts with the first irrelevant file**. There is no free irrelevant file, because the irrelevance term is linear and the overlap term is already saturated. Adding three irrelevant files increases the tokens by **79%** and takes **0.1204** from expected acceptance — this is the plainest row in which the added machinery makes the work worse. Counted acceptance is misleading here too, and it can be seen why: it goes 19, 19, 20, 17, and in the second run 19, 18, 18, 20. Someone looking at the first run could say "two irrelevant files raised acceptance". The expected acceptance column closes that reading. In a set of twenty-four tasks the weight of a single acceptance is 0.0417, that is, larger than the real effect of an irrelevant file; counted acceptance does not have the resolution to show this detail and requires a larger set of tasks. ## The Provider's Hit Rate Is Independent of Cost The second table turns context selection into a capability. At every hit rate the provider sends the **same number of files**, so the token column is **2274** in all five rows. The only thing that changes is **which** files are sent, and expected acceptance moves **between 0.4960 and 0.8233**. Context quality is not bought with tokens here; the measured difference comes entirely from the selection itself. The last row of this table is the same as the selected row of the previous table, and it has to be: a **hit rate of 1.00** is a provider that knows the required set in every task. Such a provider is a measurement fiction, not a setup. Knowing which files are required is information very close to the task having been solved; a real provider **guesses** it from name similarity, call relations, or past changes. This is why 0.8233 is not a target but an **upper bound**, and the table shows not the cost of getting close to that bound but the cost of failing to get close. Even at a hit rate of zero, the share of required files found is **0.4667**, because when one of the two files of a two-file task is dropped, the other stays in place. So a bad provider is not zero, only weak — and an expected acceptance of 0.4960 is above sending no context at all (0.3383). The real decision point is in the middle. Sending the whole codebase gives **0.6921**. The provider cannot beat it at a 0.50 hit rate (**0.6404**) and beats it at a 0.75 hit rate (**0.7050**). So a context provider that can find the required files in **about three of four tasks** is better than sending everything; if it cannot, it is worse, and because it is cheap, this is hard to notice. Since the cost column is fixed, the provider's badness does not show on the invoice, only in acceptance. ## Summary - Enlarging the context raises expected acceptance from 0.3383 to 0.6921 and every step costs exactly 624 tokens; the curve is regular and never falls at any step. - Selecting the context gives 0.8233 on the same set of tasks; it does this with 45 files and 2274 tokens, that is, 0.1312 higher than the whole codebase and at about half the cost. - At almost equal budget the difference is even larger: batch of two files 0.5402 with 2352 tokens, selected context 0.8233 with 2274 tokens. - Every irrelevant file takes exactly 0.0400 from expected acceptance and adds 624 tokens; there is no threshold, it starts with the first file and loses 0.1204 at three files. - Counted acceptance does not distinguish in either of these decisions (19/24 against 18/24, reversed in the second run); what makes the distinction is the run-independent expected acceptance column. - The token cost of the context provider is independent of its hit rate and fixed at 2274; for it to beat sending the whole codebase, it has to find the required files in about three of four tasks. ## Next Step In these two lessons every task was **a single call**: the context went, the suggestion came, the developer accepted it or did not. The decision point was single and right there. A development job often does not stand like that — reading a file, changing a function, updating the callers, and reformatting the report, when delegated with a single request, produce a **multi-step** job, and the developer no longer looks at every step but at the place where they stop. The next lesson measures this stopping place: when stopping every how many steps catches an error **before it propagates**, how many steps are wasted, and how many calls the stopping itself costs.