--- title: 'Context Window' source: 'https://academia.sh/en/courses/introduction-to-ai-engineering/context-window' course: 'Introduction to AI Engineering' language: en updated: '2026-10-21T05:02:30+00:00' license: 'CC BY-SA 4.0' objectives: - 'Counts what a call does when the context window is exceeded and the share of failure' - 'Compares the tokens saved and the hit rate lost by dropping from the head, dropping from the middle and summarizing on the same task' - 'Ties the hit rate of the truncation decision to the position of the evidence in the prompt and measures how the number changes as the window grows' --- # Context Window Measuring what happens when the window is exceeded and how many tokens three truncation decisions save and how much they lower the hit rate: in a window of 256 tokens, 0.5967 of 300 fictional prompts do not fit, and the bare call gives a full-response rate of 0.3833 at 282.3 tokens. Dropping from the head gives 0.7433 at 217.5 tokens, dropping from the middle gives 0.7367 at the same tokens, and summarizing raises the code hit rate to 0.9667 at 100.6 tokens but drops the step hit rate to 0.0000. Which decision truncation makes depends on the position of the evidence: dropping from the head gives 0.5351 in the head bucket, dropping from the middle gives 0.3627 in the middle bucket. When the window goes from 128 to 384, the full response rises from 0.3767 to 0.9867. The previous lesson counted how a response is produced step by step and what the selection rule costs, but it kept the input fixed at 30 tokens. In a real scaffolding the input is not fixed. The prompt grows: an instruction, a few of the field team's documents, the ticket stream and the question are put side by side. The model, on the other hand, has a fixed **context window**, and when that window is exceeded something happens. This lesson measures that "something". The window is a limit; what will truncate a prompt that exceeds the limit is a **decision**, and the decision belongs to the engineer. Three decisions are put side by side on the same task — dropping from the head, dropping from the middle, summarizing — and each is written with two numbers: how many tokens it saved, how much it lowered the hit rate. - **LM35.** The document pool and the tickets are **fictional**: they are generated inside the lesson, seed 20260218. The pool is 24 documents, each 28 tokens; the records come from no source system. - **LM36.** A document carries a header line (document number, object, status, fault code) and five procedure steps. - **LM37.** The prompt has four parts: instruction, documents, ticket, question. Truncation is done at **document integrity**; no half document is left. The instruction, the ticket and the question are never dropped in any decision. - **LM38.** The task has two fields: the **fault code** and the **second step** of the correct document. A full response is both being correct at once. - **LM39.** The model is a **simulator**. It reads the prompt as long as it fits in the window; if it does not fit, the call fails with a **window error** and no response returns. - **LM40.** The second failure of the simulator is the **timeout**, and its share grows with the prompt length. This share depends on the seed; the second run is taken by changing the seed. - **LM41.** The summarizing procedure is done here by a rule: only the header line of each document is left. In a real scaffolding the summary is a model call and has **a cost of its own**. - **LM42.** The criterion is the same on every row: tokens per prompt (smaller is better), share of failure (smaller is better), code and step hit rates and full-response rate (larger is better). - **LM43.** The **position** of the evidence in the prompt is recorded: the order of the target document in the document list. - **LM44.** In the window sweep the share of failure is not counted; that table is independent of the run. - **LM45.** The **bare call** in this lesson is sending the prompt without any truncation. ## A Prompt That Exceeds the Window The prompt starts with the instruction, continues with four to fourteen documents, and ends with the ticket and the question. The window is 256 tokens. ```python # SIMULATOR — the model is a simulator written in the lesson with the standard # library; no real endpoint is called. The document pool and the tickets are FICTIONAL, seed 20260218. SEED, M32, WINDOW, TICKETS = 20260218, 0xFFFFFFFF, 256, 300 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value OBJECTS = ["invoice", "tariff", "meter", "valve", "seal", "line"] STATUSES = ["high", "missing", "broken", "old"] ACTIONS = ["open record", "dispatch team", "repeat reading", "send information", "compare tariff", "replace seal"] INSTRUCTION = ("instruction : read the documents below , find the one matching " "this ticket , write the code and second step .").split() QUESTION = "question : what is the code and second step ?".split() DOCUMENT_POOL = [] for i, n in enumerate(OBJECTS): for j, d in enumerate(STATUSES): r = rng(SEED + 900 + i * 4 + j) b = ["document", str(101 + i * 4 + j), ":", n, d, "code", f"{'ABCDEF'[i]}{j + 1}"] for a in range(1, 6): b += ["step", str(a)] + ACTIONS[int(r() * 6)].split() DOCUMENT_POOL.append({"seq": b + ["end"], "object": n, "status": d, "code": b[6], "step2": b[13:15]}) DATA = [] for i in range(TICKETS): r = rng(SEED + i) h = int(r() * 24) target = DOCUMENT_POOL[h] others = [DOCUMENT_POOL[(h + 1 + int(r() * 23)) % 24] for _ in range(3 + int(r() * 11))] position = int(r() * (len(others) + 1)) documents = ([b["seq"] for b in others[:position]] + [target["seq"]] + [b["seq"] for b in others[position:]]) tail = ["ticket", ":", target["object"], "status", target["status"], ",", "reply", "awaited", "."] + QUESTION DATA.append({"documents": documents, "tail": tail, "position": position, "code": target["code"], "step2": target["step2"], "object": target["object"], "status": target["status"]}) def join_prompt(documents, tail): d = list(INSTRUCTION) for b in documents: d += b return d + tail lengths = [len(join_prompt(x["documents"], x["tail"])) for x in DATA] print(f"fictional document pool {len(DOCUMENT_POOL)} documents, each" f" {len(DOCUMENT_POOL[0]['seq'])} tokens") print(f"{TICKETS} prompts: {min(lengths)}-{max(lengths)} tokens, mean {sum(lengths) / TICKETS:.1f};" f" document count {min(len(x['documents']) for x in DATA)}" f"-{max(len(x['documents']) for x in DATA)}") print(f"window {WINDOW} tokens; prompts exceeding the window" f" {sum(u > WINDOW for u in lengths) / TICKETS:.4f}") print("\n" + " ".join(DOCUMENT_POOL[9]["seq"])) print(" ".join(DATA[0]["tail"])) ``` ``` fictional document pool 24 documents, each 28 tokens 300 prompts: 152-432 tokens, mean 282.3; document count 4-14 window 256 tokens; prompts exceeding the window 0.5967 document 110 : meter missing code C2 step 1 send information step 2 send information step 3 open record step 4 replace seal step 5 compare tariff end ticket : line status old , reply awaited . question : what is the code and second step ? ``` The mean prompt is 282.3 tokens and the window is 256: **0.5967 of the prompts do not fit.** This share is a property of the scaffolding, not of the model. As the number of documents goes from 4 to 14 the prompt grows from 152 tokens to 432, and which side of the limit one stays on is decided by the procedure that builds the prompt. ## Three Truncation Decisions The decisions work at document integrity. Dropping from the head removes the oldest document, dropping from the middle removes the one in the middle of the list; summarizing drops no document but leaves only the header of each. A fourth decision combines the two: summarize everything, then put the last documents back in full as far as the budget allows. ```python def drop_head(x, p): bl = list(x["documents"]) while len(bl) > 1 and len(join_prompt(bl, x["tail"])) > p: bl.pop(0) return join_prompt(bl, x["tail"]) def drop_middle(x, p): bl = list(x["documents"]) while len(bl) > 1 and len(join_prompt(bl, x["tail"])) > p: bl.pop(len(bl) // 2) return join_prompt(bl, x["tail"]) def summarize(x, p): return join_prompt([b[:7] for b in x["documents"]], x["tail"]) def summarize_keep_tail(x, p): bl = [b[:7] for b in x["documents"]] for i in range(len(bl) - 1, -1, -1): attempt = bl[:i] + [x["documents"][i]] + bl[i + 1:] if len(join_prompt(attempt, x["tail"])) > p: break bl = attempt return join_prompt(bl, x["tail"]) def answer(seq, object_name, status): code = step = None for i in range(len(seq) - 6): if (seq[i] == "document" and seq[i + 3] == object_name and seq[i + 4] == status and seq[i + 5] == "code"): code = seq[i + 6] for j in range(i + 7, len(seq) - 3): if seq[j] == "end": break if seq[j] == "step" and seq[j + 1] == "2": step = seq[j + 2:j + 4] break break return code, step def call(prompt, seed, i): """Failures of the simulator: the window overflow is certain, the timeout is a share that grows with the prompt length.""" if len(prompt) > WINDOW: return "window" return "timeout" if rng(seed + i)() < 0.02 + 0.06 * len(prompt) / WINDOW else None def measure(name, trim, seed=SEED, write=True): tokens = window_errors = timeouts = code_hits = step_hits = full = 0 for i, x in enumerate(DATA): prompt = trim(x, WINDOW) if trim else join_prompt(x["documents"], x["tail"]) tokens += len(prompt) h = call(prompt, seed, i) window_errors += h == "window" timeouts += h == "timeout" if h: continue k, a = answer(prompt, x["object"], x["status"]) code_hits += k == x["code"] step_hits += a == x["step2"] full += k == x["code"] and a == x["step2"] n = len(DATA) if write: print(f"{name:<22} {tokens / n:>8.1f} {window_errors / n:>8.4f} {timeouts / n:>7.4f}" f" {code_hits / n:>7.4f} {step_hits / n:>7.4f} {full / n:>7.4f}") return full / n print(f"{'decision':<22} {'tokens':>8} {'window':>8} {'timeout':>7}" f" {'code':>7} {'step':>7} {'full':>7}") DECISIONS = [("bare call", None), ("drop head", drop_head), ("drop middle", drop_middle), ("summarize", summarize), ("summarize, keep tail", summarize_keep_tail)] for name, f in DECISIONS: measure(name, f) ``` ``` decision tokens window timeout code step full bare call 282.3 0.5967 0.0200 0.3833 0.3833 0.3833 drop head 217.5 0.0000 0.0600 0.7433 0.7433 0.7433 drop middle 217.5 0.0000 0.0600 0.7367 0.7367 0.7367 summarize 100.6 0.0000 0.0333 0.9667 0.0000 0.0000 summarize, keep tail 221.9 0.0000 0.0633 0.9367 0.7167 0.7167 ``` The **bare call** spends 282.3 tokens and the full-response rate is 0.3833. The source of the loss is not the hit rate but **failure**: 0.5967 of the calls give no response at all because they do not fit in the window. A call that hits the limit does not produce a wrong answer, it produces no answer. **Dropping from the head** is the plainest truncation: it brings the tokens down to 217.5, zeroes the window error and raises the full response to 0.7433. The **cost** has three items — the dropped documents, an extra truncation step and the timeout share rising from 0.0200 to 0.0600. This last item goes against intuition: truncation did not shorten the average prompt, it **lengthened** it, because long prompts that previously could not be sent at all are now sent. **Summarizing** is the cheapest decision: 100.6 tokens, 0.3564 of the bare call. It raises the code hit rate to 0.9667 — because all the document headers stay, the correct code is always found. The step hit rate, however, is **0.0000** and the full-response rate is zero too. The summary has thrown away the thing that was asked. **How much a summary loses is read not from the summary itself but from the question.** The fourth decision gives two separate answers on two criteria: on code hit rate it passes dropping from the head by a wide margin, 0.9367 against its 0.7433, on full response it stays below it at 0.7167 and is 4.4 tokens more expensive. **Which one wins depends on which column is read**, and the choice of column is a product decision. ## Where Truncation Cuts The hit rate of truncation does not fit in a single number: whether the document a decision drops is accurate or destructive depends on where the sought evidence stands. ```python DECISIONS = [("drop head", drop_head), ("drop middle", drop_middle), ("summarize", summarize), ("summarize, keep tail", summarize_keep_tail)] def bucket(x): n = len(x["documents"]) return 0 if x["position"] < n / 3 else (1 if x["position"] < 2 * n / 3 else 2) print(f"{'decision':<22} {'head':>7} {'middle':>7} {'tail':>7} {'run 2':>9}") for name, f in DECISIONS: d, t = [0, 0, 0], [0, 0, 0] for i, x in enumerate(DATA): prompt = f(x, WINDOW) j = bucket(x) t[j] += 1 if call(prompt, SEED, i): continue k, a = answer(prompt, x["object"], x["status"]) d[j] += k == x["code"] and a == x["step2"] second = measure(name, f, SEED + 7000, False) print(f"{name:<22} " + " ".join(f"{a / b:>7.4f}" for a, b in zip(d, t)) + f" {second:>9.4f}") print("tickets per bucket" + "".join(f"{t[j]:>8}" for j in range(3)).rjust(28)) print(f"\n{'window':>8} {'drop head':>11} {'summarize keep':>15} {'summarize':>9}") for p in (128, 192, 256, 384): s = [] for f in (drop_head, summarize_keep_tail, summarize): hit = 0 for i, x in enumerate(DATA): prompt = f(x, p) if len(prompt) > p: continue k, a = answer(prompt, x["object"], x["status"]) hit += k == x["code"] and a == x["step2"] s.append(hit / len(DATA)) print(f"{p:>8} {s[0]:>11.4f} {s[1]:>15.4f} {s[2]:>9.4f}") ``` ``` decision head middle tail run 2 drop head 0.5351 0.8235 0.9286 0.7367 drop middle 0.9474 0.3627 0.9048 0.7000 summarize 0.0000 0.0000 0.0000 0.0000 summarize, keep tail 0.5439 0.7353 0.9286 0.7000 tickets per bucket 114 102 84 window drop head summarize keep summarize 128 0.3767 0.2400 0.0000 192 0.6367 0.6100 0.0000 256 0.8000 0.7667 0.0000 384 0.9867 0.9833 0.0000 ``` The first three columns open up the totals of the decisions, which are close to each other. Dropping from the head gives 0.5351 in the head bucket and 0.9286 in the tail bucket; dropping from the middle rises to 0.9474 in the head bucket but falls to 0.3627 in the middle bucket. **The total hit rates of the two decisions are almost the same (0.7433 and 0.7367) and their error distributions are each other's opposite.** For a product these are not the same decision: if the evidence sits in the old records, dropping from the head errs systematically, if it sits in the middle of the stream, dropping from the middle does. The last column is the second run of the same scaffolding. Only the timeout draw changed, and dropping from the head gave 0.7367 instead of 0.7433, dropping from the middle 0.7000 instead of 0.7367. **The same truncation decision, the same data, two runs, two numbers.** The difference of 0.0066 between the decisions falls below this variability: this table read from a single run would have made dropping from the head the winner, and the second run makes the two equal. The lower table leaves the share of failure out and is independent of the run. As the window goes from 128 to 384 the full response of dropping from the head rises from 0.3767 to 0.9867: the whole of the truncation decision is the effort to close the shortfall in the window, and if the window is big enough no decision is needed. The column of summarizing is 0.0000 at every window — **a loss that does not come from the window is not closed by enlarging the window.** As long as the lost information stays outside the prompt it does not come back whatever the limit is, and the question that separates these two tables is this: does the loss come from the limit, or from the decision made to meet the limit. ## Summary - The context window is a fixed limit; when it is exceeded this simulator gives no response at all. In the bare call 0.5967 of the prompts do not fit and the full-response rate stays at 0.3833. The documents and the tickets are fictional, seed 20260218. - Dropping from the head brings the prompt from 282.3 down to 217.5 tokens and raises the full response to 0.7433. Its cost is the dropped documents, an extra step and the timeout share rising from 0.0200 to 0.0600. - Summarizing is the cheapest decision (100.6 tokens) and raises the code hit rate to 0.9667, but because it throws away the step that was asked, the full-response rate is 0.0000. - The hit rate of truncation depends on the position of the evidence: dropping from the head gives 0.5351 in the head bucket, dropping from the middle gives 0.3627 in the middle bucket. Their totals are almost the same, their error distributions are opposite. - The same scaffolding gives 0.7367 instead of 0.7433 in the second run; the difference between the decisions is below this variability. When the window goes from 128 to 384 the full response rises from 0.3767 to 0.9867. ## Next Step In five lessons the surface of the model was measured: which unit it works with, why it can use distant context, what it learned from where, how it produces a response and where its limit lies. The window is a limit and how to fill it is a decision — this lesson compared three forms of that decision with numbers. But until now everything has been done under the same assumption: that if the correct information is put into the prompt, the model will use it correctly. **What the model does well** was never asked. The next lesson splits a task set into task classes and counts in which class the bare call collapses.