--- title: 'Context Construction' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/context-assembly' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Computes how many chunks fit into the context window and shows what the pipeline does when the limit is exceeded.' - 'Measures that as the chunk count grows retrieval recall rises while the generation hit rate falls.' - 'Counts the share of change in the answer when the same chunks are given in different orders, separating it from run-to-run variability.' - 'Reads the concept accuracy and the record type accuracy of the cited chunk separately and determines in which half the error lies.' --- # Context Construction Placing the retrieved chunks into the prompt and measuring that this placement is a mechanism in its own right: after the instruction and the query, 13.4 chunks fit into a window of two hundred forty tokens, yet at the eleventh chunk six of the forty queries, at the twelfth and the thirteenth fifteen of them, exceed the limit. When the chunk count is raised from one to ten, retrieval recall rises from 0.7750 to 0.9750 while generation falls from 0.7750 to 0.5750. When the context always holds one relevant chunk with four irrelevant chunks beside it, generation is 0.7750 with the relevant chunk at the head, 0.6250 at the tail and 0.2000 in the middle, and when the order of the same chunks changes, 0.7000 of the answers change in the middle layout; when the seed is changed in the same order, the share of change is 0.4250. The previous two lessons corrected what gets retrieved: the query was translated into the index language, and the candidates were reranked in a second stage. In both, the retrieved chunks were put into the prompt **in the order they were ranked** and **in whatever number was asked for**. This lesson measures that step, because constructing the context is also a mechanism, and it has two constraints. The first is the **limit**: the context window is fixed, and when the chunk count exceeds it the request returns with no answer. The second is the **order**: when the same chunks are placed in different positions in the same prompt, the chunk the model looks at changes. In this lesson every answer also **cites** the code of the chunk it used; grounding makes it readable in which half the error lies. - **RP54.** The code is a **simulator**; no real embedding or model endpoint is called. The corpus, the concepts and the queries are **fictional**, seed **20260218**. - **RP55.** The target definition and the pipeline continue from `retrieval-pipeline/05`. Transformation and reranking are **closed**; the only setup measured is the construction of the context. - **RP56.** The context window is **240 tokens**. The window was defined in the Introduction to AI Engineering course and is not repeated; here it is used as a **constraint**. - **RP57.** Every chunk enters the context **with its code**, and the answer cites the code of the chunk it used. Because the simulator builds the answer from the chunk it picked, **answer faithfulness is complete by definition**; what is measured is **the accuracy of the citation**. - **RP58.** A request that exceeds the window returns with no answer, and that query is counted **wrong**. - **RP59.** The **position weight** of the generation simulator is modeled: the first chunk 1.0, the last chunk 0.8, those in between 0.45. What this lesson measures is the consequence of this weight. - **RP60.** In the controlled experiment the context **always** holds one relevant chunk, and j irrelevant chunks are placed beside it; this fixes retrieval and measures only the context. - **RP61. The share of change** is the share with which **the cited chunk** changes when the same chunks are given in a different order. The comparison baseline is the run in which the order is fixed and the seed changes. ## Context Budget The budget of a prompt consists of three items: the instruction, the query and the chunks. The first two are fixed, and the third grows with k. The block below sets up the pipeline, derives the budget, and prints retrieval and generation side by side while raising k from one to thirteen. ```python # SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus, # concepts and queries are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def text_of(d): return d["title"] + " " + " ".join(d["paras"]) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- pipeline: chunks, hybrid retrieval, target definition and generation CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]] for p in CHUNKS: p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"])) for s in QUERIES: s["record"] = s["text"].split()[-1] def retrieve(q, k=5, w=0.5): qv, qb = embed(q), set(tokenize(q)) s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i) for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1])) return [CHUNKS[i] for _, i in s[:k]] def relevant(s, p): return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h WINDOW = 240 COUNTER = [0] ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . " def answer(query, chunks, seed=SEED, temperature=0.12): """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight.""" COUNTER[0] += 1 prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None} r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks) score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb)) for i, x in enumerate(chunks)] w = [math.exp(a / temperature) for a in score] chosen = chunks[pick(r(), [a / sum(w) for a in w])] return {"status": "ok", "input": len(toks), "chunk": chosen, "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))} def build_prompt(query, chunks): return (ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query) per_chunk = sum(len(tokenize(f"[{p['code']}] {p['text']} .")) for p in CHUNKS) / len(CHUNKS) per_query = sum(len(tokenize(s["text"])) for s in QUERIES) / 40 print("window %d ; instruction %d + query %.1f + per chunk %.1f tokens -> chunks that fit %.1f" % (WINDOW, len(tokenize(ANSWER_INSTRUCTION)), per_query, per_chunk, (WINDOW - len(tokenize(ANSWER_INSTRUCTION)) - per_query) / per_chunk)) print(" k context avg largest overflow recall@k generation cited concept cited record") for k in (1, 3, 5, 8, 10, 11, 12, 13): lens, rec, gen, over, cited_concept, cited_record = [], 0, 0, 0, 0, 0 for s in QUERIES: g = retrieve(s["text"], k) lens.append(len(tokenize(build_prompt(s["text"], g)))) rec += any(relevant(s, p) for p in g) y = answer(s["text"], g) if y["chunk"] is None: over += 1 continue cited_concept += CONCEPTS[s["concept"]][1] in y["chunk"]["text"] cited_record += s["record"] in y["chunk"]["bag"] gen += relevant(s, y["chunk"]) print("%2d %11.1f %7d %8d %8.4f %10.4f %13.4f %12.4f" % (k, sum(lens) / 40, max(lens), over, rec / 40, gen / 40, cited_concept / 40, cited_record / 40)) ``` ``` window 240 ; instruction 20 + query 5.5 + per chunk 16.0 tokens -> chunks that fit 13.4 k context avg largest overflow recall@k generation cited concept cited record 1 43.6 48 0 0.7750 0.7750 1.0000 0.7750 3 76.2 88 0 0.7750 0.7000 1.0000 0.7000 5 109.0 127 0 0.8000 0.6750 1.0000 0.6750 8 157.4 186 0 0.8500 0.5500 1.0000 0.5500 10 189.9 226 0 0.9750 0.5750 1.0000 0.5750 11 206.4 246 6 0.9750 0.5500 0.8500 0.5500 12 222.7 266 15 0.9750 0.4500 0.6250 0.4500 13 238.9 286 15 0.9750 0.4500 0.6250 0.4500 ``` ## Recall Rises While Generation Falls The budget calculation says **13.4 chunks** on average, and the table shows that the limit arrives earlier than that: up to ten chunks no request exceeds the limit; at eleven **six** of the forty queries do, at twelve **fifteen**, and at thirteen **fifteen** again. The average context at the thirteenth chunk is 238.9 tokens, and the largest is 286. The limit collides not with the average but with **the largest**; even when the average is 206.4, six requests return with no answer. The real finding is that the two columns go in **opposite directions**. When k goes from one to ten, retrieval recall rises **from 0.7750 to 0.9750**, while generation falls **from 0.7750 to 0.5750**. A plan that looks at the retrieval measure enlarges k and thinks it has improved; the generation measure says the opposite. Why the rule of the course asks for two columns can be seen in these rows. The last two columns say where the error is. The **cited concept** share stays at 1.0000 until k reaches ten: the model almost always cites a chunk that describes the correct concept. The **cited record** share, on the other hand, falls to 0.5750 together with generation. That is, the error is not in the concept but in the **record type**. When the limit is exceeded both fall at once for the overflowing requests, because there is no answer at all. What makes this distinction readable is the citation itself; if the answer were only text, the two kinds of error would stay inside a single number. ## The Share of the Order The k sweep changes two things together: how many chunks are in the context and where the relevant chunk stands. The block below isolates the second. In every query **one relevant chunk** is put into the context, j irrelevant chunks are added beside it, and the same set is given in three different orders. The last column prints, as the baseline, the run in which the order is fixed and only the seed changes. ```python def experiment(j, seed=SEED): """The context always holds one relevant chunk; j irrelevant chunks are placed beside it in three different layouts.""" hits, diff, seed_diff = {"head": 0, "middle": 0, "tail": 0}, {"middle": 0, "tail": 0}, 0 for s in QUERIES: candidates = retrieve(s["text"], 40) good = next(p for p in candidates if relevant(s, p)) bad = [p for p in candidates if not relevant(s, p)][:j] layout = {"head": [good] + bad, "tail": bad + [good], "middle": bad[:j // 2] + [good] + bad[j // 2:]} picked = {} for a in ("head", "middle", "tail"): picked[a] = answer(s["text"], layout[a], seed)["chunk"] hits[a] += picked[a] is not None and relevant(s, picked[a]) for a in ("middle", "tail"): diff[a] += picked[a] is not picked["head"] seed_diff += answer(s["text"], layout["head"], seed + 1)["chunk"] is not picked["head"] return [hits[a] / 40 for a in ("head", "middle", "tail")], diff["middle"] / 40, diff["tail"] / 40, seed_diff / 40 print("irrelevant generation: head middle tail | share of change: middle tail run-to-run") for j in (0, 2, 4, 8): u, dm, dt, sd = experiment(j) print("%10d %.4f %.4f %.4f | %.4f %.4f %10.4f" % (j, u[0], u[1], u[2], dm, dt, sd)) print("second run (seed 20260219), j=4: head %.4f middle %.4f tail %.4f" % tuple(experiment(4, SEED + 1)[0])) ``` ``` irrelevant generation: head middle tail | share of change: middle tail run-to-run 0 1.0000 1.0000 1.0000 | 0.0000 0.0000 0.0000 2 0.8250 0.2250 0.5000 | 0.7000 0.4500 0.3750 4 0.7750 0.2000 0.6250 | 0.7000 0.5250 0.4250 8 0.6500 0.1500 0.6000 | 0.8500 0.5500 0.6000 second run (seed 20260219), j=4: head 0.7500 middle 0.1500 tail 0.5750 ``` ## The Price of an Irrelevant Chunk The first row sets up the control: when the context holds only the relevant chunk, generation is **1.0000** and the order changes nothing. The drop in every later row is the price of the irrelevant chunks that were added. With the relevant chunk at the **head**, eight irrelevant chunks take generation from 1.0000 down to **0.6500**. With the same chunk in the **middle** it goes from 1.0000 down to **0.1500**. The drop has no threshold: even the first two irrelevant chunks take generation down to 0.8250 at the head and **0.2250** in the middle. This is the counterpart, in the retrieval context, of the finding measured in `ai-assisted-development/02`, and the added information here is this: the size of the price depends on **where the relevant chunk stands**. The row with four irrelevant chunks makes the order readable on its own. The same five chunks, the same seed, only the order different: generation is **0.7750 at the head**, **0.6250 at the tail**, **0.2000 in the middle**. The head and the middle are far apart. Whether this difference is real is tested with the last column: in the run where the order is fixed and only the seed changes, **0.4250** of the answers change, while when the order changes **0.7000** of them do in the middle layout. The numbers of the second run in the same row (0.7500 / 0.1500 / 0.5750) give the same ordering too, and the gap between the head and the middle stays as large in both runs; no layout moves by more than 0.0500 from one run to the other. The rule that follows from this is: **the order of the context is a design decision, and without being written down the measurement cannot be repeated.** The same retrieval, the same chunks and the same model swing between 0.7750 and 0.2000 in generation when the arrangement of the chunks changes. The precision that reranking bought with 1000 calls in `retrieval-pipeline/06` is spent if the chunks are put into the context in a bad order. ## Summary - After the instruction and the query, 13.4 chunks fit into a window of two hundred forty tokens; at the eleventh chunk six of the forty queries, at the twelfth and the thirteenth fifteen of them, exceed the limit and return with no answer. - The limit collides not with the average context but with the largest: even when the average is 206.4 tokens, six requests exceed the window. - When the chunk count goes from one to ten, retrieval recall rises from 0.7750 to 0.9750 while generation falls from 0.7750 to 0.5750; the two measures go in opposite directions. - The cited concept share stays at 1.0000 while the cited record share falls to 0.5750: the error is not in the concept but in the record type, and what makes this distinction readable is grounding. - With the relevant chunk at the head eight irrelevant chunks take generation from 1.0000 to 0.6500, and in the middle to 0.1500, and the drop has no threshold. - When the same five chunks are given in a different order, generation swings between 0.7750 and 0.2000 and 0.7000 of the answers change in the middle layout; when the seed changes with the order fixed, the share of change is 0.4250. ## Next Step In seven lessons the pipeline was built from end to end: the corpus was prepared, split, indexed, refreshed, the query was transformed, the candidates were reranked and the context was constructed. Each lesson defended its own mechanism with its own number, but all of these numbers carry the same flaw: a setup being **good** does not mean that the correct chunk arriving and the correct answer being produced happen **in the same query**. The table of this lesson has a row where the cited concept share is 1.0000 while generation is 0.5500; there the model pointed at the right source and read the wrong record. The next lesson counts these four cases separately and shows which one a single number hides.