Context Construction

The previous two lessons corrected what gets retrieved: the query was translated into the index language, and the candidates were reranked in a second stage. In both, the retrieved chunks were put into the prompt in the order they were ranked and in whatever number was asked for. This lesson measures that step, because constructing the context is also a mechanism, and it has two constraints.

The first is the limit: the context window is fixed, and when the chunk count exceeds it the request returns with no answer. The second is the order: when the same chunks are placed in different positions in the same prompt, the chunk the model looks at changes. In this lesson every answer also cites the code of the chunk it used; grounding makes it readable in which half the error lies.

• RP54. The code is a simulator; no real embedding or model endpoint is called. The corpus, the concepts and the queries are fictional, seed 20260218.
• RP55. The target definition and the pipeline continue from retrieval-pipeline/05. Transformation and reranking are closed; the only setup measured is the construction of the context.
• RP56. The context window is 240 tokens. The window was defined in the Introduction to AI Engineering course and is not repeated; here it is used as a constraint.
• RP57. Every chunk enters the context with its code, and the answer cites the code of the chunk it used. Because the simulator builds the answer from the chunk it picked, answer faithfulness is complete by definition; what is measured is the accuracy of the citation.
• RP58. A request that exceeds the window returns with no answer, and that query is counted wrong.
• RP59. The position weight of the generation simulator is modeled: the first chunk 1.0, the last chunk 0.8, those in between 0.45. What this lesson measures is the consequence of this weight.
• RP60. In the controlled experiment the context always holds one relevant chunk, and j irrelevant chunks are placed beside it; this fixes retrieval and measures only the context.
• RP61. The share of change is the share with which the cited chunk changes when the same chunks are given in a different order. The comparison baseline is the run in which the order is fixed and the seed changes.

Context Budget

The budget of a prompt consists of three items: the instruction, the query and the chunks. The first two are fixed, and the third grows with k. The block below sets up the pipeline, derives the budget, and prints retrieval and generation side by side while raising k from one to thirteen.

# SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus,
# concepts and queries are fictional, seed 20260218.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


def text_of(d):
    return d["title"] + " " + " ".join(d["paras"])


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0      # the out-of-domain embedding does not recognize the query language
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# ---- pipeline: chunks, hybrid retrieval, target definition and generation
CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]]
for p in CHUNKS:
    p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"]))
for s in QUERIES:
    s["record"] = s["text"].split()[-1]


def retrieve(q, k=5, w=0.5):
    qv, qb = embed(q), set(tokenize(q))
    s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i)
                for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1]))
    return [CHUNKS[i] for _, i in s[:k]]


def relevant(s, p):
    return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"]


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


WINDOW = 240
COUNTER = [0]
ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . "


def answer(query, chunks, seed=SEED, temperature=0.12):
    """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight."""
    COUNTER[0] += 1
    prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query
    toks = tokenize(prompt)
    if len(toks) > WINDOW:
        return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None}
    r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks)
    score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb))
             for i, x in enumerate(chunks)]
    w = [math.exp(a / temperature) for a in score]
    chosen = chunks[pick(r(), [a / sum(w) for a in w])]
    return {"status": "ok", "input": len(toks), "chunk": chosen,
            "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))}


def build_prompt(query, chunks):
    return (ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks)
            + " Question: " + query)


per_chunk = sum(len(tokenize(f"[{p['code']}] {p['text']} .")) for p in CHUNKS) / len(CHUNKS)
per_query = sum(len(tokenize(s["text"])) for s in QUERIES) / 40
print("window %d ; instruction %d + query %.1f + per chunk %.1f tokens -> chunks that fit %.1f"
      % (WINDOW, len(tokenize(ANSWER_INSTRUCTION)), per_query, per_chunk,
         (WINDOW - len(tokenize(ANSWER_INSTRUCTION)) - per_query) / per_chunk))
print(" k  context avg  largest  overflow  recall@k  generation  cited concept  cited record")
for k in (1, 3, 5, 8, 10, 11, 12, 13):
    lens, rec, gen, over, cited_concept, cited_record = [], 0, 0, 0, 0, 0
    for s in QUERIES:
        g = retrieve(s["text"], k)
        lens.append(len(tokenize(build_prompt(s["text"], g))))
        rec += any(relevant(s, p) for p in g)
        y = answer(s["text"], g)
        if y["chunk"] is None:
            over += 1
            continue
        cited_concept += CONCEPTS[s["concept"]][1] in y["chunk"]["text"]
        cited_record += s["record"] in y["chunk"]["bag"]
        gen += relevant(s, y["chunk"])
    print("%2d  %11.1f  %7d  %8d  %8.4f  %10.4f  %13.4f  %12.4f"
          % (k, sum(lens) / 40, max(lens), over, rec / 40, gen / 40,
             cited_concept / 40, cited_record / 40))

window 240 ; instruction 20 + query 5.5 + per chunk 16.0 tokens -> chunks that fit 13.4
 k  context avg  largest  overflow  recall@k  generation  cited concept  cited record
 1         43.6       48         0    0.7750      0.7750         1.0000        0.7750
 3         76.2       88         0    0.7750      0.7000         1.0000        0.7000
 5        109.0      127         0    0.8000      0.6750         1.0000        0.6750
 8        157.4      186         0    0.8500      0.5500         1.0000        0.5500
10        189.9      226         0    0.9750      0.5750         1.0000        0.5750
11        206.4      246         6    0.9750      0.5500         0.8500        0.5500
12        222.7      266        15    0.9750      0.4500         0.6250        0.4500
13        238.9      286        15    0.9750      0.4500         0.6250        0.4500

Recall Rises While Generation Falls

The budget calculation says 13.4 chunks on average, and the table shows that the limit arrives earlier than that: up to ten chunks no request exceeds the limit; at eleven six of the forty queries do, at twelve fifteen, and at thirteen fifteen again. The average context at the thirteenth chunk is 238.9 tokens, and the largest is 286. The limit collides not with the average but with the largest; even when the average is 206.4, six requests return with no answer.

The real finding is that the two columns go in opposite directions. When k goes from one to ten, retrieval recall rises from 0.7750 to 0.9750, while generation falls from 0.7750 to 0.5750. A plan that looks at the retrieval measure enlarges k and thinks it has improved; the generation measure says the opposite. Why the rule of the course asks for two columns can be seen in these rows.

The last two columns say where the error is. The cited concept share stays at 1.0000 until k reaches ten: the model almost always cites a chunk that describes the correct concept. The cited record share, on the other hand, falls to 0.5750 together with generation. That is, the error is not in the concept but in the record type. When the limit is exceeded both fall at once for the overflowing requests, because there is no answer at all. What makes this distinction readable is the citation itself; if the answer were only text, the two kinds of error would stay inside a single number.

The Share of the Order

The k sweep changes two things together: how many chunks are in the context and where the relevant chunk stands. The block below isolates the second. In every query one relevant chunk is put into the context, j irrelevant chunks are added beside it, and the same set is given in three different orders. The last column prints, as the baseline, the run in which the order is fixed and only the seed changes.

def experiment(j, seed=SEED):
    """The context always holds one relevant chunk; j irrelevant chunks are placed beside it in three different layouts."""
    hits, diff, seed_diff = {"head": 0, "middle": 0, "tail": 0}, {"middle": 0, "tail": 0}, 0
    for s in QUERIES:
        candidates = retrieve(s["text"], 40)
        good = next(p for p in candidates if relevant(s, p))
        bad = [p for p in candidates if not relevant(s, p)][:j]
        layout = {"head": [good] + bad, "tail": bad + [good],
                  "middle": bad[:j // 2] + [good] + bad[j // 2:]}
        picked = {}
        for a in ("head", "middle", "tail"):
            picked[a] = answer(s["text"], layout[a], seed)["chunk"]
            hits[a] += picked[a] is not None and relevant(s, picked[a])
        for a in ("middle", "tail"):
            diff[a] += picked[a] is not picked["head"]
        seed_diff += answer(s["text"], layout["head"], seed + 1)["chunk"] is not picked["head"]
    return [hits[a] / 40 for a in ("head", "middle", "tail")], diff["middle"] / 40, diff["tail"] / 40, seed_diff / 40


print("irrelevant  generation:   head  middle    tail | share of change: middle    tail  run-to-run")
for j in (0, 2, 4, 8):
    u, dm, dt, sd = experiment(j)
    print("%10d              %.4f  %.4f  %.4f |                  %.4f  %.4f  %10.4f"
          % (j, u[0], u[1], u[2], dm, dt, sd))
print("second run (seed 20260219), j=4: head %.4f middle %.4f tail %.4f"
      % tuple(experiment(4, SEED + 1)[0]))

irrelevant  generation:   head  middle    tail | share of change: middle    tail  run-to-run
         0              1.0000  1.0000  1.0000 |                  0.0000  0.0000      0.0000
         2              0.8250  0.2250  0.5000 |                  0.7000  0.4500      0.3750
         4              0.7750  0.2000  0.6250 |                  0.7000  0.5250      0.4250
         8              0.6500  0.1500  0.6000 |                  0.8500  0.5500      0.6000
second run (seed 20260219), j=4: head 0.7500 middle 0.1500 tail 0.5750

The Price of an Irrelevant Chunk

The first row sets up the control: when the context holds only the relevant chunk, generation is 1.0000 and the order changes nothing. The drop in every later row is the price of the irrelevant chunks that were added.

With the relevant chunk at the head, eight irrelevant chunks take generation from 1.0000 down to 0.6500. With the same chunk in the middle it goes from 1.0000 down to 0.1500. The drop has no threshold: even the first two irrelevant chunks take generation down to 0.8250 at the head and 0.2250 in the middle. This is the counterpart, in the retrieval context, of the finding measured in ai-assisted-development/02, and the added information here is this: the size of the price depends on where the relevant chunk stands.

The row with four irrelevant chunks makes the order readable on its own. The same five chunks, the same seed, only the order different: generation is 0.7750 at the head, 0.6250 at the tail, 0.2000 in the middle. The head and the middle are far apart. Whether this difference is real is tested with the last column: in the run where the order is fixed and only the seed changes, 0.4250 of the answers change, while when the order changes 0.7000 of them do in the middle layout. The numbers of the second run in the same row (0.7500 / 0.1500 / 0.5750) give the same ordering too, and the gap between the head and the middle stays as large in both runs; no layout moves by more than 0.0500 from one run to the other.

The rule that follows from this is: the order of the context is a design decision, and without being written down the measurement cannot be repeated. The same retrieval, the same chunks and the same model swing between 0.7750 and 0.2000 in generation when the arrangement of the chunks changes. The precision that reranking bought with 1000 calls in retrieval-pipeline/06 is spent if the chunks are put into the context in a bad order.

Summary

• After the instruction and the query, 13.4 chunks fit into a window of two hundred forty tokens; at the eleventh chunk six of the forty queries, at the twelfth and the thirteenth fifteen of them, exceed the limit and return with no answer.
• The limit collides not with the average context but with the largest: even when the average is 206.4 tokens, six requests exceed the window.
• When the chunk count goes from one to ten, retrieval recall rises from 0.7750 to 0.9750 while generation falls from 0.7750 to 0.5750; the two measures go in opposite directions.
• The cited concept share stays at 1.0000 while the cited record share falls to 0.5750: the error is not in the concept but in the record type, and what makes this distinction readable is grounding.
• With the relevant chunk at the head eight irrelevant chunks take generation from 1.0000 to 0.6500, and in the middle to 0.1500, and the drop has no threshold.
• When the same five chunks are given in a different order, generation swings between 0.7750 and 0.2000 and 0.7000 of the answers change in the middle layout; when the seed changes with the order fixed, the share of change is 0.4250.

Next Step

In seven lessons the pipeline was built from end to end: the corpus was prepared, split, indexed, refreshed, the query was transformed, the candidates were reranked and the context was constructed. Each lesson defended its own mechanism with its own number, but all of these numbers carry the same flaw: a setup being good does not mean that the correct chunk arriving and the correct answer being produced happen in the same query. The table of this lesson has a row where the cited concept share is 1.0000 while generation is 0.5500; there the model pointed at the right source and read the wrong record. The next lesson counts these four cases separately and shows which one a single number hides.