--- title: 'Measuring Retrieval and Generation Quality Separately' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/measuring-retrieval-and-generation-quality-separately' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures retrieval recall and retrieval precision as a column separate from generation accuracy.' - 'Counts the four combinations of right chunk and right answer and shows that a single rate hides these four.' - 'Tells apart, with a subgroup breakdown, that two setups giving the same single number fail for different reasons.' - 'Computes why the baseline of the call without retrieval can be inflated by chance and why it swings between two runs.' --- # Measuring Retrieval and Generation Quality Separately Showing that a single end-to-end accuracy rate stacks four separate cases on top of one another: on the forty-query fictional set the baseline without retrieval comes out at 0.3750, and the whole of this rate is chance, because the correct chunk is never retrieved. With a three-chunk context the single number is 0.9500, and with a five-chunk context it is also 0.9500; but with three chunks the two failures are retrieval faults, with five chunks the same two failures are generation faults, and the single number does not tell them apart. Retrieval recall is 0.9500 with three chunks and 1.0000 with five; the cost goes from 120 chunks and 1872 tokens to 200 chunks and 2932 tokens. The difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the three-chunk context. The class subgroup shows that both faults are collected in the same class. The previous lesson gave the same chunks in different orders and counted the share of change in the answer. The pipeline is now built: the document was prepared, split into chunks, indexed, the query was transformed in a single step, the candidates were reranked and the context was constructed under the limit. Each of the seven lessons defended its own step with its own number. This lesson stops where those numbers come together. When a retrieval-augmented generation pipeline is run, what is usually left is a single rate: how many questions were answered correctly. The claim of this lesson is that this single rate stacks **four separate cases** on top of one another, and that from inside the stack it cannot be read which stage is broken. The evaluation procedure itself was established in the Model Evaluation, Interpretability and Ethics course, together with its lessons on model evaluation and subgroups, and is not repeated here; the only thing taken over from there is **the baseline rule**, and in this course the baseline is **the call without retrieval**. - **RP71.** The vocabulary, the concept axis, the embedding and the query set are **fictional** and are produced inside the lesson. There is no real institution, product, document, vector store or embedding endpoint. - **RP72.** The concept has one surface in the **document language** (`tier limit`) and another surface in the **query language** (`which band am i in`), and the two share no token. - **RP73.** **The retrieval measure**: if the document surface of the target procedure appears inside the retrieved chunks, retrieval is right. This measure is independent of the seed. - **RP74.** **The generation measure**: if the procedure named in the answer is the target procedure, generation is right. The answer comes from a **language model simulator**; the simulator does not call a real endpoint. - **RP75.** The **prior knowledge of the simulator is at the class level only**. From the query language it infers which class the query belongs to and gives an equal, small base score to the four procedures of that class, but it does not know a procedure name on its own. Every procedure name that appears in the context adds a point. - **RP76.** **Cost** is written in three units: the number of retrieved chunks, the tokens entering the prompt and the retrieval step. Time is not written. ## The Retrieval Column: Did the Evidence Arrive The first column never looks at the model. The only question is whether the retrieved chunks carry the target procedure. Two rates stand side by side: **retrieval recall** is the share of queries for which at least one right chunk arrived, and **retrieval precision** is the share of the retrieved chunks that are right. ```python # FICTIONAL. The corpus, the concept axis, the embedding and the query set are produced inside the # lesson; there is no real institution, product, document or embedding endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) CHUNKS = [{"code": d["code"], "label": d["label"], "text": p} for d in DOCUMENTS for p in d["paras"]] VECS = [embed(p["text"]) for p in CHUNKS] def retrieve(q, k): v = embed(q) s = sorted(range(len(CHUNKS)), key=lambda i: (-cosine(v, VECS[i]), i)) return [CHUNKS[i] for i in s[:k]] def measure_retrieval(k): """The retrieval column only: do the retrieved chunks carry the target procedure.""" recall = precise = 0 for s in QUERIES: surface = CONCEPTS[s["concept"]][1] carriers = [p for p in retrieve(s["text"], k) if surface in p["text"]] recall += 1 if carriers else 0 precise += len(carriers) / k return recall / len(QUERIES), precise / len(QUERIES) print(f"corpus {len(DOCUMENTS)} documents , {len(CHUNKS)} chunks , queries {len(QUERIES)}") print("\n k retrieval recall retrieval precision chunks retrieved") for k in (1, 3, 5): a, ke = measure_retrieval(k) print(f"{k:>3} {a:>16.4f} {ke:>19.4f} {k * len(QUERIES):>16}") ``` ``` corpus 200 documents , 693 chunks , queries 40 k retrieval recall retrieval precision chunks retrieved 1 0.9500 0.9500 40 3 0.9500 0.9500 120 5 1.0000 0.9600 200 ``` Going from one chunk to three does **not move** recall: 0.9500 stays the same, while the number of retrieved chunks triples. Going to five makes recall **1.0000**, that is, the evidence is now inside the context for every query. The retrieval column says this much and cannot say more: it knows that the chunk **arrived**, and it does not know whether it was used. ## The Generation Column and the Four Cases The second column runs the same queries with a **language model simulator**. The simulator does not call a real endpoint; it scores the procedure names that appear in the context, gives a small base score to the four procedures of the class it recognizes from the query, and samples with a temperature. When the two columns intersect, every query falls into one of four boxes. ```python from math import exp def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def generate(q, context, seed=SEED, temperature=0.6): """FICTIONAL language model simulator. It scores the procedure names that appear in the context; its prior knowledge is at the class level only, it does not know the procedure name.""" prompt = q + " || " + " ".join(p["text"] for p in context) dm, sq = " " + prompt.lower() + " ", " " + q.lower() + " " label = next((CONCEPTS[k][0] for k in KEYS if CONCEPTS[k][2] in sq), None) score = {} for k in KEYS: p = (0.15 if CONCEPTS[k][0] == label else 0.0) + dm.count(CONCEPTS[k][1]) if p: score[k] = p if not score: return None, prompt names = list(score) a = [exp(score[k] / temperature) for k in names] t = sum(a) return names[pick(rng(seed ^ trace_id(prompt))(), [x / t for x in a])], prompt def run_setup(k, seed=SEED): d = {"rr": 0, "rw": 0, "wr": 0, "ww": 0, "tok": 0} for s in QUERIES: context = retrieve(s["text"], k) if k else [] reply, prompt = generate(s["text"], context, seed) d["tok"] += len(tokenize(prompt)) got_chunk = any(CONCEPTS[s["concept"]][1] in p["text"] for p in context) got_answer = reply == s["concept"] d["rr" if got_chunk and got_answer else "rw" if got_chunk else "wr" if got_answer else "ww"] += 1 return d n = len(QUERIES) print("setup right+right right+wrong wrong+right wrong+wrong single number tokens") for name, k in (("no retrieval", 0), ("k = 1", 1), ("k = 3", 3), ("k = 5", 5)): d = run_setup(k) print(f"{name:<12} {d['rr']:>11} {d['rw']:>11} {d['wr']:>11} {d['ww']:>11}" f" {(d['rr'] + d['wr']) / n:>13.4f} {d['tok']:>9}") print("\nsecond run (seed 20260219) , single number and run-to-run difference") for name, k in (("no retrieval", 0), ("k = 1", 1), ("k = 3", 3), ("k = 5", 5)): a = (lambda d: (d["rr"] + d["wr"]) / n)(run_setup(k)) b = (lambda d: (d["rr"] + d["wr"]) / n)(run_setup(k, SEED + 1)) print(f"{name:<12} {a:>7.4f} {b:>9.4f} {b - a:>+10.4f}") print("\nclass subgroup (right+right / right+wrong / wrong+right / wrong+wrong)") print(f"{'class':<12} {'k = 3':>16} {'k = 5':>16}") for label in CLASSES: row = [] for k in (3, 5): c = [0, 0, 0, 0] for s in QUERIES: if s["label"] != label: continue context = retrieve(s["text"], k) y, _ = generate(s["text"], context) got_chunk = any(CONCEPTS[s["concept"]][1] in p["text"] for p in context) c[0 if got_chunk and y == s["concept"] else 1 if got_chunk else 2 if y == s["concept"] else 3] += 1 row.append(" / ".join(str(x) for x in c)) print(f"{label:<12} {row[0]:>16} {row[1]:>16}") ``` ``` setup right+right right+wrong wrong+right wrong+wrong single number tokens no retrieval 0 0 15 25 0.3750 258 k = 1 35 3 0 2 0.8750 796 k = 3 38 0 0 2 0.9500 1872 k = 5 38 2 0 0 0.9500 2932 second run (seed 20260219) , single number and run-to-run difference no retrieval 0.3750 0.3250 -0.0500 k = 1 0.8750 0.8000 -0.0750 k = 3 0.9500 0.9500 +0.0000 k = 5 0.9500 0.9500 +0.0000 class subgroup (right+right / right+wrong / wrong+right / wrong+wrong) class k = 3 k = 5 tariff 8 / 0 / 0 / 0 8 / 0 / 0 / 0 fault 7 / 0 / 0 / 2 7 / 2 / 0 / 0 meter 10 / 0 / 0 / 0 10 / 0 / 0 / 0 outage 6 / 0 / 0 / 0 6 / 0 / 0 / 0 maintenance 7 / 0 / 0 / 0 7 / 0 / 0 / 0 ``` ## The Same Number, a Different Composition The first row of the table gives the baseline. The call without retrieval comes out at **0.3750**, and **the whole of this rate comes from the third box**: the right chunk was never retrieved, and fifteen answers came out right from the class-level guess. Someone who looks at the single number says the model knows more than a third of the questions. The four-cell contingency of retrieval and generation says, in the same row, that **no information was relied on**. This distinction prevents mistaking chance for a capability. The real demonstration is in the last two rows. The single number of the three-chunk and the five-chunk setups is **exactly the same: 0.9500**. Their composition is not. With three chunks the two failures are in the **wrong chunk + wrong answer** box, that is, the evidence never arrived; the problem is in retrieval. With five chunks retrieval recall rises to 1.0000 and those two failures move to the **right chunk + wrong answer** box: the evidence is in the context and the model did not use it. This second kind is an **answer faithfulness** fault and its fix is entirely different. The single number never shows this move, but the cost does: from 120 chunks to 200 chunks, from 1872 tokens to **2932 tokens**. The subgroup breakdown writes where which fix is to be made. Both faults are collected in **a single class**, the `fault` class; the other four classes are complete in both setups. Going to five did not rescue the lost queries, it only **changed the name** of the fault. Finally, the difference between the two runs says how much of the single number can be trusted. For the baseline without retrieval the difference is **0.0500**; a number that rests on chance is not fixed by anything. For one chunk, which brings the evidence for only part of the queries, the difference is **0.0750**. For three and five chunks the difference is **0.0000**: because the evidence in the context opens the score gap wide enough, sampling no longer turns the result. What fixes the number is not the apparatus itself but **the evidence that enters the context**. ## Summary - Retrieval and generation are two separate columns: the first measures that the evidence arrived, the second measures that the evidence was used, and one cannot take the place of the other. - The baseline without retrieval comes out at 0.3750 on this workload, and the whole of this rate is in the **wrong chunk + right answer** box, that is, it is chance. - The single number of the three-chunk and the five-chunk setups is also 0.9500, but in the first the two failures are retrieval faults and in the second the same two failures are generation faults. - Going to five chunks takes recall from 0.9500 to 1.0000; the cost is going from 120 chunks to 200 chunks and from 1872 tokens to 2932 tokens, while the end-to-end number does not change. - The class subgroup collects both faults in the `fault` class; the difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the three-chunk context. ## Next Step This lesson separated not when retrieval wins but **where the gain comes from**. One question was left unanswered: the retrieval pipeline is not the only way to give a model information. For the same task the model itself can be changed too; in the Building Applications with Model APIs course the number for fine-tuning was measured, and it was seen that with sixty examples it could not beat the prompt setup. The next lesson puts these two solutions on the same table and measures them side by side on three separate tasks: it counts **which problem** each one solves and on which task **retrieval does not win**.