--- title: 'What Is Retrieval-Augmented Generation' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/what-is-retrieval-augmented-generation' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:30+00:00' license: 'CC BY-SA 4.0' objectives: - "Splits retrieval-augmented generation into two separately measured stages, retrieval and generation, and prints each stage's own number." - 'Runs the same query set in three setups, a no-retrieval baseline, retrieval with an in-domain embedding and wrong chunks with an out-of-domain embedding, and compares their generation hits.' - 'Shows with numbers that a wrongly retrieved chunk drops generation below the no-retrieval baseline, and writes the cost of the pipeline as tokens, chunks and retrieval steps.' - "Computes this course's measurement threshold from the six-seed run and separates the case where the correct chunk arrived but the answer was wrong." --- # What Is Retrieval-Augmented Generation Running the same forty queries in three setups over a fictional corpus of two hundred documents and printing retrieval and generation as two separate columns: the no-retrieval baseline scores 0.2000 on generation, while the pipeline built with an in-domain embedding raises retrieval recall to 0.9500 and generation to 0.9250, and the same pipeline built with an out-of-domain embedding leaves recall at 0.1750 and drops generation to 0.0500, below the baseline. The cost is written side by side: from 3033 tokens to 4848 tokens, from zero retrieval steps to twenty-seven thousand seven hundred twenty steps, from 379.1 tokens per correct answer to 131.0 tokens, and to 2457.5 tokens in the wrong-chunk run. The run of six seeds gives this workload a threshold of 0.0750, and one of the thirty-eight queries that receive the correct chunk is still answered wrongly. The six lessons of the previous topic did a single job: given a query, find a chunk in the corpus. The embedding was set up as a search key, the loss of domain fit was measured, a similarity measure was chosen, an indexed search replaced the exhaustive search, a metadata filter was added, and two rankings were merged. In all six the measure sat in the same place: was the retrieved chunk correct. The thing nobody asked is what that chunk does once it is handed to the model. **Retrieval-augmented generation** means putting chunks selected from the corpus into the prompt and calling the model with those chunks. The rule this lesson sets up holds for sixteen lessons: **retrieval and generation are measured separately.** When the two are reduced to one number, it is not visible which stage broke things. - **RP1.** The code is a **simulator**; no real embedding or model endpoint is called. The corpus, the concept axis and the queries are **fictional**, seed **20260218**. - **RP2.** The definition of an embedding, cosine similarity and neighborhood were measured in the Natural Language Processing and Computer Vision course and are **not repeated**. Here the embedding is only a **search key**. - **RP3.** Every concept has a **document surface** (`tier limit`) and a different **query surface** (`which band am i in`), and the two share no token: the word the subscriber asks with is not the word the record is written in. - **RP4.** The simulator's internal trace recognizes the **document surface**, and recognizes only a seven-concept part of the query surface. This is the only source of the no-retrieval baseline. - **RP5.** In this lesson the retrieval unit is **the source's own paragraph boundary**, and it is called a **chunk**; that the boundary is a decision is measured in the third lesson. - **RP6.** **Retrieval recall** is the share of queries for which at least one chunk carrying the target concept is in the top three; **retrieval precision** is the proportion of target-concept chunks in the top three; the **class hit rate** is the proportion of the top three that come from the target class. The class hit rate is the easy measure, the concept hit rate is the hard one. - **RP7.** The **generation measure** is the answer naming the target concept, and it is not summed into one column with the retrieval measure. The rate limit depends on the call count: the thirty-seventh of forty calls returns empty, and because it falls in the same place in all three setups it does not distort the comparison. ## Corpus and Unit of Measure The block below sets up the fictional corpus, the concept axis, the embedding simulator and the query set. Each of the twenty concepts is tied to a class and is written with two surfaces. ```python # FICTIONAL. The corpus, concept axis and embedding below are a simulator; no real # embedding endpoint is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- chunk: in this lesson the unit is the source's own paragraph boundary CHUNKS = [{"code": d["code"], "label": d["label"], "text": p, "concepts": [k for k in KEYS if CONCEPTS[k][1] in p]} for d in DOCUMENTS for p in d["paras"]] print("documents %d chunks %d concepts %d queries %d tokens per chunk %.1f" % (len(DOCUMENTS), len(CHUNKS), len(KEYS), len(QUERIES), sum(len(tokenize(x["text"])) for x in CHUNKS) / len(CHUNKS))) print("labels", {s: sum(d["label"] == s for d in DOCUMENTS) for s in CLASSES}) ``` ``` documents 200 chunks 693 concepts 20 queries 40 tokens per chunk 13.0 labels {'tariff': 52, 'fault': 46, 'meter': 39, 'outage': 32, 'maintenance': 31} ``` The scale reads off from here: six hundred ninety-three chunks, thirteen tokens per chunk. The answer to a query sits inside a stack of about nine thousand tokens, and the context window of 240 tokens holds only a small part of it. ## Three Setups The second block sets up the language model simulator and runs the same forty queries in three setups. The simulator comes from the family built in the Introduction to AI Engineering course: it counts tokens, carries a context window and a stopping condition, can hit a rate limit, and gives the same response when it is called with the same prompt and the same seed. The only adaptation is the response space: the model writes one of twenty concepts. The **no-retrieval baseline** calls the model with the query alone. The **retrieval** setup puts the three nearest chunks, found with the in-domain embedding, into the prompt. The **wrong-chunk run** builds the same pipeline with an out-of-domain embedding, and it is not a hand-broken scaffolding: it is the pipeline that results when an embedding that does not recognize the query language is chosen. ```python import math # ---- FICTIONAL language model simulator: the response space is twenty concepts WINDOW, MAX_OUTPUT = 240, 24 PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"} FREE_FORM = ["request linked to {k} concept and logged", "{k} record created and passed to the relevant department", "your report was received and the relevant unit will reply"] COUNTER = [0] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 b = tokenize(prompt) if len(b) > WINDOW: return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: # the rate limit depends on the call count return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) dm = " " + prompt.lower() + " " p = {k: 0.15 for k in KEYS} for k in KEYS: p[k] += 2.0 * dm.count(CONCEPTS[k][1]) # the model recognizes the document language if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm: p[k] += 2.0 # the known share of the query language if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: w = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [x / sum(w) for x in w])] y = k if "one" in b else FREE_FORM[int(r() * 3)].format(k=k) cb = tokenize(y) if len(cb) > max_output: cb, y = cb[:max_output], " ".join(cb[:max_output]) return {"status": "ok", "input": len(b), "output": len(cb), "response": y} def read_concept(y): return next((s for s in y.split() if s in KEYS), None) INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) + " . Reply with one concept . ") VEC = {True: [embed(x["text"]) for x in CHUNKS], False: [embed(x["text"], domain=False) for x in CHUNKS]} def run(mode, seed=SEED, k=3): COUNTER[0] = 0 recall = precision = label_hit = correct = inp = out = 0 for s in QUERIES: if mode == "no retrieval": prompt = INSTRUCTION + "Request: " + s["text"] else: domain = mode == "retrieval" v = embed(s["text"], domain=domain) top = sorted(range(len(CHUNKS)), key=lambda i: -cosine(v, VEC[domain][i]))[:k] prompt = (INSTRUCTION + "Source: " + " . ".join(CHUNKS[i]["text"] for i in top) + " . Request: " + s["text"]) h = [s["concept"] in CHUNKS[i]["concepts"] for i in top] recall += any(h) precision += sum(h) / k label_hit += sum(CHUNKS[i]["label"] == s["label"] for i in top) / k y = call(prompt, seed) inp, out = inp + y["input"], out + y["output"] correct += read_concept(y["response"]) == s["concept"] n = len(QUERIES) return {"recall": recall / n, "precision": precision / n, "label_hit": label_hit / n, "generation": correct / n, "tokens": inp + out, "chunks": 0 if mode == "no retrieval" else k * n, "steps": 0 if mode == "no retrieval" else len(CHUNKS) * n, "correct": correct} print("setup RETRIEVAL: recall precision label GENERATION COST: tokens chunks" " steps tokens/correct") for mode in ("no retrieval", "retrieval", "wrong chunks"): a = run("wrong" if mode.startswith("wrong") else mode) e = (" - - -" if mode == "no retrieval" else " %.4f %.4f %.4f" % (a["recall"], a["precision"], a["label_hit"])) print("%-13s%s %.4f %6d %5d %6d %14.1f" % (mode, e, a["generation"], a["tokens"], a["chunks"], a["steps"], a["tokens"] / a["correct"])) ``` ``` setup RETRIEVAL: recall precision label GENERATION COST: tokens chunks steps tokens/correct no retrieval - - - 0.2000 3033 0 0 379.1 retrieval 0.9500 0.9500 0.9500 0.9250 4848 120 27720 131.0 wrong chunks 0.1750 0.1750 0.2500 0.0500 4915 120 27720 2457.5 ``` ## Three Readings of the Table The first reading is what retrieval adds. The no-retrieval baseline is **0.2000**, and the queries it answers correctly are the ones that happen to fall on concepts the simulator recognizes from the query language. The pipeline built with the in-domain embedding raises generation to **0.9250**, against that baseline of 0.2000. The second reading is the cost. Total tokens rise **from 3033 to 4848**, three chunks are retrieved for each query and the whole corpus is scanned: six hundred ninety-three comparisons per query, **27720 retrieval steps** over forty queries. A step is not a model call, but it does not come free either. Read per correct answer, the arithmetic reverses: **from 379.1 tokens to 131.0 tokens**. The third reading is the second claim of this course: **retrieval also breaks things.** The wrong-chunk run spends **more** tokens than the baseline (4915) and drops generation to **0.0500**, that is, below the baseline's **0.2000**. The retrieved chunks are written in the document language and the model recognizes the document language, so it reads the wrong concepts **with confidence**; the no-retrieval model's uncertain guess turns into the confident error of a wrong source. The cost per correct answer rises to **2457.5 tokens**. The retrieval column is also read with two numbers: in the out-of-domain run the class hit rate is **0.2500**, which is the level of a random pick among five classes, and the concept hit rate is **0.1750**. Had a single hit number been printed, it would have stayed unclear which measure was being discussed. ## Threshold and Separating the Two Columns A difference counts as a difference only if it is larger than the run-to-run variability. This threshold was measured on its own workload in the Building Applications with Model APIs course; this course does **not inherit** it. The third block runs the same three setups with six seeds and then counts where the retrieval and generation columns come apart. ```python print("setup generation hits of six seeds spread") threshold = 0.0 for mode in ("no retrieval", "retrieval", "wrong"): v = [run(mode, SEED + i)["generation"] for i in range(6)] threshold = max(threshold, max(v) - min(v)) print("%-13s %s %.4f" % (mode, " ".join("%.4f" % x for x in v), max(v) - min(v))) print("threshold of this course %.4f (%d of the forty queries)" % (threshold, round(threshold * 40))) COUNTER[0] = 0 gc = gw = 0 for s in QUERIES: v = embed(s["text"]) top = sorted(range(len(CHUNKS)), key=lambda i: -cosine(v, VEC[True][i]))[:3] if not any(s["concept"] in CHUNKS[i]["concepts"] for i in top): continue prompt = (INSTRUCTION + "Source: " + " . ".join(CHUNKS[i]["text"] for i in top) + " . Request: " + s["text"]) d = read_concept(call(prompt)["response"]) == s["concept"] gc, gw = gc + d, gw + (not d) print("%d queries with the correct chunk: %d correct, %d wrong" % (gc + gw, gc, gw)) ``` ``` setup generation hits of six seeds spread no retrieval 0.2000 0.2000 0.2250 0.1500 0.2250 0.1750 0.0750 retrieval 0.9250 0.9250 0.9250 0.9250 0.9250 0.9250 0.0000 wrong 0.0500 0.0500 0.1000 0.0750 0.1250 0.0750 0.0750 threshold of this course 0.0750 (3 of the forty queries) 38 queries with the correct chunk: 37 correct, 1 wrong ``` The threshold of this course is **0.0750**: three of the forty queries. A generation difference smaller than this is **unmeasured** on this workload. The spread also changes from setup to setup: 0.0750 for the no-retrieval baseline, **0.0000** for the retrieval pipeline. The output of a model that was given the source does not depend on the seed, because the correct concept scores far above the others. Uncertainty grows where knowledge is absent; this is the less-told second gain of retrieval. The retrieval measure, on the other hand, is independent of the seed, because the embedding and the ranking are deterministic. For this reason the threshold applies only to the **generation** column. The payoff of keeping the two columns apart is the last line: of the **thirty-eight** queries that receive the correct chunk, thirty-seven are answered correctly and **one** wrongly. For that query the failure is in the generation stage; had a single number been printed, that query would have fallen into the same box as the two queries whose chunk could not be retrieved. ## Summary - Retrieval-augmented generation is putting chunks selected from the corpus into the prompt and calling the model with those chunks; the rule of the course is that retrieval and generation are **measured separately**. - On the same forty queries the no-retrieval baseline gives 0.2000 generation, and the pipeline built with the in-domain embedding gives 0.9250. - The gain is not read without the cost written: total tokens rise from 3033 to 4848, three chunks and 693 comparisons are added per query; tokens per correct answer, however, fall from 379.1 to 131.0. - The same pipeline built with an out-of-domain embedding drops generation to 0.0500, that is, **below** the no-retrieval baseline, and does so while spending more tokens: retrieval also breaks things. - The measurement threshold of this workload comes out as 0.0750 from the six-seed run; the spread of the retrieval pipeline is 0.0000. - One of the thirty-eight queries that receive the correct chunk is still answered wrongly. ## Next Step In this lesson the corpus arrived ready: clean paragraphs, boundaries in the right places, code and class fields kept apart. A real source does not arrive like this; field records carry form headers, page footers, sentences split in the middle of a line, and code fields embedded in the text. What these leftovers do to retrieval was never asked: which preparation raises recall, and **what the preparation itself throws away**. The next lesson starts from the raw form of the same corpus, measures three levels of preparation on the same query set, and counts the lines that a crude cleaning rule throws away.