What Is Retrieval-Augmented Generation

The six lessons of the previous topic did a single job: given a query, find a chunk in the corpus. The embedding was set up as a search key, the loss of domain fit was measured, a similarity measure was chosen, an indexed search replaced the exhaustive search, a metadata filter was added, and two rankings were merged. In all six the measure sat in the same place: was the retrieved chunk correct.

The thing nobody asked is what that chunk does once it is handed to the model. Retrieval-augmented generation means putting chunks selected from the corpus into the prompt and calling the model with those chunks. The rule this lesson sets up holds for sixteen lessons: retrieval and generation are measured separately. When the two are reduced to one number, it is not visible which stage broke things.

• RP1. The code is a simulator; no real embedding or model endpoint is called. The corpus, the concept axis and the queries are fictional, seed 20260218.
• RP2. The definition of an embedding, cosine similarity and neighborhood were measured in the Natural Language Processing and Computer Vision course and are not repeated. Here the embedding is only a search key.
• RP3. Every concept has a document surface (tier limit) and a different query surface (which band am i in), and the two share no token: the word the subscriber asks with is not the word the record is written in.
• RP4. The simulator’s internal trace recognizes the document surface, and recognizes only a seven-concept part of the query surface. This is the only source of the no-retrieval baseline.
• RP5. In this lesson the retrieval unit is the source’s own paragraph boundary, and it is called a chunk; that the boundary is a decision is measured in the third lesson.
• RP6. Retrieval recall is the share of queries for which at least one chunk carrying the target concept is in the top three; retrieval precision is the proportion of target-concept chunks in the top three; the class hit rate is the proportion of the top three that come from the target class. The class hit rate is the easy measure, the concept hit rate is the hard one.
• RP7. The generation measure is the answer naming the target concept, and it is not summed into one column with the retrieval measure. The rate limit depends on the call count: the thirty-seventh of forty calls returns empty, and because it falls in the same place in all three setups it does not distort the comparison.

Corpus and Unit of Measure

The block below sets up the fictional corpus, the concept axis, the embedding simulator and the query set. Each of the twenty concepts is tied to a class and is written with two surfaces.

# FICTIONAL. The corpus, concept axis and embedding below are a simulator; no real
# embedding endpoint is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0      # the out-of-domain embedding does not recognize the query language
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# ---- chunk: in this lesson the unit is the source's own paragraph boundary
CHUNKS = [{"code": d["code"], "label": d["label"], "text": p,
           "concepts": [k for k in KEYS if CONCEPTS[k][1] in p]}
          for d in DOCUMENTS for p in d["paras"]]

print("documents %d  chunks %d  concepts %d  queries %d  tokens per chunk %.1f"
      % (len(DOCUMENTS), len(CHUNKS), len(KEYS), len(QUERIES),
         sum(len(tokenize(x["text"])) for x in CHUNKS) / len(CHUNKS)))
print("labels", {s: sum(d["label"] == s for d in DOCUMENTS) for s in CLASSES})

documents 200  chunks 693  concepts 20  queries 40  tokens per chunk 13.0
labels {'tariff': 52, 'fault': 46, 'meter': 39, 'outage': 32, 'maintenance': 31}

The scale reads off from here: six hundred ninety-three chunks, thirteen tokens per chunk. The answer to a query sits inside a stack of about nine thousand tokens, and the context window of 240 tokens holds only a small part of it.

Three Setups

The second block sets up the language model simulator and runs the same forty queries in three setups. The simulator comes from the family built in the Introduction to AI Engineering course: it counts tokens, carries a context window and a stopping condition, can hit a rate limit, and gives the same response when it is called with the same prompt and the same seed. The only adaptation is the response space: the model writes one of twenty concepts.

The no-retrieval baseline calls the model with the query alone. The retrieval setup puts the three nearest chunks, found with the in-domain embedding, into the prompt. The wrong-chunk run builds the same pipeline with an out-of-domain embedding, and it is not a hand-broken scaffolding: it is the pipeline that results when an embedding that does not recognize the query language is chosen.

import math
# ---- FICTIONAL language model simulator: the response space is twenty concepts
WINDOW, MAX_OUTPUT = 240, 24
PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"}
FREE_FORM = ["request linked to {k} concept and logged",
             "{k} record created and passed to the relevant department",
             "your report was received and the relevant unit will reply"]
COUNTER = [0]


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT):
    COUNTER[0] += 1
    b = tokenize(prompt)
    if len(b) > WINDOW:
        return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:                             # the rate limit depends on the call count
        return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    dm = " " + prompt.lower() + " "
    p = {k: 0.15 for k in KEYS}
    for k in KEYS:
        p[k] += 2.0 * dm.count(CONCEPTS[k][1])           # the model recognizes the document language
        if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm:
            p[k] += 2.0                                  # the known share of the query language
    if temperature <= 0.0:
        k = max(sorted(p), key=lambda a: p[a])
    else:
        w = [math.exp(p[a] / temperature) for a in sorted(p)]
        k = sorted(p)[pick(r(), [x / sum(w) for x in w])]
    y = k if "one" in b else FREE_FORM[int(r() * 3)].format(k=k)
    cb = tokenize(y)
    if len(cb) > max_output:
        cb, y = cb[:max_output], " ".join(cb[:max_output])
    return {"status": "ok", "input": len(b), "output": len(cb), "response": y}


def read_concept(y):
    return next((s for s in y.split() if s in KEYS), None)


INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) +
               " . Reply with one concept . ")
VEC = {True: [embed(x["text"]) for x in CHUNKS],
       False: [embed(x["text"], domain=False) for x in CHUNKS]}


def run(mode, seed=SEED, k=3):
    COUNTER[0] = 0
    recall = precision = label_hit = correct = inp = out = 0
    for s in QUERIES:
        if mode == "no retrieval":
            prompt = INSTRUCTION + "Request: " + s["text"]
        else:
            domain = mode == "retrieval"
            v = embed(s["text"], domain=domain)
            top = sorted(range(len(CHUNKS)), key=lambda i: -cosine(v, VEC[domain][i]))[:k]
            prompt = (INSTRUCTION + "Source: " + " . ".join(CHUNKS[i]["text"] for i in top) +
                      " . Request: " + s["text"])
            h = [s["concept"] in CHUNKS[i]["concepts"] for i in top]
            recall += any(h)
            precision += sum(h) / k
            label_hit += sum(CHUNKS[i]["label"] == s["label"] for i in top) / k
        y = call(prompt, seed)
        inp, out = inp + y["input"], out + y["output"]
        correct += read_concept(y["response"]) == s["concept"]
    n = len(QUERIES)
    return {"recall": recall / n, "precision": precision / n, "label_hit": label_hit / n,
            "generation": correct / n, "tokens": inp + out,
            "chunks": 0 if mode == "no retrieval" else k * n,
            "steps": 0 if mode == "no retrieval" else len(CHUNKS) * n, "correct": correct}


print("setup          RETRIEVAL: recall precision label  GENERATION  COST: tokens chunks"
      "   steps  tokens/correct")
for mode in ("no retrieval", "retrieval", "wrong chunks"):
    a = run("wrong" if mode.startswith("wrong") else mode)
    e = ("       -        -      -" if mode == "no retrieval"
         else "  %.4f   %.4f %.4f" % (a["recall"], a["precision"], a["label_hit"]))
    print("%-13s%s  %.4f          %6d %5d %6d %14.1f"
          % (mode, e, a["generation"], a["tokens"], a["chunks"], a["steps"],
             a["tokens"] / a["correct"]))

setup          RETRIEVAL: recall precision label  GENERATION  COST: tokens chunks   steps  tokens/correct
no retrieval        -        -      -  0.2000            3033     0      0          379.1
retrieval      0.9500   0.9500 0.9500  0.9250            4848   120  27720          131.0
wrong chunks   0.1750   0.1750 0.2500  0.0500            4915   120  27720         2457.5

Three Readings of the Table

The first reading is what retrieval adds. The no-retrieval baseline is 0.2000, and the queries it answers correctly are the ones that happen to fall on concepts the simulator recognizes from the query language. The pipeline built with the in-domain embedding raises generation to 0.9250, against that baseline of 0.2000.

The second reading is the cost. Total tokens rise from 3033 to 4848, three chunks are retrieved for each query and the whole corpus is scanned: six hundred ninety-three comparisons per query, 27720 retrieval steps over forty queries. A step is not a model call, but it does not come free either. Read per correct answer, the arithmetic reverses: from 379.1 tokens to 131.0 tokens.

The third reading is the second claim of this course: retrieval also breaks things. The wrong-chunk run spends more tokens than the baseline (4915) and drops generation to 0.0500, that is, below the baseline’s 0.2000. The retrieved chunks are written in the document language and the model recognizes the document language, so it reads the wrong concepts with confidence; the no-retrieval model’s uncertain guess turns into the confident error of a wrong source. The cost per correct answer rises to 2457.5 tokens.

The retrieval column is also read with two numbers: in the out-of-domain run the class hit rate is 0.2500, which is the level of a random pick among five classes, and the concept hit rate is 0.1750. Had a single hit number been printed, it would have stayed unclear which measure was being discussed.

Threshold and Separating the Two Columns

A difference counts as a difference only if it is larger than the run-to-run variability. This threshold was measured on its own workload in the Building Applications with Model APIs course; this course does not inherit it. The third block runs the same three setups with six seeds and then counts where the retrieval and generation columns come apart.

print("setup          generation hits of six seeds                        spread")
threshold = 0.0
for mode in ("no retrieval", "retrieval", "wrong"):
    v = [run(mode, SEED + i)["generation"] for i in range(6)]
    threshold = max(threshold, max(v) - min(v))
    print("%-13s %s   %.4f" % (mode, " ".join("%.4f" % x for x in v), max(v) - min(v)))
print("threshold of this course %.4f  (%d of the forty queries)" % (threshold, round(threshold * 40)))

COUNTER[0] = 0
gc = gw = 0
for s in QUERIES:
    v = embed(s["text"])
    top = sorted(range(len(CHUNKS)), key=lambda i: -cosine(v, VEC[True][i]))[:3]
    if not any(s["concept"] in CHUNKS[i]["concepts"] for i in top):
        continue
    prompt = (INSTRUCTION + "Source: " + " . ".join(CHUNKS[i]["text"] for i in top) +
              " . Request: " + s["text"])
    d = read_concept(call(prompt)["response"]) == s["concept"]
    gc, gw = gc + d, gw + (not d)
print("%d queries with the correct chunk: %d correct, %d wrong" % (gc + gw, gc, gw))

setup          generation hits of six seeds                        spread
no retrieval  0.2000 0.2000 0.2250 0.1500 0.2250 0.1750   0.0750
retrieval     0.9250 0.9250 0.9250 0.9250 0.9250 0.9250   0.0000
wrong         0.0500 0.0500 0.1000 0.0750 0.1250 0.0750   0.0750
threshold of this course 0.0750  (3 of the forty queries)
38 queries with the correct chunk: 37 correct, 1 wrong

The threshold of this course is 0.0750: three of the forty queries. A generation difference smaller than this is unmeasured on this workload. The spread also changes from setup to setup: 0.0750 for the no-retrieval baseline, 0.0000 for the retrieval pipeline. The output of a model that was given the source does not depend on the seed, because the correct concept scores far above the others. Uncertainty grows where knowledge is absent; this is the less-told second gain of retrieval.

The retrieval measure, on the other hand, is independent of the seed, because the embedding and the ranking are deterministic. For this reason the threshold applies only to the generation column. The payoff of keeping the two columns apart is the last line: of the thirty-eight queries that receive the correct chunk, thirty-seven are answered correctly and one wrongly. For that query the failure is in the generation stage; had a single number been printed, that query would have fallen into the same box as the two queries whose chunk could not be retrieved.

Summary

• Retrieval-augmented generation is putting chunks selected from the corpus into the prompt and calling the model with those chunks; the rule of the course is that retrieval and generation are measured separately.
• On the same forty queries the no-retrieval baseline gives 0.2000 generation, and the pipeline built with the in-domain embedding gives 0.9250.
• The gain is not read without the cost written: total tokens rise from 3033 to 4848, three chunks and 693 comparisons are added per query; tokens per correct answer, however, fall from 379.1 to 131.0.
• The same pipeline built with an out-of-domain embedding drops generation to 0.0500, that is, below the no-retrieval baseline, and does so while spending more tokens: retrieval also breaks things.
• The measurement threshold of this workload comes out as 0.0750 from the six-seed run; the spread of the retrieval pipeline is 0.0000.
• One of the thirty-eight queries that receive the correct chunk is still answered wrongly.

Next Step

In this lesson the corpus arrived ready: clean paragraphs, boundaries in the right places, code and class fields kept apart. A real source does not arrive like this; field records carry form headers, page footers, sentences split in the middle of a line, and code fields embedded in the text. What these leftovers do to retrieval was never asked: which preparation raises recall, and what the preparation itself throws away. The next lesson starts from the raw form of the same corpus, measures three levels of preparation on the same query set, and counts the lines that a crude cleaning rule throws away.