Measuring Retrieval and Generation Quality Separately

The previous lesson gave the same chunks in different orders and counted the share of change in the answer. The pipeline is now built: the document was prepared, split into chunks, indexed, the query was transformed in a single step, the candidates were reranked and the context was constructed under the limit. Each of the seven lessons defended its own step with its own number.

This lesson stops where those numbers come together. When a retrieval-augmented generation pipeline is run, what is usually left is a single rate: how many questions were answered correctly. The claim of this lesson is that this single rate stacks four separate cases on top of one another, and that from inside the stack it cannot be read which stage is broken. The evaluation procedure itself was established in the Model Evaluation, Interpretability and Ethics course, together with its lessons on model evaluation and subgroups, and is not repeated here; the only thing taken over from there is the baseline rule, and in this course the baseline is the call without retrieval.

• RP71. The vocabulary, the concept axis, the embedding and the query set are fictional and are produced inside the lesson. There is no real institution, product, document, vector store or embedding endpoint.
• RP72. The concept has one surface in the document language (tier limit) and another surface in the query language (which band am i in), and the two share no token.
• RP73. The retrieval measure: if the document surface of the target procedure appears inside the retrieved chunks, retrieval is right. This measure is independent of the seed.
• RP74. The generation measure: if the procedure named in the answer is the target procedure, generation is right. The answer comes from a language model simulator; the simulator does not call a real endpoint.
• RP75. The prior knowledge of the simulator is at the class level only. From the query language it infers which class the query belongs to and gives an equal, small base score to the four procedures of that class, but it does not know a procedure name on its own. Every procedure name that appears in the context adds a point.
• RP76. Cost is written in three units: the number of retrieved chunks, the tokens entering the prompt and the retrieval step. Time is not written.

The Retrieval Column: Did the Evidence Arrive

The first column never looks at the model. The only question is whether the retrieved chunks carry the target procedure. Two rates stand side by side: retrieval recall is the share of queries for which at least one right chunk arrived, and retrieval precision is the share of the retrieved chunks that are right.

# FICTIONAL. The corpus, the concept axis, the embedding and the query set are produced inside the
# lesson; there is no real institution, product, document or embedding endpoint.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0      # the out-of-domain embedding does not recognize the query language
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

CHUNKS = [{"code": d["code"], "label": d["label"], "text": p}
          for d in DOCUMENTS for p in d["paras"]]
VECS = [embed(p["text"]) for p in CHUNKS]


def retrieve(q, k):
    v = embed(q)
    s = sorted(range(len(CHUNKS)), key=lambda i: (-cosine(v, VECS[i]), i))
    return [CHUNKS[i] for i in s[:k]]


def measure_retrieval(k):
    """The retrieval column only: do the retrieved chunks carry the target procedure."""
    recall = precise = 0
    for s in QUERIES:
        surface = CONCEPTS[s["concept"]][1]
        carriers = [p for p in retrieve(s["text"], k) if surface in p["text"]]
        recall += 1 if carriers else 0
        precise += len(carriers) / k
    return recall / len(QUERIES), precise / len(QUERIES)


print(f"corpus {len(DOCUMENTS)} documents , {len(CHUNKS)} chunks , queries {len(QUERIES)}")
print("\n  k retrieval recall retrieval precision chunks retrieved")
for k in (1, 3, 5):
    a, ke = measure_retrieval(k)
    print(f"{k:>3} {a:>16.4f} {ke:>19.4f} {k * len(QUERIES):>16}")

corpus 200 documents , 693 chunks , queries 40

  k retrieval recall retrieval precision chunks retrieved
  1           0.9500              0.9500               40
  3           0.9500              0.9500              120
  5           1.0000              0.9600              200

Going from one chunk to three does not move recall: 0.9500 stays the same, while the number of retrieved chunks triples. Going to five makes recall 1.0000, that is, the evidence is now inside the context for every query. The retrieval column says this much and cannot say more: it knows that the chunk arrived, and it does not know whether it was used.

The Generation Column and the Four Cases

The second column runs the same queries with a language model simulator. The simulator does not call a real endpoint; it scores the procedure names that appear in the context, gives a small base score to the four procedures of the class it recognizes from the query, and samples with a temperature. When the two columns intersect, every query falls into one of four boxes.

from math import exp


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def generate(q, context, seed=SEED, temperature=0.6):
    """FICTIONAL language model simulator. It scores the procedure names that appear in the context;
    its prior knowledge is at the class level only, it does not know the procedure name."""
    prompt = q + " || " + " ".join(p["text"] for p in context)
    dm, sq = " " + prompt.lower() + " ", " " + q.lower() + " "
    label = next((CONCEPTS[k][0] for k in KEYS if CONCEPTS[k][2] in sq), None)
    score = {}
    for k in KEYS:
        p = (0.15 if CONCEPTS[k][0] == label else 0.0) + dm.count(CONCEPTS[k][1])
        if p:
            score[k] = p
    if not score:
        return None, prompt
    names = list(score)
    a = [exp(score[k] / temperature) for k in names]
    t = sum(a)
    return names[pick(rng(seed ^ trace_id(prompt))(), [x / t for x in a])], prompt


def run_setup(k, seed=SEED):
    d = {"rr": 0, "rw": 0, "wr": 0, "ww": 0, "tok": 0}
    for s in QUERIES:
        context = retrieve(s["text"], k) if k else []
        reply, prompt = generate(s["text"], context, seed)
        d["tok"] += len(tokenize(prompt))
        got_chunk = any(CONCEPTS[s["concept"]][1] in p["text"] for p in context)
        got_answer = reply == s["concept"]
        d["rr" if got_chunk and got_answer else "rw" if got_chunk else "wr" if got_answer else "ww"] += 1
    return d


n = len(QUERIES)
print("setup        right+right right+wrong wrong+right wrong+wrong single number    tokens")
for name, k in (("no retrieval", 0), ("k = 1", 1), ("k = 3", 3), ("k = 5", 5)):
    d = run_setup(k)
    print(f"{name:<12} {d['rr']:>11} {d['rw']:>11} {d['wr']:>11} {d['ww']:>11}"
          f" {(d['rr'] + d['wr']) / n:>13.4f} {d['tok']:>9}")

print("\nsecond run (seed 20260219) , single number and run-to-run difference")
for name, k in (("no retrieval", 0), ("k = 1", 1), ("k = 3", 3), ("k = 5", 5)):
    a = (lambda d: (d["rr"] + d["wr"]) / n)(run_setup(k))
    b = (lambda d: (d["rr"] + d["wr"]) / n)(run_setup(k, SEED + 1))
    print(f"{name:<12} {a:>7.4f} {b:>9.4f} {b - a:>+10.4f}")

print("\nclass subgroup (right+right / right+wrong / wrong+right / wrong+wrong)")
print(f"{'class':<12} {'k = 3':>16} {'k = 5':>16}")
for label in CLASSES:
    row = []
    for k in (3, 5):
        c = [0, 0, 0, 0]
        for s in QUERIES:
            if s["label"] != label:
                continue
            context = retrieve(s["text"], k)
            y, _ = generate(s["text"], context)
            got_chunk = any(CONCEPTS[s["concept"]][1] in p["text"] for p in context)
            c[0 if got_chunk and y == s["concept"] else 1 if got_chunk else 2 if y == s["concept"] else 3] += 1
        row.append(" / ".join(str(x) for x in c))
    print(f"{label:<12} {row[0]:>16} {row[1]:>16}")

setup        right+right right+wrong wrong+right wrong+wrong single number    tokens
no retrieval           0           0          15          25        0.3750       258
k = 1                 35           3           0           2        0.8750       796
k = 3                 38           0           0           2        0.9500      1872
k = 5                 38           2           0           0        0.9500      2932

second run (seed 20260219) , single number and run-to-run difference
no retrieval  0.3750    0.3250    -0.0500
k = 1         0.8750    0.8000    -0.0750
k = 3         0.9500    0.9500    +0.0000
k = 5         0.9500    0.9500    +0.0000

class subgroup (right+right / right+wrong / wrong+right / wrong+wrong)
class                   k = 3            k = 5
tariff          8 / 0 / 0 / 0    8 / 0 / 0 / 0
fault           7 / 0 / 0 / 2    7 / 2 / 0 / 0
meter          10 / 0 / 0 / 0   10 / 0 / 0 / 0
outage          6 / 0 / 0 / 0    6 / 0 / 0 / 0
maintenance     7 / 0 / 0 / 0    7 / 0 / 0 / 0

The Same Number, a Different Composition

The first row of the table gives the baseline. The call without retrieval comes out at 0.3750, and the whole of this rate comes from the third box: the right chunk was never retrieved, and fifteen answers came out right from the class-level guess. Someone who looks at the single number says the model knows more than a third of the questions. The four-cell contingency of retrieval and generation says, in the same row, that no information was relied on. This distinction prevents mistaking chance for a capability.

The real demonstration is in the last two rows. The single number of the three-chunk and the five-chunk setups is exactly the same: 0.9500. Their composition is not. With three chunks the two failures are in the wrong chunk + wrong answer box, that is, the evidence never arrived; the problem is in retrieval. With five chunks retrieval recall rises to 1.0000 and those two failures move to the right chunk + wrong answer box: the evidence is in the context and the model did not use it. This second kind is an answer faithfulness fault and its fix is entirely different. The single number never shows this move, but the cost does: from 120 chunks to 200 chunks, from 1872 tokens to 2932 tokens.

The subgroup breakdown writes where which fix is to be made. Both faults are collected in a single class, the fault class; the other four classes are complete in both setups. Going to five did not rescue the lost queries, it only changed the name of the fault.

Finally, the difference between the two runs says how much of the single number can be trusted. For the baseline without retrieval the difference is 0.0500; a number that rests on chance is not fixed by anything. For one chunk, which brings the evidence for only part of the queries, the difference is 0.0750. For three and five chunks the difference is 0.0000: because the evidence in the context opens the score gap wide enough, sampling no longer turns the result. What fixes the number is not the apparatus itself but the evidence that enters the context.

Summary

• Retrieval and generation are two separate columns: the first measures that the evidence arrived, the second measures that the evidence was used, and one cannot take the place of the other.
• The baseline without retrieval comes out at 0.3750 on this workload, and the whole of this rate is in the wrong chunk + right answer box, that is, it is chance.
• The single number of the three-chunk and the five-chunk setups is also 0.9500, but in the first the two failures are retrieval faults and in the second the same two failures are generation faults.
• Going to five chunks takes recall from 0.9500 to 1.0000; the cost is going from 120 chunks to 200 chunks and from 1872 tokens to 2932 tokens, while the end-to-end number does not change.
• The class subgroup collects both faults in the fault class; the difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the three-chunk context.

Next Step

This lesson separated not when retrieval wins but where the gain comes from. One question was left unanswered: the retrieval pipeline is not the only way to give a model information. For the same task the model itself can be changed too; in the Building Applications with Model APIs course the number for fine-tuning was measured, and it was seen that with sixty examples it could not beat the prompt setup. The next lesson puts these two solutions on the same table and measures them side by side on three separate tasks: it counts which problem each one solves and on which task retrieval does not win.