Measuring Retrieval and Fine-Tuning Side by Side

The previous lesson split the single number of the retrieval pipeline into four cases and showed that the single number hides which stage is broken. The split also exposed this: the job of retrieval is to change the information entering the prompt. It never touches the model itself.

There is a way to touch the model, and it has been measured. In the Building Applications with Model APIs course, fine-tuning was tried on the same support task: the baseline was 0.6000, the prompt setup with sixty examples was 0.9667, and fine-tuning with the same sixty examples was 0.9500. The context per request went from 28.48 tokens to 623.48 tokens. That procedure is not repeated here; those numbers are the input of this lesson. The question of this lesson is a different one: given a task, which of the two solves which problem, and on which task retrieval gains nothing at all.

• RP77. The vocabulary, the concept axis, the embedding, the query set and the refresh are fictional and are produced inside the lesson. There is no real institution, product, document or embedding endpoint.
• RP78. The knowledge freshness task: every procedure has an validity stamp. After the fine-tuning snapshot, the stamp changed in a part of the procedures. The right answer is the stamp in force.
• RP79. The format compliance task: the answer is required to fit a fixed pattern with two fields (procedure | stamp). This pattern is fictional and concerns only the output channel.
• RP80. The domain language task: finding which procedure the surface in the query language (which band am i in) corresponds to. This is the very same job that the previous lesson measured.
• RP81. Fine-tuning here is a learning simulator: for a procedure it has seen in the training set it resolves the query surface directly, in a class it has seen it applies the output pattern, and it gives the validity stamp from the snapshot. For a procedure it has not seen it falls back to the guess of the language model simulator.
• RP82. The cost is in two separate units and is not added up: for retrieval, extra tokens per request and the retrieval step; for fine-tuning, the number of labeled examples. The two have no common unit.
• RP83. The measures are taken over forty queries; retrieval brings three chunks.

Three Tasks, One Workload

The fictional refresh changes the validity stamp of a part of the procedures. The fine-tuning snapshot carries the old one, the corpus carries the new one.

# FICTIONAL. The corpus, the concept axis, the embedding, the query set and the refresh are
# produced inside the lesson; there is no real institution, product, document or embedding endpoint.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0      # the out-of-domain embedding does not recognize the query language
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# FICTIONAL refresh: a part of the procedures changed after the fine-tuning snapshot.
STAMP_TRAIN, STAMP_CURRENT = {}, {}
for i, k in enumerate(KEYS):
    STAMP_TRAIN[k] = f"y{i + 1:02d}a"
    STAMP_CURRENT[k] = (f"y{i + 1:02d}b"
                        if rng(SEED + 31 * (i + 1))() < 0.45 else f"y{i + 1:02d}a")
STALE = [k for k in KEYS if STAMP_TRAIN[k] != STAMP_CURRENT[k]]

CHUNKS = [{"code": d["code"], "main": d["main"],
           "text": p + " effective " + STAMP_CURRENT[d["main"]]}
          for d in DOCUMENTS for p in d["paras"]]
VECS = [embed(p["text"]) for p in CHUNKS]
TRAINING = [DOCUMENTS[int(rng(SEED + 65537 * (i + 1))() * len(DOCUMENTS))]["main"] for i in range(48)]

print(f"corpus {len(DOCUMENTS)} documents , {len(CHUNKS)} chunks , queries {len(QUERIES)}")
print(f"refreshed procedures {len(STALE)} / {len(KEYS)} , queries asking about a stale procedure "
      f"{sum(1 for s in QUERIES if s['concept'] in STALE)} / {len(QUERIES)}")
print(f"training set {len(TRAINING)} labeled examples , procedures covered "
      f"{len(set(TRAINING))} , classes covered {len({CONCEPTS[x][0] for x in set(TRAINING)})}")

corpus 200 documents , 693 chunks , queries 40
refreshed procedures 9 / 20 , queries asking about a stale procedure 13 / 40
training set 48 labeled examples , procedures covered 20 , classes covered 5

Nine of the twenty procedures were refreshed, and thirteen of the forty queries now ask for an answer other than the one fine-tuning knows. The training set is forty-eight labeled examples and it is fully covering: it sees all twenty of the twenty procedures and all five of the five classes. This is the best condition that can be given to fine-tuning.

Four Setups

The same forty queries are solved in four setups: the baseline without retrieval, retrieval alone, fine-tuning alone and the two together. The three tasks are three separate columns and are not gathered into a single number.

from math import exp


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def retrieve(q, k=3):
    v = embed(q)
    s = sorted(range(len(CHUNKS)), key=lambda i: (-cosine(v, VECS[i]), i))
    return [CHUNKS[i] for i in s[:k]]


def solve(s, retrieval, tuned, seed=SEED, temperature=0.6):
    """Retrieval changes the input of the prompt, fine-tuning changes the model itself."""
    context = retrieve(s["text"]) if retrieval else []
    prompt = s["text"] + " || " + " ".join(p["text"] for p in context)
    seen = set(TRAINING[:tuned])
    sq = " " + s["text"].lower() + " "
    concept = next((x for x in KEYS if CONCEPTS[x][2] in sq and x in seen), None)
    if concept is None:                  # fine-tuning has not seen this procedure: the simulator guesses
        dm = " " + prompt.lower() + " "
        label = next((CONCEPTS[x][0] for x in KEYS if CONCEPTS[x][2] in sq), None)
        sc = {x: (0.15 if CONCEPTS[x][0] == label else 0.0) + dm.count(CONCEPTS[x][1]) for x in KEYS}
        names = [x for x in KEYS if sc[x]]
        a = [exp(sc[x] / temperature) for x in names]
        t = sum(a)
        concept = names[pick(rng(seed ^ trace_id(prompt))(), [x / t for x in a])]
    if retrieval:
        b = context[0]["text"].split()
        stamp = b[b.index("effective") + 1]
    else:
        stamp = STAMP_TRAIN[concept] if tuned else "none"
    fmt_ok = tuned > 0 and CONCEPTS[concept][0] in {CONCEPTS[x][0] for x in seen}
    return concept, stamp, fmt_ok, len(tokenize(prompt))


def measure(retrieval, tuned, seed=SEED):
    n = len(QUERIES)
    t = d = b = tok = 0
    for s in QUERIES:
        concept, stamp, fmt, nt = solve(s, retrieval, tuned, seed)
        t += stamp == STAMP_CURRENT[s["concept"]]
        d += concept == s["concept"]
        b += fmt
        tok += nt
    return t / n, b / n, d / n, tok


SETUPS = (("no-retrieval baseline", 0, 0), ("retrieval only", 1, 0),
          ("fine-tuning only", 0, 48), ("both together", 1, 48))
print("setup                 freshness  format domain language    tokens retrieval steps labeled examples")
for name, e, i in SETUPS:
    t, b, d, tok = measure(e, i)
    print(f"{name:<21} {t:>9.4f} {b:>7.4f} {d:>15.4f} {tok:>9} {e * len(QUERIES):>15} {i:>16}")

print("\nfine-tuning example count sweep (no retrieval)")
print("examples seen concepts seen classes freshness  format domain language")
for i in (0, 6, 12, 24, 48):
    t, b, d, _ = measure(0, i)
    g = set(TRAINING[:i])
    print(f"{i:>8} {len(g):>13} {len({CONCEPTS[x][0] for x in g}):>12}"
          f" {t:>9.4f} {b:>7.4f} {d:>15.4f}")

print("\nsecond run (seed 20260219) , domain language hit rate and run-to-run difference")
for name, e, i in SETUPS:
    a, b = measure(e, i)[2], measure(e, i, SEED + 1)[2]
    print(f"{name:<21} {a:>7.4f} {b:>8.4f} {b - a:>+9.4f}")

setup                 freshness  format domain language    tokens retrieval steps labeled examples
no-retrieval baseline    0.0000  0.0000          0.3750       258               0                0
retrieval only           0.9500  0.0000          0.9500      2232              40                0
fine-tuning only         0.6750  1.0000          1.0000       258               0               48
both together            0.9500  1.0000          1.0000      2232              40               48

fine-tuning example count sweep (no retrieval)
examples seen concepts seen classes freshness  format domain language
       0             0            0    0.0000  0.0000          0.3750
       6             6            4    0.3750  0.7500          0.5750
      12             9            5    0.3750  1.0000          0.6250
      24            15            5    0.4000  1.0000          0.7250
      48            20            5    0.6750  1.0000          1.0000

second run (seed 20260219) , domain language hit rate and run-to-run difference
no-retrieval baseline  0.3750   0.3250   -0.0500
retrieval only         0.9500   0.9500   +0.0000
fine-tuning only       1.0000   1.0000   +0.0000
both together          1.0000   1.0000   +0.0000

The Task Retrieval Does Not Win

The three columns give three separate answers, and none covers another.

On knowledge freshness retrieval wins. Retrieval alone is 0.9500, fully covering fine-tuning is 0.6750. The gap is not chance: what is lost is exactly the thirteen queries that ask about the nine refreshed procedures. The sweep column confirms this. While the example count goes from six to forty-eight, domain language climbs from 0.5750 to 1.0000, but freshness stands still at 0.3750 (0.4000 at twenty-four examples) and tops out at 0.6750 only when the coverage is complete. More examples do not correct this, because staleness is not a coverage problem: the model remembers correctly what it learned correctly, and what it learned is no longer in force.

On format compliance retrieval gains nothing. The baseline without retrieval is 0.0000, retrieval alone is also 0.0000. The three retrieved chunks enrich the context and do not touch the output pattern. Fine-tuning makes the same task 1.0000 with twelve examples; what is needed to learn the pattern is not procedure coverage but class coverage, and five classes are complete at twelve examples. Retrieval changes the input of the prompt, fine-tuning changes the output behavior of the model; format is in the second set.

On domain language both win, and with full coverage fine-tuning comes out ahead. Retrieval is 0.9500, fine-tuning 1.0000. But the 1.0000 of fine-tuning depends on the forty-eight examples that see all twenty procedures; with twenty-four examples the same number is 0.7250. The 0.9500 of retrieval, on the other hand, asks for no labeled example.

Because the costs carry no common unit, they are written separately. Retrieval takes the total tokens from 258 to 2232 and adds forty retrieval steps. The tokens of fine-tuning stay at 258 and there is no retrieval step, but it asks for forty-eight labeled examples, and each time the stamp is refreshed a part of these examples becomes invalid. The two together give the best column on all three tasks and pay both costs at once.

The difference between two runs is in the last table. The baseline without retrieval falls from 0.3750 to 0.3250 in the second run (−0.0500); a number that does not rest on information is the only one that moves. In the other three setups the difference is 0.0000. Both the evidence in the context and the learned mapping prevent sampling from turning the result.

Summary

• Retrieval changes the input of the prompt, fine-tuning changes the output behavior of the model; this distinction decides which task is solved with which.
• On knowledge freshness retrieval makes 0.9500 and fully covering fine-tuning makes 0.6750; what is lost is exactly the thirteen queries that ask about the nine refreshed procedures.
• Increasing the example count does not correct freshness: while domain language goes from 0.5750 to 1.0000, freshness stays at 0.3750 and tops out at 0.6750.
• On format compliance retrieval adds nothing from 0.0000 to 0.0000; fine-tuning makes 1.0000 with twelve examples, because the pattern needs not procedure but class coverage.
• The costs are not added up: retrieval takes 258 tokens to 2232 and adds forty retrieval steps, and fine-tuning keeps the tokens fixed and asks for forty-eight labeled examples.

Next Step

Both solutions share an assumption: that everything in the corpus is plain text. The fictional documents were built entirely of sentences, and chunking meant cutting a sequence of sentences. The real procedure documents of the field team are not like that; inside them are tables whose rows and columns carry the meaning. When a table is cut like a sentence, the row and the header are separated from each other and the chunk alone says nothing. The next lesson measures this on the same query set: chunking the table row by row and block by block does not give the same recall.