Reranking

The previous lesson corrected the query before sending it to the index and counted how good the top five chunks are together. What remained is the distribution inside those five: why the head of the retrieved set is not full, and what it costs to fix the head.

Half of the answer lies in the score itself. The first stage compresses the query into a vector without seeing the chunk; this vector can say “search for this concept”, but it cannot say “search for this concept and this record type together”, because a single similarity number does not carry the intersection of two conditions. The second stage reads the query and the chunk together and can search for the two conditions separately. The cost is plain: one call for every candidate.

• RP47. The code is a simulator; no real embedding, ranker or model endpoint is called. The corpus, the concepts and the queries are fictional, seed 20260218.
• RP48. The definition of the target continues from retrieval-pipeline/05: the target of a query is both a concept and a record type, and a chunk is relevant only if it carries both. The pipeline is the same too.
• RP49. This lesson closes the transformation step and takes the query set in the form in which it was defined; the only setup measured is reranking. The cost of transformation was measured in 05.
• RP50. The reranker is one model call per candidate and sees the query and the chunk in a single prompt. Its score is the sum of two conditions: concept match and record type match.
• RP51. The reranker is also a model, and its score is noisy. The amplitude of the noise exceeds the weight of one condition, so it cannot always see a difference of a single condition.
• RP52. The number of chunks that enter the context is fixed at three; k itself and the context limit are the subject of retrieval-pipeline/07.
• RP53. Latency is written as a class. Because the candidates are scored independently of one another they can be called in parallel, but the order of the stages is mandatory: the second cannot start before the first finishes.

The Limit of a Single Stage

The block below sets up the pipeline and prints what recall and precision do as k grows in single-stage retrieval. The two columns go in opposite directions; the tension between them is the reason for this lesson.

# SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus,
# concepts and queries are fictional, seed 20260218.
import math
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


def text_of(d):
    return d["title"] + " " + " ".join(d["paras"])


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0      # the out-of-domain embedding does not recognize the query language
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# ---- pipeline: chunks, hybrid retrieval, target definition and generation
CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]]
for p in CHUNKS:
    p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"]))
for s in QUERIES:
    s["record"] = s["text"].split()[-1]


def retrieve(q, k=5, w=0.5):
    qv, qb = embed(q), set(tokenize(q))
    s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i)
                for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1]))
    return [CHUNKS[i] for _, i in s[:k]]


def relevant(s, p):
    return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"]


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


WINDOW = 240
COUNTER = [0]
ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . "


def answer(query, chunks, seed=SEED, temperature=0.12):
    """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight."""
    COUNTER[0] += 1
    prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query
    toks = tokenize(prompt)
    if len(toks) > WINDOW:
        return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None}
    r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks)
    score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb))
             for i, x in enumerate(chunks)]
    w = [math.exp(a / temperature) for a in score]
    chosen = chunks[pick(r(), [a / sum(w) for a in w])]
    return {"status": "ok", "input": len(toks), "chunk": chosen,
            "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))}


print("corpus %d documents %d chunks ; target chunk set average %.1f chunks"
      % (len(DOCUMENTS), len(CHUNKS),
         sum(len([p for p in CHUNKS if relevant(s, p)]) for s in QUERIES) / 40))
print("single-stage retrieval:  k  recall@k  precision@k")
for k in (1, 3, 5, 10, 20, 40):
    a = sum(any(relevant(s, p) for p in retrieve(s["text"], k)) for s in QUERIES) / 40
    e = sum(sum(relevant(s, p) for p in retrieve(s["text"], k)) / k for s in QUERIES) / 40
    print("                        %2d  %8.4f  %.4f" % (k, a, e))

corpus 200 documents 693 chunks ; target chunk set average 8.9 chunks
single-stage retrieval:  k  recall@k  precision@k
                         1    0.7750  0.7750
                         3    0.7750  0.4167
                         5    0.8000  0.2700
                        10    0.9750  0.2425
                        20    1.0000  0.2425
                        40    1.0000  0.1831

Recall needs twenty chunks to catch all forty queries. Precision says the opposite: when k is twenty, only 0.2425 of the top twenty is relevant, that is, about three wrong chunks enter the context next to every correct chunk. At k of three precision is 0.4167, but recall stays at 0.7750 and a quarter of the queries never see a correct chunk.

There is no way to improve these two columns at the same time in a single-stage pipeline, because the order comes out of a single score. Reranking suggests this: take recall from a wide candidate set, take precision from a second scorer.

The Second Stage

The block below defines the ranker and runs the same forty queries five times, raising the candidate count from three to forty-eight. The number of chunks that enter the context is always three; the only thing that changes is how many candidates those three are chosen from.

RERANK_INSTRUCTION = "Is the query about the same record as the chunk . Chunk: "
TRACE = {}                           # the tokens of both languages are tied to the same concept
for k in KEYS:
    for m in (CONCEPTS[k][1], CONCEPTS[k][2]):
        for t in tokenize(m):
            if len(t) > 3:
                TRACE.setdefault(t, k)


def score_pair(q, p, seed=SEED, noise=1.4):
    """FICTIONAL second stage: it sees the query and the chunk TOGETHER and searches for the two conditions separately."""
    COUNTER[0] += 1
    toks = tokenize(RERANK_INSTRUCTION + p["text"] + " Query: " + q)
    r, qb = rng(seed ^ trace_id(q + " | " + p["text"])), set(tokenize(q))
    concept_match = bool({TRACE[t] for t in qb if t in TRACE} & {TRACE[t] for t in p["bag"] if t in TRACE})
    record_match = bool(qb & p["bag"] & set(COMMON))
    return 1.0 * concept_match + 1.0 * record_match + noise * (r() - 0.5), len(toks) + 1


def measure(n, seed=SEED, k=3):
    COUNTER[0] = 0
    rec = prec = gen = calls = toks = 0
    for s in QUERIES:
        candidates = retrieve(s["text"], max(n, k))
        if n > k:
            scores = []
            for p in candidates:
                v, b = score_pair(s["text"], p, seed)
                scores.append(v)
                calls, toks = calls + 1, toks + b
            candidates = [p for _, p in sorted(zip(scores, candidates), key=lambda t: -t[0])]
        top = candidates[:k]
        rec += any(relevant(s, p) for p in top)
        prec += sum(relevant(s, p) for p in top) / k
        y = answer(s["text"], top, seed)
        calls, toks = calls + 1, toks + y["input"] + y["output"]
        gen += relevant(s, y["chunk"])
    return rec / 40, prec / 40, gen / 40, calls, toks


print("candidates  recall@3  precision@3  generation  calls    tokens  latency class")
previous, gap = None, [0.0, 0.0, 0.0]
for n in (3, 6, 12, 24, 48):
    a, e, u, c, b = measure(n)
    i = measure(n, SEED + 1)
    gap = [max(f, abs(x - y)) for f, x, y in zip(gap, (a, e, u), i[:3])]
    print("%10d  %8.4f  %11.4f  %10.4f  %5d  %8d  %s"
          % (n, a, e, u, c, b, "single stage" if n == 3 else "two stages, %d extra calls" % n))
print("run-to-run difference (seed 20260219): recall %.4f  precision %.4f  generation %.4f" % tuple(gap))

candidates  recall@3  precision@3  generation  calls    tokens  latency class
         3    0.7750       0.4167      0.7000     40      3654  single stage
         6    0.8000       0.4917      0.7250    280     12226  two stages, 6 extra calls
        12    0.9750       0.8250      0.9250    520     20796  two stages, 12 extra calls
        24    1.0000       0.9000      1.0000   1000     38003  two stages, 24 extra calls
        48    0.9500       0.9083      0.9500   1960     71797  two stages, 48 extra calls
run-to-run difference (seed 20260219): recall 0.0000  precision 0.0250  generation 0.1000

As the Candidate Count Grows

The first row is the baseline: no reranking, the first three go straight into the context, precision is 0.4167, generation is 0.7000, forty calls.

With six candidates precision rises to 0.4917. The gain is 0.0750 and stands above the run-to-run difference (0.0250), so it is real; but in return calls go from forty to two hundred eighty and tokens from 3654 to 12226. Generation rises to 0.7250 in this row; the gain is 0.0250, and because the run-to-run difference of generation is 0.1000, this row cannot be read as “it raised generation”; it is read as this measurement cannot tell.

With twelve candidates the table changes: precision 0.8250, recall 0.9750, generation 0.9250. Against the baseline there is a gain of 0.4083 in precision and 0.2250 in generation, and both are far above the threshold. With twenty-four candidates recall becomes 1.0000, precision 0.9000 and generation 1.0000; this setup puts the correct chunk into the context and into the source of the answer for all forty queries.

On the cost side the same row writes one thousand calls and 38003 tokens. The baseline was forty calls; that is twenty-five times. The latency class has changed too: the pipeline went from a single stage to two stages, and the second stage scales with the candidate count. The candidates can be scored in parallel, but none of them can start before the first stage finishes, that is, while the length of the chain stays fixed at two steps, its width goes up to twenty-four.

The Gain That Cannot Be Measured and the Loss That Is Measured

The last row is the real finding of this lesson. When the candidate count is raised from twenty-four to forty-eight, precision goes from 0.9000 to 0.9083. The gain is 0.0083, a third of the run-to-run difference. This row cannot be written as “it increased precision”; whether it increased it cannot be known with this measurement. What is paid in return, on the other hand, has been measured: calls go from 1000 to 1960, tokens from 38003 to 71797.

There is one more thing that can be measured in the same row, and its direction is the opposite. The recall of the top three falls from 1.0000 to 0.9500; the drop is 0.0500 and the run-to-run difference of recall is 0.0000, so it is real. Generation also goes down from 1.0000 to 0.9500, although that drop stays below the run-to-run difference of generation (0.1000). As the candidate set grows, more wrong chunks come in front of the noisy ranker and the chance that it carries one of them into the top three goes up. Because the noise of the ranker can mask a difference of a single condition, a wrong candidate passing a correct one becomes unavoidable in a crowd.

This is the instance, in this lesson, of the course’s second claim: enlarging the retrieval machinery made the job worse. The right reading is not to pick the row with the highest precision but to find the place where the gain falls below the threshold and to stop there. In this workload that place is twenty-four candidates.

Summary

• Because a single similarity score compresses the query without seeing the chunk, it cannot search for the concept and the record type together; recall asks for twenty chunks, while precision at the top three stays at 0.4167.
• The second stage, which reads the candidates together with the query, raises precision to 0.8250 with twelve candidates and to 0.9000 with twenty-four, and generation from 0.7000 to 1.0000.
• The cost is one call per candidate: forty calls and 3654 tokens become 1000 calls and 38003 tokens with twenty-four candidates; the latency class goes from a single stage to two stages.
• With forty-eight candidates precision rises by only 0.0083 and stays below the run-to-run difference of 0.0250; this row cannot be read as having gained.
• In the same row the recall of the top three falls from 1.0000 to 0.9500, and this drop is above the threshold of 0.0000: enlarging the candidate set presents the noisy ranker with more wrong candidates.

Next Step

In all the setups of this lesson three chunks entered the context, and always in the same order, that is, in the order the ranker gave. Two numbers show that this choice has not been questioned: where those three chunks stand against the limit of the context, and whether the order of those three changes the answer. The next lesson counts how many chunks fit into the context limit, shows what the pipeline does when the limit is exceeded, and measures the share of change in the answer by giving the same chunks in different orders.