--- title: Reranking source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/reranking' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Shows what the fact that a single similarity score cannot search for two conditions at the same time does to precision at the top k.' - 'Builds reranking as one model call per candidate and writes the calls, the tokens and the latency class separately.' - 'Computes where, as the candidate count grows, the gain in the precision of the top three falls below the run-to-run difference.' - 'Defines the setup in which enlarging the candidate set lowers the recall of the top three.' --- # Reranking Splitting retrieval into two stages and counting side by side what the second stage gains and what it makes you pay: in the single-stage pipeline the precision of the top three stays at 0.4167, because a single similarity score cannot search for the concept and the record type of the query at the same time. The second stage, which reads the candidates together with the query, raises precision to 0.8250 with twelve candidates and to 0.9000 with twenty-four, and takes generation from 0.7000 to 1.0000. With forty-eight candidates precision rises by only 0.0083, that is, it stays below the run-to-run difference of 0.0250; in return the recall of the top three falls from 1.0000 to 0.9500, the call count goes from forty to one thousand nine hundred sixty, and tokens go from 3654 to 71797. The previous lesson corrected the query before sending it to the index and counted how good the top five chunks are **together**. What remained is the distribution inside those five: why the **head** of the retrieved set is not full, and what it costs to fix the head. Half of the answer lies in the score itself. The first stage compresses the query into a vector **without seeing the chunk**; this vector can say "search for this concept", but it cannot say "search for this concept **and** this record type together", because a single similarity number does not carry the intersection of two conditions. The **second stage** reads the query and the chunk **together** and can search for the two conditions separately. The cost is plain: one call for every candidate. - **RP47.** The code is a **simulator**; no real embedding, ranker or model endpoint is called. The corpus, the concepts and the queries are **fictional**, seed **20260218**. - **RP48.** The definition of the target continues from `retrieval-pipeline/05`: the target of a query is both a **concept** and a **record type**, and a chunk is relevant only if it carries both. The pipeline is the same too. - **RP49.** This lesson **closes the transformation step** and takes the query set in the form in which it was defined; the only setup measured is reranking. The cost of transformation was measured in `05`. - **RP50.** The reranker is **one model call per candidate** and sees the query and the chunk in a single prompt. Its score is the sum of two conditions: concept match and record type match. - **RP51.** The reranker is also a model, and its score is **noisy**. The amplitude of the noise exceeds the weight of one condition, so it cannot always see a difference of a single condition. - **RP52.** The number of chunks that enter the context is **fixed at three**; k itself and the context limit are the subject of `retrieval-pipeline/07`. - **RP53.** Latency is written as a **class**. Because the candidates are scored independently of one another they can be called in parallel, but the order of the stages is mandatory: the second cannot start before the first finishes. ## The Limit of a Single Stage The block below sets up the pipeline and prints what recall and precision do as k grows in single-stage retrieval. The two columns go in **opposite directions**; the tension between them is the reason for this lesson. ```python # SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus, # concepts and queries are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def text_of(d): return d["title"] + " " + " ".join(d["paras"]) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- pipeline: chunks, hybrid retrieval, target definition and generation CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]] for p in CHUNKS: p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"])) for s in QUERIES: s["record"] = s["text"].split()[-1] def retrieve(q, k=5, w=0.5): qv, qb = embed(q), set(tokenize(q)) s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i) for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1])) return [CHUNKS[i] for _, i in s[:k]] def relevant(s, p): return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h WINDOW = 240 COUNTER = [0] ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . " def answer(query, chunks, seed=SEED, temperature=0.12): """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight.""" COUNTER[0] += 1 prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None} r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks) score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb)) for i, x in enumerate(chunks)] w = [math.exp(a / temperature) for a in score] chosen = chunks[pick(r(), [a / sum(w) for a in w])] return {"status": "ok", "input": len(toks), "chunk": chosen, "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))} print("corpus %d documents %d chunks ; target chunk set average %.1f chunks" % (len(DOCUMENTS), len(CHUNKS), sum(len([p for p in CHUNKS if relevant(s, p)]) for s in QUERIES) / 40)) print("single-stage retrieval: k recall@k precision@k") for k in (1, 3, 5, 10, 20, 40): a = sum(any(relevant(s, p) for p in retrieve(s["text"], k)) for s in QUERIES) / 40 e = sum(sum(relevant(s, p) for p in retrieve(s["text"], k)) / k for s in QUERIES) / 40 print(" %2d %8.4f %.4f" % (k, a, e)) ``` ``` corpus 200 documents 693 chunks ; target chunk set average 8.9 chunks single-stage retrieval: k recall@k precision@k 1 0.7750 0.7750 3 0.7750 0.4167 5 0.8000 0.2700 10 0.9750 0.2425 20 1.0000 0.2425 40 1.0000 0.1831 ``` Recall needs **twenty** chunks to catch all forty queries. Precision says the opposite: when k is twenty, only **0.2425** of the top twenty is relevant, that is, about three wrong chunks enter the context next to every correct chunk. At k of three precision is **0.4167**, but recall stays at **0.7750** and a quarter of the queries never see a correct chunk. There is no way to improve these two columns at the same time in a single-stage pipeline, because the order comes out of a single score. Reranking suggests this: **take recall from a wide candidate set, take precision from a second scorer.** ## The Second Stage The block below defines the ranker and runs the same forty queries five times, raising the candidate count from three to forty-eight. The number of chunks that enter the context is always three; the only thing that changes is **how many candidates** those three are chosen from. ```python RERANK_INSTRUCTION = "Is the query about the same record as the chunk . Chunk: " TRACE = {} # the tokens of both languages are tied to the same concept for k in KEYS: for m in (CONCEPTS[k][1], CONCEPTS[k][2]): for t in tokenize(m): if len(t) > 3: TRACE.setdefault(t, k) def score_pair(q, p, seed=SEED, noise=1.4): """FICTIONAL second stage: it sees the query and the chunk TOGETHER and searches for the two conditions separately.""" COUNTER[0] += 1 toks = tokenize(RERANK_INSTRUCTION + p["text"] + " Query: " + q) r, qb = rng(seed ^ trace_id(q + " | " + p["text"])), set(tokenize(q)) concept_match = bool({TRACE[t] for t in qb if t in TRACE} & {TRACE[t] for t in p["bag"] if t in TRACE}) record_match = bool(qb & p["bag"] & set(COMMON)) return 1.0 * concept_match + 1.0 * record_match + noise * (r() - 0.5), len(toks) + 1 def measure(n, seed=SEED, k=3): COUNTER[0] = 0 rec = prec = gen = calls = toks = 0 for s in QUERIES: candidates = retrieve(s["text"], max(n, k)) if n > k: scores = [] for p in candidates: v, b = score_pair(s["text"], p, seed) scores.append(v) calls, toks = calls + 1, toks + b candidates = [p for _, p in sorted(zip(scores, candidates), key=lambda t: -t[0])] top = candidates[:k] rec += any(relevant(s, p) for p in top) prec += sum(relevant(s, p) for p in top) / k y = answer(s["text"], top, seed) calls, toks = calls + 1, toks + y["input"] + y["output"] gen += relevant(s, y["chunk"]) return rec / 40, prec / 40, gen / 40, calls, toks print("candidates recall@3 precision@3 generation calls tokens latency class") previous, gap = None, [0.0, 0.0, 0.0] for n in (3, 6, 12, 24, 48): a, e, u, c, b = measure(n) i = measure(n, SEED + 1) gap = [max(f, abs(x - y)) for f, x, y in zip(gap, (a, e, u), i[:3])] print("%10d %8.4f %11.4f %10.4f %5d %8d %s" % (n, a, e, u, c, b, "single stage" if n == 3 else "two stages, %d extra calls" % n)) print("run-to-run difference (seed 20260219): recall %.4f precision %.4f generation %.4f" % tuple(gap)) ``` ``` candidates recall@3 precision@3 generation calls tokens latency class 3 0.7750 0.4167 0.7000 40 3654 single stage 6 0.8000 0.4917 0.7250 280 12226 two stages, 6 extra calls 12 0.9750 0.8250 0.9250 520 20796 two stages, 12 extra calls 24 1.0000 0.9000 1.0000 1000 38003 two stages, 24 extra calls 48 0.9500 0.9083 0.9500 1960 71797 two stages, 48 extra calls run-to-run difference (seed 20260219): recall 0.0000 precision 0.0250 generation 0.1000 ``` ## As the Candidate Count Grows The first row is the baseline: no reranking, the first three go straight into the context, precision is **0.4167**, generation is **0.7000**, forty calls. With six candidates precision rises to **0.4917**. The gain is 0.0750 and stands above the run-to-run difference (0.0250), so it is real; but in return calls go **from forty to two hundred eighty** and tokens from 3654 to 12226. Generation rises to **0.7250** in this row; the gain is 0.0250, and because the run-to-run difference of generation is 0.1000, this row cannot be read as "it raised generation"; it is read as **this measurement cannot tell**. With twelve candidates the table changes: precision **0.8250**, recall **0.9750**, generation **0.9250**. Against the baseline there is a gain of 0.4083 in precision and 0.2250 in generation, and both are far above the threshold. With twenty-four candidates recall becomes **1.0000**, precision **0.9000** and generation **1.0000**; this setup puts the correct chunk into the context and into the source of the answer for all forty queries. On the cost side the same row writes **one thousand calls** and **38003 tokens**. The baseline was forty calls; that is twenty-five times. The latency class has changed too: the pipeline went from a single stage to **two stages**, and the second stage scales with the candidate count. The candidates can be scored in parallel, but none of them can start before the first stage finishes, that is, while the length of the chain stays fixed at two steps, its **width** goes up to twenty-four. ## The Gain That Cannot Be Measured and the Loss That Is Measured The last row is the real finding of this lesson. When the candidate count is raised from twenty-four to forty-eight, precision goes from 0.9000 to **0.9083**. The gain is **0.0083**, **a third** of the run-to-run difference. This row cannot be written as "it increased precision"; **whether it increased it cannot be known with this measurement**. What is paid in return, on the other hand, has been measured: calls go from 1000 to **1960**, tokens from 38003 to **71797**. There is one more thing that can be measured in the same row, and its direction is the opposite. The recall of the top three falls **from 1.0000 to 0.9500**; the drop is 0.0500 and the run-to-run difference of recall is 0.0000, so it is **real**. Generation also goes down from 1.0000 to 0.9500, although that drop stays below the run-to-run difference of generation (0.1000). As the candidate set grows, more wrong chunks come in front of the noisy ranker and the chance that it carries one of them into the top three goes up. Because the noise of the ranker can mask a difference of a single condition, a wrong candidate passing a correct one becomes unavoidable in a crowd. This is the instance, in this lesson, of the course's second claim: **enlarging the retrieval machinery made the job worse**. The right reading is not to pick the row with the highest precision but to find the place where the gain falls below the threshold and to stop there. In this workload that place is **twenty-four candidates**. ## Summary - Because a single similarity score compresses the query without seeing the chunk, it cannot search for the concept and the record type together; recall asks for twenty chunks, while precision at the top three stays at 0.4167. - The second stage, which reads the candidates together with the query, raises precision to 0.8250 with twelve candidates and to 0.9000 with twenty-four, and generation from 0.7000 to 1.0000. - The cost is one call per candidate: forty calls and 3654 tokens become 1000 calls and 38003 tokens with twenty-four candidates; the latency class goes from a single stage to two stages. - With forty-eight candidates precision rises by only 0.0083 and stays below the run-to-run difference of 0.0250; this row cannot be read as having gained. - In the same row the recall of the top three falls from 1.0000 to 0.9500, and this drop is above the threshold of 0.0000: enlarging the candidate set presents the noisy ranker with more wrong candidates. ## Next Step In all the setups of this lesson **three** chunks entered the context, and always in the same order, that is, in the order the ranker gave. Two numbers show that this choice has not been questioned: where those three chunks stand against the limit of the context, and whether the **order** of those three changes the answer. The next lesson counts how many chunks fit into the context limit, shows what the pipeline does when the limit is exceeded, and measures the share of change in the answer by giving the same chunks in different orders.