--- title: 'Hybrid Search' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/hybrid-search' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures keyword and embedding search separately on three fictional query classes and counts which one collapses in which class.' - 'Shows that rank fusion changes the top-one hit rate and the recall up to rank twenty in opposite directions.' - 'Finds the query class in which the fusion stays worse than both setups and explains the cause with the overlap count of the two lists.' --- # Hybrid Search Measuring keyword matching and embedding search separately and together. Three setups run side by side on three fictional query classes: in subscriber language keyword search is 0.0250 and embedding is 0.5000; in document language both are 0.5000; on the document code query keyword search is 1.0000 and embedding is 0.0000. Rank fusion gives 0.4000 in the document language class, that is, worse than both; this holds at constants of 5, 20 and 60. In the code class fusion drops the share of keeping the target document in the first five from 1.0000 to 0.0500. The reason is the number of documents shared by the first five of the two lists: 0.12 in subscriber language, 1.05 in document language. A document that stands second in both lists passes a document that is first in one and absent from the other. Fusion raises recall up to rank twenty from 0.7122 to 0.7214; the place it gains is depth, the place it loses is the first rank. The previous lesson built the link between the query and the document from a single place: the angle of the embedding vectors. The filter narrowed this link, it did not change it. Yet what is searched in a support corpus is not always a meaning. When a document code, an error number or the exact name of a procedure is searched, the question asked is not "what is similar to this" but "where exactly does this occur". Semantic similarity passes by the side of this question. **Hybrid search** is running two rankings together for the same query and gathering their results into a single list. The expectation is that the whole is better than its parts: let keyword search catch the exact match that the embedding cannot see, and let the embedding catch the synonym that keyword search cannot see. This lesson measures three setups (keyword only, embedding only, combined) side by side on three query classes. One of the results does not meet the expectation: in one query class the combined setup stays worse than **both procedures**. - **EV48.** The corpus, the embedding procedure, the tokenizer and the query classes are **fictional**; no real search engine, vector store or embedding endpoint is called. The seed is **20260218**. - **EV49.** The **searchable text** of the document is kept together with its code. Because the code is not a concept surface form, it does not change the embedding vector; this is shown in the block. - **EV50. The three query classes** are derived from the same documents and differ only in their surface forms: **subscriber language** carries the query-side form, **document language** carries the document-side form, and the **code query** carries the document code. All three are forty queries. - **EV51. The keyword ranking** is the number of overlapping tokens. Relevance scoring was built and measured in the Search Engines and Text Retrieval course of the Databases curriculum; it is not repeated here. In this lesson the only function of the keyword side is to produce **a second ranking**. - **EV52. Fusion is rank-based**: the score of a document is the sum of the inverses of its ranks in the two lists, `1 / (constant + rank)`. The constant is 20 by default and is also tested at 5 and 60. - **EV53. The top-one hit rate** is the share of the first result that carries the target concept, **retrieval precision** is the same share for the first five results, and **retrieval recall** is the share of the documents carrying the target concept that are caught in the first twenty. The **target document** column is the share of the document from which the query was derived being in the first five. - **EV54.** Every concept is carried by more than one document; for this reason precision and recall are read at different depths and are not reduced to a single number. - **EV55.** All three rankings are **deterministic**: the same query gives the same list in two runs. The run-independent measure of this lesson is **the number of documents shared by the first five of the two lists**, and it is a property of the pair of procedures, independent of the query. - **EV56.** A query that carries no concept surface form produces a **zero vector**; all similarities become zero and the order falls to the storage order. This is not a defect but the embedding leaving its domain of definition, and it is measured in the code class. - **EV57.** This lesson is the close of the topic; what the retrieved chunk does to the model is not measured and is left to the next topic. ## Three Query Classes ```python # FICTIONAL corpus, FICTIONAL embedding simulator, FICTIONAL query classes. No real endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) def _to_concepts(m): # the surface form is first mapped to a concept v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): if CONCEPTS[k][1] in dm: v[i] += 1.0 if CONCEPTS[k][2] in dm: v[i] += 1.0 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM): # the concept is then reduced to fewer dimensions v = [0.0] * dim for i, a in enumerate(_to_concepts(m)): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) K, CONSTANT = 5, 20 SEARCHABLE = [d["code"] + " " + doc_text(d) for d in DOCUMENTS] # the code enters the searchable text of the document VEC = [embed(a) for a in SEARCHABLE] TOKEN_SETS = [set(tokenize(a)) for a in SEARCHABLE] def queries(mode): # three separate surface forms from the same documents q = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] o = COMMON[int(r() * len(COMMON))] q.append({"text": {"subscriber": f"{CONCEPTS[d['main']][2]} {o}", "document": f"{CONCEPTS[d['main']][1]} {o}", "code": f"{d['code']} numbered document"}[mode], "concept": d["main"], "code": d["code"]}) return q QUERY_CLASSES = {k: queries(k) for k in ("subscriber", "document", "code")} def keyword_rank(m): # by the number of overlapping tokens qb = set(tokenize(m)) return sorted(range(len(DOCUMENTS)), key=lambda i: -len(qb & TOKEN_SETS[i])) def embedding_rank(m): qv = embed(m) return sorted(range(len(DOCUMENTS)), key=lambda i: -cosine(qv, VEC[i])) def hybrid_rank(m, constant=CONSTANT): # rank fusion pk = {i: 1.0 / (constant + n) for n, i in enumerate(keyword_rank(m))} pe = {i: 1.0 / (constant + n) for n, i in enumerate(embedding_rank(m))} return sorted(range(len(DOCUMENTS)), key=lambda i: -(pk[i] + pe[i])) for k, s in QUERY_CLASSES.items(): print(f"{k:10s} example query: {s[0]['text']}") print(f"document code does not change the embedding: " f"{embed(SEARCHABLE[0]) == embed(doc_text(DOCUMENTS[0]))} , code query vector length " f"{sum(x * x for x in embed(QUERY_CLASSES['code'][0]['text'])):.1f}") ``` ``` subscriber example query: how it gets saved form document example query: reading procedure form code example query: SY194 numbered document document code does not change the embedding: True , code query vector length 0.0 ``` The three queries derived from the same document and ask the same thing; the only place they differ is the surface form. The last line measures two things. Adding the document code to the searchable text **does not change** the embedding vector: the code is not a concept surface form, so the embedding does not see it. The second is harsher: the vector length of the code query is **0.0**. Because there is no recognized concept in the query, the vector is zero, all the cosine similarities come out zero and the ranking falls to the storage order. The embedding is not mistaken on this query, it stays **outside its domain of definition**. ## The Table of the Three Setups ```python def measure(qset, rank): first = prec = rec = target = 0.0 for q in qset: relevant = {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]} s = rank(q["text"]) first += s[0] in relevant prec += len(set(s[:K]) & relevant) / K rec += len(set(s[:20]) & relevant) / len(relevant) target += any(DOCUMENTS[i]["code"] == q["code"] for i in s[:K]) return [x / len(qset) for x in (first, prec, rec, target)] SETUPS = (("keyword", keyword_rank), ("embedding", embedding_rank), ("hybrid", hybrid_rank)) print("class setup top1 precision@5 recall@20 target document@5") for mode, qset in QUERY_CLASSES.items(): for name, f in SETUPS: d = measure(qset, f) print(f"{mode:10s} {name:9s} {d[0]:>6.4f} {d[1]:>11.4f} {d[2]:>9.4f} {d[3]:>17.4f}") print("hybrid top1 in the document class , constant 5 / 20 / 60: " + " / ".join(f"{measure(QUERY_CLASSES['document'], lambda m, c=c: hybrid_rank(m, c))[0]:.4f}" for c in (5, 20, 60))) ``` ``` class setup top1 precision@5 recall@20 target document@5 subscriber keyword 0.0250 0.0800 0.0967 0.0250 subscriber embedding 0.5000 0.4700 0.7122 0.1500 subscriber hybrid 0.4250 0.2900 0.4616 0.1250 document keyword 0.5000 0.3650 0.6616 0.1750 document embedding 0.5000 0.4700 0.7122 0.1500 document hybrid 0.4000 0.4250 0.7214 0.1250 code keyword 1.0000 0.2500 0.1950 1.0000 code embedding 0.0000 0.0800 0.1078 0.0250 code hybrid 0.0000 0.0850 0.1950 0.0500 hybrid top1 in the document class , constant 5 / 20 / 60: 0.4500 / 0.4000 / 0.4000 ``` The first two triples build the justification for hybrid search. In subscriber language keyword search is 0.0250 and embedding is 0.5000; on the code query keyword search is **1.0000** and embedding is **0.0000**. The two procedures work in exactly opposite places on the same corpus, and choosing a single procedure means sacrificing one of the query classes entirely. The hybrid rows show that this sacrifice does not go away. In subscriber language the hybrid is 0.4250, a little below the embedding; this is the expected cost. In document language, while both procedures give 0.5000, the hybrid gives **0.4000**. The total is worse than **both** of its parts. The last line says that this is not the game of one choice of constant: at 5, 20 and 60 it is 0.4500 / 0.4000 / 0.4000, and all three are below 0.5000. The code class is the most expensive row. Keyword search puts the target document in the first five in forty of the forty queries; the hybrid drops this to **0.0500**. In the query class that is the reason hybrid search exists, fusion sinks the working procedure together with the one that does not work. The cause is in the table itself: the embedding side works with a zero vector in this class and the order it produces carries no information, but the fusion does not know this and gives that order **equal weight**. One column looks in the opposite direction. In document language recall@20 is 0.6616 for keyword search, 0.7122 for the embedding, and **0.7214** for the hybrid. Up to rank twenty the fusion really does gain. In the code class too, recall@20 rises from 0.1078 to 0.1950. The place where fusion gains is **depth**, the place where it loses is the **first rank**, and in a retrieval pipeline what enters the prompt is the first few chunks. ## What the Fusion Loses ```python print("class shared in first five both correct hybrid wrong") for mode, qset in QUERY_CLASSES.items(): shared = lost = 0 for q in qset: relevant = {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]} kw, em, hy = (f(q["text"]) for _, f in SETUPS) shared += len(set(kw[:K]) & set(em[:K])) lost += kw[0] in relevant and em[0] in relevant and hy[0] not in relevant print(f"{mode:10s} {shared / len(qset):>20.2f} {lost:>25}") def relevant_docs(q): return {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]} q = next(x for x in QUERY_CLASSES["document"] if keyword_rank(x["text"])[0] in relevant_docs(x) and embedding_rank(x["text"])[0] in relevant_docs(x) and hybrid_rank(x["text"])[0] not in relevant_docs(x)) print(f"\nquery: {q['text']} -- target concept: {q['concept']}") for name, f in SETUPS: print(f" {name:9s}", [f"{DOCUMENTS[i]['code']}/{DOCUMENTS[i]['main']}" for i in f(q["text"])[:3]]) ``` ``` class shared in first five both correct hybrid wrong subscriber 0.12 0 document 1.05 3 code 4.03 0 query: pressure drop region -- target concept: pressure keyword ['AR002/pressure', 'AR015/crew', 'AR018/pressure'] embedding ['AR079/pressure', 'AR088/pressure', 'AR101/crew'] hybrid ['AR033/crew', 'AR015/crew', 'AR018/pressure'] ``` The shared documents column is a number that belongs not to any query but to the pair of procedures itself, and it is independent of the run. In subscriber language the first five of the two lists share **0.12** documents on average: the procedures almost never agree. In document language it is 1.05, that is, one in five. The example shows the consequence of this number. Keyword search puts AR002 and the embedding puts AR079 at the first rank; both carry the target concept, so **both procedures are correct**. The hybrid puts AR033 at the first rank, and that document belongs to the `crew` concept. AR002 is first in one list and far behind in the other; its score comes from a single large term. AR033 is high in both lists but first in neither, and the sum of two medium terms passes one large term. Rank fusion rewards **the document of compromise**; when the two sides do not agree, the second choice of both wins. In the document language class this happens in three of the forty queries and never in the opposite direction: there is not a single example in which the hybrid finds the right one on a query where both procedures were wrong. The counterpart of the three-query drop in the first rank is zero gain. The 4.03 in the code row is the trap. The high overlap here is not agreement: because the embedding side works with a zero vector, its list is the storage order, and the tail of the keyword side after the exact match is the same order. The two lists share **the same uninformed tail**. Fusion cannot tell apart the two lists being the same from both being empty; if it could, it would give keyword search full weight in this class. Hybrid search asks not for an average but for a **decision**: which side has the say on which query. ## Summary - On three query classes the two procedures work in opposite places: in subscriber language keyword search is 0.0250 and embedding is 0.5000, on the code query keyword search is 1.0000 and embedding is 0.0000. - A query that carries no concept surface form produces a zero vector; the embedding is not mistaken, it leaves its domain of definition and its ranking falls to the storage order. - In the document language class both procedures give 0.5000 while rank fusion gives 0.4000; the combined setup is worse than both and this holds at constants of 5, 20 and 60. - The cause is that the first five of the two lists share 1.05 documents: the document of compromise that stands second in both lists passes the document that is first in one and absent from the other. - In the code class fusion drops the share of the target document being in the first five from 1.0000 to 0.0500; giving equal weight to an uninformed list sinks the working procedure as well. - Fusion gains in depth: in document language recall@20 rises from 0.7122 to 0.7214. The gain is at rank twenty, the loss is at the first rank, and what enters the prompt is the first rank. ## Next Step Through six lessons a single question was asked and answered better each time: which chunk of the corpus belongs to this query. The embedding became a search key, the domain fit was measured, the similarity measure was chosen, what the index loses was counted, the point where the filter breaks the neighborhood was found and the fusion of two rankings was weighed. All of these lessons are on the **retrieval** side, and the measure of all of them is the retrieved chunk itself. What is not measured is what comes after. The found chunk is put into a prompt and given to a model; from that point on, the right chunk producing the right answer is an assumption and was never tested in this topic. A wrong answer can be produced even though the right chunk was retrieved, the model can give the right answer by chance even though the wrong chunk was retrieved, and worst of all, the retrieved chunk can lead the model to produce a **worse** answer than its own answer without retrieval. The next topic builds the retrieval pipeline from end to end and separates two columns in its first lesson: was the right chunk retrieved, and was the right answer produced with what was retrieved.