Hybrid Search

The previous lesson built the link between the query and the document from a single place: the angle of the embedding vectors. The filter narrowed this link, it did not change it. Yet what is searched in a support corpus is not always a meaning. When a document code, an error number or the exact name of a procedure is searched, the question asked is not “what is similar to this” but “where exactly does this occur”. Semantic similarity passes by the side of this question.

Hybrid search is running two rankings together for the same query and gathering their results into a single list. The expectation is that the whole is better than its parts: let keyword search catch the exact match that the embedding cannot see, and let the embedding catch the synonym that keyword search cannot see. This lesson measures three setups (keyword only, embedding only, combined) side by side on three query classes. One of the results does not meet the expectation: in one query class the combined setup stays worse than both procedures.

• EV48. The corpus, the embedding procedure, the tokenizer and the query classes are fictional; no real search engine, vector store or embedding endpoint is called. The seed is 20260218.
• EV49. The searchable text of the document is kept together with its code. Because the code is not a concept surface form, it does not change the embedding vector; this is shown in the block.
• EV50. The three query classes are derived from the same documents and differ only in their surface forms: subscriber language carries the query-side form, document language carries the document-side form, and the code query carries the document code. All three are forty queries.
• EV51. The keyword ranking is the number of overlapping tokens. Relevance scoring was built and measured in the Search Engines and Text Retrieval course of the Databases curriculum; it is not repeated here. In this lesson the only function of the keyword side is to produce a second ranking.
• EV52. Fusion is rank-based: the score of a document is the sum of the inverses of its ranks in the two lists, 1 / (constant + rank). The constant is 20 by default and is also tested at 5 and 60.
• EV53. The top-one hit rate is the share of the first result that carries the target concept, retrieval precision is the same share for the first five results, and retrieval recall is the share of the documents carrying the target concept that are caught in the first twenty. The target document column is the share of the document from which the query was derived being in the first five.
• EV54. Every concept is carried by more than one document; for this reason precision and recall are read at different depths and are not reduced to a single number.
• EV55. All three rankings are deterministic: the same query gives the same list in two runs. The run-independent measure of this lesson is the number of documents shared by the first five of the two lists, and it is a property of the pair of procedures, independent of the query.
• EV56. A query that carries no concept surface form produces a zero vector; all similarities become zero and the order falls to the storage order. This is not a defect but the embedding leaving its domain of definition, and it is measured in the code class.
• EV57. This lesson is the close of the topic; what the retrieved chunk does to the model is not measured and is left to the next topic.

Three Query Classes

# FICTIONAL corpus, FICTIONAL embedding simulator, FICTIONAL query classes. No real endpoint.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                 # FICTIONAL tokenizer, pieces of at most five characters
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":             ("tariff", "tier threshold", "which band am i in"),
    "unit_rate":        ("tariff", "unit price", "fee per cubic metre"),
    "subscriber_type":  ("tariff", "subscriber type", "home or shop"),
    "switch":           ("tariff", "tariff switch", "changing plan"),
    "fault_code":       ("fault", "fault code", "error id"),
    "leak":             ("fault", "line leakage", "water escaping"),
    "pressure":         ("fault", "pressure drop", "weak flow"),
    "crew":             ("fault", "field crew", "who will be sent"),
    "index_value":      ("meter", "index value", "what the dial shows"),
    "reading":          ("meter", "reading procedure", "how it gets saved"),
    "replacement":      ("meter", "meter replacement", "new hardware"),
    "verify":           ("meter", "reading verification", "typed in wrong"),
    "planned":          ("outage", "planned outage", "announced stoppage"),
    "notice":           ("outage", "notice channel", "where i hear about it"),
    "duration":         ("outage", "outage duration", "how long it lasts"),
    "restoration":      ("outage", "service restoration", "when it comes back"),
    "periodic":         ("maintenance", "periodic maintenance", "regular checkup"),
    "valve":            ("maintenance", "valve inspection", "tap shutoff routine"),
    "filter_swap":      ("maintenance", "filter swap", "cleaning part"),
    "calibration":      ("maintenance", "calibration record", "gauge accuracy"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "region", "term", "step", "form", "approval", "note"]

DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paragraphs, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts,
                      "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs})


def doc_text(d):
    return d["title"] + " " + " ".join(d["paragraphs"])


def _to_concepts(m):             # the surface form is first mapped to a concept
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        if CONCEPTS[k][1] in dm:
            v[i] += 1.0
        if CONCEPTS[k][2] in dm:
            v[i] += 1.0
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM):           # the concept is then reduced to fewer dimensions
    v = [0.0] * dim
    for i, a in enumerate(_to_concepts(m)):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


K, CONSTANT = 5, 20
SEARCHABLE = [d["code"] + " " + doc_text(d) for d in DOCUMENTS]   # the code enters the searchable text of the document
VEC = [embed(a) for a in SEARCHABLE]
TOKEN_SETS = [set(tokenize(a)) for a in SEARCHABLE]


def queries(mode):               # three separate surface forms from the same documents
    q = []
    for i in range(40):
        r = rng(SEED + 104729 * i)
        d = DOCUMENTS[int(r() * len(DOCUMENTS))]
        o = COMMON[int(r() * len(COMMON))]
        q.append({"text": {"subscriber": f"{CONCEPTS[d['main']][2]} {o}",
                           "document": f"{CONCEPTS[d['main']][1]} {o}",
                           "code": f"{d['code']} numbered document"}[mode],
                  "concept": d["main"], "code": d["code"]})
    return q


QUERY_CLASSES = {k: queries(k) for k in ("subscriber", "document", "code")}


def keyword_rank(m):             # by the number of overlapping tokens
    qb = set(tokenize(m))
    return sorted(range(len(DOCUMENTS)), key=lambda i: -len(qb & TOKEN_SETS[i]))


def embedding_rank(m):
    qv = embed(m)
    return sorted(range(len(DOCUMENTS)), key=lambda i: -cosine(qv, VEC[i]))


def hybrid_rank(m, constant=CONSTANT):   # rank fusion
    pk = {i: 1.0 / (constant + n) for n, i in enumerate(keyword_rank(m))}
    pe = {i: 1.0 / (constant + n) for n, i in enumerate(embedding_rank(m))}
    return sorted(range(len(DOCUMENTS)), key=lambda i: -(pk[i] + pe[i]))


for k, s in QUERY_CLASSES.items():
    print(f"{k:10s} example query: {s[0]['text']}")
print(f"document code does not change the embedding: "
      f"{embed(SEARCHABLE[0]) == embed(doc_text(DOCUMENTS[0]))} , code query vector length "
      f"{sum(x * x for x in embed(QUERY_CLASSES['code'][0]['text'])):.1f}")

subscriber example query: how it gets saved form
document   example query: reading procedure form
code       example query: SY194 numbered document
document code does not change the embedding: True , code query vector length 0.0

The three queries derived from the same document and ask the same thing; the only place they differ is the surface form. The last line measures two things. Adding the document code to the searchable text does not change the embedding vector: the code is not a concept surface form, so the embedding does not see it. The second is harsher: the vector length of the code query is 0.0. Because there is no recognized concept in the query, the vector is zero, all the cosine similarities come out zero and the ranking falls to the storage order. The embedding is not mistaken on this query, it stays outside its domain of definition.

The Table of the Three Setups

def measure(qset, rank):
    first = prec = rec = target = 0.0
    for q in qset:
        relevant = {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]}
        s = rank(q["text"])
        first += s[0] in relevant
        prec += len(set(s[:K]) & relevant) / K
        rec += len(set(s[:20]) & relevant) / len(relevant)
        target += any(DOCUMENTS[i]["code"] == q["code"] for i in s[:K])
    return [x / len(qset) for x in (first, prec, rec, target)]


SETUPS = (("keyword", keyword_rank), ("embedding", embedding_rank), ("hybrid", hybrid_rank))
print("class      setup       top1 precision@5 recall@20 target document@5")
for mode, qset in QUERY_CLASSES.items():
    for name, f in SETUPS:
        d = measure(qset, f)
        print(f"{mode:10s} {name:9s} {d[0]:>6.4f} {d[1]:>11.4f} {d[2]:>9.4f} {d[3]:>17.4f}")
print("hybrid top1 in the document class , constant 5 / 20 / 60: "
      + " / ".join(f"{measure(QUERY_CLASSES['document'], lambda m, c=c: hybrid_rank(m, c))[0]:.4f}"
                   for c in (5, 20, 60)))

class      setup       top1 precision@5 recall@20 target document@5
subscriber keyword   0.0250      0.0800    0.0967            0.0250
subscriber embedding 0.5000      0.4700    0.7122            0.1500
subscriber hybrid    0.4250      0.2900    0.4616            0.1250
document   keyword   0.5000      0.3650    0.6616            0.1750
document   embedding 0.5000      0.4700    0.7122            0.1500
document   hybrid    0.4000      0.4250    0.7214            0.1250
code       keyword   1.0000      0.2500    0.1950            1.0000
code       embedding 0.0000      0.0800    0.1078            0.0250
code       hybrid    0.0000      0.0850    0.1950            0.0500
hybrid top1 in the document class , constant 5 / 20 / 60: 0.4500 / 0.4000 / 0.4000

The first two triples build the justification for hybrid search. In subscriber language keyword search is 0.0250 and embedding is 0.5000; on the code query keyword search is 1.0000 and embedding is 0.0000. The two procedures work in exactly opposite places on the same corpus, and choosing a single procedure means sacrificing one of the query classes entirely.

The hybrid rows show that this sacrifice does not go away. In subscriber language the hybrid is 0.4250, a little below the embedding; this is the expected cost. In document language, while both procedures give 0.5000, the hybrid gives 0.4000. The total is worse than both of its parts. The last line says that this is not the game of one choice of constant: at 5, 20 and 60 it is 0.4500 / 0.4000 / 0.4000, and all three are below 0.5000.

The code class is the most expensive row. Keyword search puts the target document in the first five in forty of the forty queries; the hybrid drops this to 0.0500. In the query class that is the reason hybrid search exists, fusion sinks the working procedure together with the one that does not work. The cause is in the table itself: the embedding side works with a zero vector in this class and the order it produces carries no information, but the fusion does not know this and gives that order equal weight.

One column looks in the opposite direction. In document language recall@20 is 0.6616 for keyword search, 0.7122 for the embedding, and 0.7214 for the hybrid. Up to rank twenty the fusion really does gain. In the code class too, recall@20 rises from 0.1078 to 0.1950. The place where fusion gains is depth, the place where it loses is the first rank, and in a retrieval pipeline what enters the prompt is the first few chunks.

What the Fusion Loses

print("class      shared in first five both correct hybrid wrong")
for mode, qset in QUERY_CLASSES.items():
    shared = lost = 0
    for q in qset:
        relevant = {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]}
        kw, em, hy = (f(q["text"]) for _, f in SETUPS)
        shared += len(set(kw[:K]) & set(em[:K]))
        lost += kw[0] in relevant and em[0] in relevant and hy[0] not in relevant
    print(f"{mode:10s} {shared / len(qset):>20.2f} {lost:>25}")

def relevant_docs(q):
    return {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]}


q = next(x for x in QUERY_CLASSES["document"]
         if keyword_rank(x["text"])[0] in relevant_docs(x)
         and embedding_rank(x["text"])[0] in relevant_docs(x)
         and hybrid_rank(x["text"])[0] not in relevant_docs(x))
print(f"\nquery: {q['text']}  --  target concept: {q['concept']}")
for name, f in SETUPS:
    print(f"  {name:9s}", [f"{DOCUMENTS[i]['code']}/{DOCUMENTS[i]['main']}" for i in f(q["text"])[:3]])

class      shared in first five both correct hybrid wrong
subscriber                 0.12                         0
document                   1.05                         3
code                       4.03                         0

query: pressure drop region  --  target concept: pressure
  keyword   ['AR002/pressure', 'AR015/crew', 'AR018/pressure']
  embedding ['AR079/pressure', 'AR088/pressure', 'AR101/crew']
  hybrid    ['AR033/crew', 'AR015/crew', 'AR018/pressure']

The shared documents column is a number that belongs not to any query but to the pair of procedures itself, and it is independent of the run. In subscriber language the first five of the two lists share 0.12 documents on average: the procedures almost never agree. In document language it is 1.05, that is, one in five.

The example shows the consequence of this number. Keyword search puts AR002 and the embedding puts AR079 at the first rank; both carry the target concept, so both procedures are correct. The hybrid puts AR033 at the first rank, and that document belongs to the crew concept. AR002 is first in one list and far behind in the other; its score comes from a single large term. AR033 is high in both lists but first in neither, and the sum of two medium terms passes one large term. Rank fusion rewards the document of compromise; when the two sides do not agree, the second choice of both wins. In the document language class this happens in three of the forty queries and never in the opposite direction: there is not a single example in which the hybrid finds the right one on a query where both procedures were wrong. The counterpart of the three-query drop in the first rank is zero gain.

The 4.03 in the code row is the trap. The high overlap here is not agreement: because the embedding side works with a zero vector, its list is the storage order, and the tail of the keyword side after the exact match is the same order. The two lists share the same uninformed tail. Fusion cannot tell apart the two lists being the same from both being empty; if it could, it would give keyword search full weight in this class. Hybrid search asks not for an average but for a decision: which side has the say on which query.

Summary

• On three query classes the two procedures work in opposite places: in subscriber language keyword search is 0.0250 and embedding is 0.5000, on the code query keyword search is 1.0000 and embedding is 0.0000.
• A query that carries no concept surface form produces a zero vector; the embedding is not mistaken, it leaves its domain of definition and its ranking falls to the storage order.
• In the document language class both procedures give 0.5000 while rank fusion gives 0.4000; the combined setup is worse than both and this holds at constants of 5, 20 and 60.
• The cause is that the first five of the two lists share 1.05 documents: the document of compromise that stands second in both lists passes the document that is first in one and absent from the other.
• In the code class fusion drops the share of the target document being in the first five from 1.0000 to 0.0500; giving equal weight to an uninformed list sinks the working procedure as well.
• Fusion gains in depth: in document language recall@20 rises from 0.7122 to 0.7214. The gain is at rank twenty, the loss is at the first rank, and what enters the prompt is the first rank.

Next Step

Through six lessons a single question was asked and answered better each time: which chunk of the corpus belongs to this query. The embedding became a search key, the domain fit was measured, the similarity measure was chosen, what the index loses was counted, the point where the filter breaks the neighborhood was found and the fusion of two rankings was weighed. All of these lessons are on the retrieval side, and the measure of all of them is the retrieved chunk itself.

What is not measured is what comes after. The found chunk is put into a prompt and given to a model; from that point on, the right chunk producing the right answer is an assumption and was never tested in this topic. A wrong answer can be produced even though the right chunk was retrieved, the model can give the right answer by chance even though the wrong chunk was retrieved, and worst of all, the retrieved chunk can lead the model to produce a worse answer than its own answer without retrieval. The next topic builds the retrieval pipeline from end to end and separates two columns in its first lesson: was the right chunk retrieved, and was the right answer produced with what was retrieved.