--- title: 'Embeddings and Retrieval-Augmented Generation' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation' language: en updated: '2026-10-21T05:02:28+00:00' license: 'CC BY-SA 4.0' lessons: 16 outcomes: - 'Builds embedding-based similarity search and measures the correctness of the retrieved chunks against a exhaustive-search baseline' - 'Writes the recall and token cost of each document preparation, chunking, indexing and reranking decision' - 'Measures retrieval quality separately from generation quality and shows which cases a single number hides' --- # Embeddings and Retrieval-Augmented Generation Choosing the information given to the model by measurement rather than by hand, and counting retrieval and generation separately: the chunks that keyword matching and embedding similarity bring on the same query set, what an out-of-domain embedding loses at the same dimension, the number of queries where cosine similarity and the dot product separate the top three results and length bias, the recall and comparison count of indexed search against a exhaustive-search baseline, the neighborhood degrading as filter selectivity increases, the query class where combining keywords and embeddings does worse than both, the no-retrieval baseline and the run with wrong chunks falling below the baseline, what document preparation adds to and drops from retrieval recall, chunk size and overlap moving recall and tokens in opposite directions, the staleness share that grows as the refresh interval lengthens, query transformation gaining on short queries and losing on long ones, what reranking adds to top-k precision and the step it adds, the number of chunks that fit within the context limit and the response share that order changes, the four-cell contingency of retrieval and generation of correct chunk and correct response, what retrieval and fine-tuning add on three tasks and the task where retrieval does not win, and what row-level and block-level chunking of a source carrying tables do to recall. ## Embeddings and Vector Search Measuring which chunk of the corpus a query retrieves and what each layer of the search scaffolding gains and loses: the retrieval precision and retrieval recall of keyword matching against embedding similarity on the same query set, the difference in retrieved chunks between an in-domain and an out-of-domain embedding at the same dimension, in how many queries cosine similarity and the dot product separate the first three results and in which one the length bias appears, the recall of an indexed search against the exhaustive search baseline read side by side with its comparison count, the neighborhood breaking down in a space that narrows as filter selectivity rises and the candidate count dropping, and the comparison of keyword-only, embedding-only and combined setups together with the query class where the combined setup stays worse than both. ### What Is an Embedding Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/what-is-an-embedding Deciding which document in the corpus a query resembles by meaning rather than by words, and counting that decision with two separate measures. Forty queries are run on a fictional corpus of two hundred documents, 693 paragraphs and twenty concepts: keyword matching leaves a retrieval precision of 0.0800 and a retrieval recall of 0.0363, while embedding similarity gives 0.4700 and 0.2258; the first result's concept hit rate rises from 0.0250 to 0.5000 and its class hit rate from 0.1000 to 1.0000. The ceiling on recall for five chunks is 0.4904, and reading the class hit rate alone hides the half share in the concept hit rate. The cost is 8000 comparisons, 240 embedding calls and a stored value count of 6400. The Building Applications with Model APIs course treated the call as a network operation across twenty-six lessons: token budget, streaming, format validation, cache, fallback model and the cost of each scaffolding were written down. The question left standing at the close was this: the **knowledge** given to the model was always put into the prompt **by hand**. Which **three** of a thousand documents in a corpus go into that prompt, **against which baseline** that choice is measured, and what a wrong choice costs were never asked. This lesson builds the first tool for that choice. The question is this: what does one look at when deciding which document in the corpus a query resembles. The most direct answer is words. The **embedding** is the second answer: the text is turned into a vector whose neighborhood reflects meaning, and similarity is computed between vectors. The definition of an embedding, cosine similarity and word neighborhood were built in the Natural Language Processing and Computer Vision course; they are not repeated here. In this lesson an embedding is a **search key** and has a single measure: the correctness of the retrieved document. - **EV1.** All the code is a **simulator**. The corpus, concept axis, embedding procedure and query set are **fictional**; none is a real institution, product or document. The seed is **20260218**. - **EV2.** What the simulator models is this: each concept has **two surface forms**. The form that appears in documents (`tier threshold`) and the form that appears in queries (`which band am i in`) **share no token**. Keyword search sees only the surface form; the embedding first turns the surface form into a **concept**. - **EV3.** The embedding procedure has two steps: a vector of twenty concepts is built, then this vector is **projected** to `DIM` dimensions. The projection is a compression; in this lesson the dimension is held fixed at thirty-two. - **EV4.** The embedding is **deterministic**: the same text gives the same vector on every call. This is the lesson's run-independent measure and it is shown in the block. - **EV5.** This lesson's **baseline is keyword search**: the number of a query's distinct tokens that appear in a document is counted, and document order decides ties. - **EV6.** For a query, the **relevant document** is a document whose main concept equals the query's target concept. The number of retrieved chunks is **five**. - **EV7. Retrieval precision** is the share of the five retrieved documents that are relevant; **retrieval recall** is the share of the relevant documents that enter these five. The pair of precision and recall was measured in the Search Engines and Text Retrieval course; the procedure is not repeated. - **EV8.** **Class hit rate is the easy measure, concept hit rate is the hard one.** The two are printed as **two separate columns**. - **EV9.** In this topic the corpus is used **at document level**; chunking is left to the retrieval pipeline. - **EV10.** The cost of retrieval is written as **the number of retrieved chunks, comparison count, embedding calls and stored value count**; raw time is not written. #### Fictional Corpus and Concept Axis The block below builds the corpus, the concept axis, the embedding simulator and the query set. The corpus is two hundred documents; each document carries a title and two to five paragraphs, and is spread across five classes and twenty concepts. The concepts are fictional procedures kept for the field crew of a measurement network: tariff tier, fault code, meter index, outage duration, valve check. The axis rests on a single design decision. The procedure the field crew writes says `tier threshold`; the question coming from the subscriber says `which band am i in`. Both point to the same concept and share not even a single token. In a real corpus this separation is partial; the simulator **carries it to the extreme** so that the difference between the two searches does not get lost inside a noise margin. ```python # SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The corpus, # concept axis, embedding procedure and query set are FICTIONAL; none is a real # institution, product or document. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) # ---- FICTIONAL embedding: the surface form is first mapped to a concept, then the concept is reduced to fewer dimensions def _to_concepts(m, in_domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc_form, query_form = CONCEPTS[k][1], CONCEPTS[k][2] if doc_form in dm: v[i] += 1.0 if query_form in dm: v[i] += 1.0 if in_domain else 0.0 # an out-of-domain embedding does not recognize the query-side form if not in_domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, in_domain=True): k = _to_concepts(m, in_domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query-side form is separate from the document-side form QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) print(f"corpus: {len(DOCUMENTS)} documents , {sum(len(d['paragraphs']) for d in DOCUMENTS)} paragraphs ," f" {len(KEYS)} concepts , {len(CLASSES)} classes") print("class distribution:", " , ".join( f"{s} {sum(1 for d in DOCUMENTS if d['label'] == s)}" for s in CLASSES)) shared = sum(len(set(tokenize(CONCEPTS[k][1])) & set(tokenize(CONCEPTS[k][2]))) for k in KEYS) print(f"shared tokens between document-side and query-side form , 20 concepts : {shared}") d = DOCUMENTS[0] print(f"\n{d['code']} ({d['label']}) title : {d['title']}") print(f" paragraph 1 : {d['paragraphs'][1]}") q = QUERIES[0] print(f"query : {q['text']} -> concept {q['concept']} , class {q['label']}") print(f" {q['concept']} in document-side form '{CONCEPTS[q['concept']][1]}' ," f" in query-side form '{CONCEPTS[q['concept']][2]}'") u, v = embed(q["text"]), embed(q["text"]) print(f"\nembedding {len(u)} dimensions , second call same vector : {u == v}") print("first six coordinates :", " ".join(f"{x:+.3f}" for x in u[:6])) ``` ``` corpus: 200 documents , 693 paragraphs , 20 concepts , 5 classes class distribution: tariff 52 , fault 46 , meter 39 , outage 32 , maintenance 31 shared tokens between document-side and query-side form , 20 concepts : 0 BK001 (maintenance) title : filter swap procedure paragraph 1 : calibration record for entry is kept and written together with filter swap query : how it gets saved form -> concept reading , class meter reading in document-side form 'reading procedure' , in query-side form 'how it gets saved' embedding 32 dimensions , second call same vector : True first six coordinates : -0.018 -0.155 -0.145 -0.101 +0.243 +0.028 ``` The first three lines of the output give the size of the corpus: two hundred documents, six hundred ninety-three paragraphs, twenty concepts, five classes. The classes are spread between fifty-two and thirty-one, so no class alone is a majority. The third line is the lesson's mechanism: across all twenty concepts, the number of shared tokens between the document-side and the query-side form is **zero**. The sample lines make this concrete. `BK001` is a document from the maintenance class, its paragraphs written in the document-side form; the first query is `how it gets saved form` and its target concept is `reading`. The reading concept appears in documents as `reading procedure` and in the query as `how it gets saved`. Between the query and the document there is **no word bridge**; a bridge can be built only through the concept. The remaining two lines close the run question: two calls with the same text give the same vector. In a call that generates text, the same prompt could give two different responses in two runs; on the embedding surface it cannot. #### The Numbers for Two Searches on the Same Query Set Each of the forty queries is run in two setups. The first is **keyword search**: the number of a query's distinct tokens that appear in a document is counted, the five highest-scoring documents are retrieved, and document order decides ties. The second is **embedding search**: the query and the documents are embedded with the same simulator, and the five documents with the highest cosine similarity are retrieved. Each setup produces four numbers; precision and recall are read over the whole retrieved set, concept and class hit rate over the first result only. ```python K = 5 TOKENS = [set(tokenize(doc_text(d))) for d in DOCUMENTS] VEC = [embed(doc_text(d)) for d in DOCUMENTS] RELEVANT = {k: sum(1 for d in DOCUMENTS if d["main"] == k) for k in KEYS} def top_k(score): return sorted(range(len(DOCUMENTS)), key=lambda i: (-score[i], i))[:K] def keyword_search(q): qb = set(tokenize(q["text"])) return top_k([len(qb & TOKENS[i]) for i in range(len(DOCUMENTS))]) def embedding_search(q): qv = embed(q["text"]) return top_k([cosine(qv, VEC[i]) for i in range(len(DOCUMENTS))]) def measure(retrieve): precision = recall = concept = label = 0.0 for q in QUERIES: top = retrieve(q) d = sum(DOCUMENTS[i]["main"] == q["concept"] for i in top) precision += d / K recall += d / RELEVANT[q["concept"]] concept += DOCUMENTS[top[0]]["main"] == q["concept"] label += DOCUMENTS[top[0]]["label"] == q["label"] n = len(QUERIES) return precision / n, recall / n, concept / n, label / n mean_relevant = sum(RELEVANT[q["concept"]] for q in QUERIES) / len(QUERIES) ceiling = sum(min(K, RELEVANT[q["concept"]]) / RELEVANT[q["concept"]] for q in QUERIES) / len(QUERIES) print(f"{len(QUERIES)} queries , k={K} , relevant documents per query {mean_relevant:.3f} ," f" recall ceiling {ceiling:.4f}") print(f"cost: {len(QUERIES) * K} retrieved chunks , {len(QUERIES) * len(DOCUMENTS)} comparisons ," f" {len(DOCUMENTS) + len(QUERIES)} embedding calls , {len(DOCUMENTS) * DIM} stored values") print("\nsetup precision recall concept class") for name, retrieve in (("keyword search", keyword_search), ("embedding", embedding_search)): p, a, kv, sn = measure(retrieve) print(f"{name:<16} {p:9.4f} {a:7.4f} {kv:7.4f} {sn:7.4f}") tied = 0 for q in QUERIES: qb = set(tokenize(q["text"])) score = [len(qb & TOKENS[i]) for i in range(len(DOCUMENTS))] tied += score.count(max(score)) print(f"\nmean number of documents sharing the highest keyword score :" f" {tied / len(QUERIES):.1f}") q = QUERIES[0] print(f"query : {q['text']} (concept {q['concept']})") for name, retrieve in (("keyword search", keyword_search), ("embedding", embedding_search)): print(f" {name:<14}", " ".join( f"{DOCUMENTS[i]['code']}/{DOCUMENTS[i]['main']}" for i in retrieve(q)[:3])) ``` ``` 40 queries , k=5 , relevant documents per query 10.825 , recall ceiling 0.4904 cost: 200 retrieved chunks , 8000 comparisons , 240 embedding calls , 6400 stored values setup precision recall concept class keyword search 0.0800 0.0363 0.0250 0.1000 embedding 0.4700 0.2258 0.5000 1.0000 mean number of documents sharing the highest keyword score : 79.2 query : how it gets saved form (concept reading) keyword search BK001/filter_swap AR002/pressure SY003/reading embedding SY025/replacement SY031/reading SY035/replacement ``` #### Reading the Table The keyword search row looks like a search result but is not. The precision is 0.0800; given that an average of 10.825 relevant documents per query stand among two hundred documents, the precision of picking five documents at random would be 0.0541. The last lines say where the difference comes from: an average of **79.2 documents share** the highest keyword score. What determines the first five is not relevance but the document order that breaks the tie. Recall stays at 0.0363. The embedding row gives 0.4700 precision and 0.2258 recall on the same queries; precision rises from 0.0800 to 0.4700. Recall has a ceiling of its own: with an average of 10.825 relevant documents per query, if five documents are retrieved, recall can be at most **0.4904**. The embedding gets close to half of this ceiling, and the remaining gap is the gap not of the scaffolding but of the **number of retrieved chunks**. The first three results of the sample query make the difference concrete. Keyword search retrieves `BK001`, `AR002` and `SY003`: three documents from three different classes, and the one document carrying the target concept has come to third place only because the tie was broken. Embedding search retrieves all three from the meter class and the second carries the target concept. #### Why the Two Columns Stay Separate The last two columns of the table are the first application of this course's rule. The embedding search's **class hit rate is 1.0000**: in forty of the forty queries the first result is from the right class. The same row's **concept hit rate is 0.5000**: in twenty queries the first result is a document of the wrong concept, from within the right class. What would have happened if it were reduced to a single measure. If the class hit rate were reported, the system would look flawless and there would be nothing left to improve. If the concept hit rate were reported, it would look as good as a coin toss, and the perfect hit at the class level would not be visible. Side by side the problem gets its name: **the scaffolding finds the class and cannot find the concept.** Since the wrongly retrieved document is from the same class, it is not irrelevant, only wrong; placed inside a response it does not look off-topic. On the keyword search row the same distinction reads the other way round: class hit rate 0.1000, concept hit rate 0.0250. The class column does not soften the failure here: with five classes, a hit rate of one in ten is below even the share of naming a single class every time. The cost row writes the price of the scaffolding: 8000 comparisons, 240 embedding calls, 6400 stored values, 200 retrieved chunks. Keyword search spends no embedding calls; the whole difference is what these 240 calls and 6400 values buy. As the corpus grows, the comparison count grows in direct proportion, because every query is compared with the whole corpus. #### Summary - A concept's document-side surface form and query-side surface form can differ; in this corpus the number of shared tokens is zero for all twenty concepts. - On the same forty queries, keyword search gives 0.0800 retrieval precision and 0.0363 retrieval recall, and embedding search gives 0.4700 and 0.2258. - In keyword search an average of 79.2 documents share the highest score; the order that breaks the tie, not relevance, determines the first five. - The recall ceiling for five chunks and an average of 10.825 relevant documents is 0.4904; the measured recall is read against this ceiling. - Class hit rate is 1.0000 while concept hit rate is 0.5000; two columns name the problem, a single number hides it. - The cost of retrieval is 8000 comparisons, 240 embedding calls, 6400 stored values and 200 retrieved chunks. #### Next Step This lesson used a single embedding, and that embedding recognizes the language of the corpus and the language of the query together. This is an assumption: if a ready-made embedding model was built with the texts of another domain, it may not recognize the query-side form. The next lesson builds two embeddings of the same dimension, one fitted to the language of this domain and the other not, and counts side by side the chunks the two retrieve on the same forty queries. The cost of the dimension decision was measured in the Building Applications with Model APIs course; what will be added on top of it is the share of domain fit. ### Embedding Models Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/embedding-models Placing two of an embedding model's three settings side by side on the same workload: dimension and domain fit. At the same thirty-two dimensions, the same 6400 stored values and the same forty queries, the in-domain embedding gives a retrieval precision of 0.4700 and a concept hit rate of 0.5000, while the out-of-domain embedding falls to 0.0650 and 0.0500; on concept and class hit rate it rises only a step above the keyword search baseline, while on the retrieved set it drops below the baseline. With a second embedding model of the same family the difference stays at 0.0250 in concept hit rate while the loss from falling out of domain is 0.4500; the first is unmeasured, the second is measured. In the dimension sweep the two models do not agree between 64 and 8 on which dimension is good, and the cliff appears only between 8 and 4. The previous lesson used a single embedding, and that embedding recognized both the language of the corpus and the language of the query. Against the keyword search baseline, retrieval precision rose from 0.0800 to 0.4700 and concept hit rate from 0.0250 to 0.5000. That was not the number of an embedding model; it was the number of an embedding model that **recognizes the query language**. With a ready-made model this assumption does not come true by itself. When an embedding model is chosen, three settings are discussed: **dimension**, **domain fit** and **language coverage**. The cost of the dimension decision was measured in the Building Applications with Model APIs course: the share of the three nearest neighbors from the same topic was 0.9259 at one hundred forty-four dimensions and 0.9028 at seventy-two dimensions, and the stored value count fell from 10368 to half. That procedure is not repeated; what this lesson adds is the share of domain fit **at the same dimension**. Language coverage looks like a third setting, but in this measure it is the same surface: the question in both cases is whether the model **recognizes the query language**. - **EV11.** All the code is a **simulator**; the corpus, concept axis and embedding procedure are **fictional**. The seed is **20260218**. No real embedding endpoint is called. - **EV12.** **Domain fit is modeled explicitly in the simulator.** The in-domain embedding recognizes both surface forms of a concept: `pressure drop` in the document-side form and `weak flow` in the query-side form go to the same concept. The out-of-domain embedding recognizes only the document-side form; it does not see the query-side form, and the only signal left to it is the **common words** shared with the documents. This is the structural counterpart of an embedding built with the texts of another domain. - **EV13.** The dimension is **held fixed**: thirty-two. The two setups store the same number of coordinates and perform the same number of multiplications; the difference between them comes **only** from domain fit. - **EV14.** A **second embedding model of the same family** is built; only the projection seed is different (20260219). For a loss to count as measured, it must **exceed the difference between the two models**. This is the lesson's run-independent baseline. - **EV15.** The baseline is again **keyword search**; the measure definitions are the same as in the previous lesson: five retrieved chunks, and a relevant document is a document whose main concept is the same as the query's target concept. - **EV16.** The dimension sweep is read **with two models together**; if the two models do not give the same ranking, there is **no measured order** in that range. - **EV17.** The storage cost is **the number of documents times the dimension**, that is, the stored value count. #### What Domain Fit Models The first block builds the corpus, the concept axis and the embedding simulator; at the end it shows domain fit on a single query. When the in-domain embedding sees the query-side form, it writes the concept's weight in full; the out-of-domain embedding **cannot see** that concept, and what remains is a thin weight distributed by the common words. ```python # SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The corpus, # concept axis, embedding procedure and query set are FICTIONAL; none is a real # institution, product or document. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) # ---- FICTIONAL embedding: the surface form is first mapped to a concept, then the concept is reduced to fewer dimensions def _to_concepts(m, in_domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc_form, query_form = CONCEPTS[k][1], CONCEPTS[k][2] if doc_form in dm: v[i] += 1.0 if query_form in dm: v[i] += 1.0 if in_domain else 0.0 # an out-of-domain embedding does not recognize the query-side form if not in_domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, in_domain=True): k = _to_concepts(m, in_domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query-side form is separate from the document-side form QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) print(f"corpus: {len(DOCUMENTS)} documents , {len(KEYS)} concepts , {len(CLASSES)} classes ," f" {len(QUERIES)} queries , embedding dimension {DIM}") q = QUERIES[3] k = q["concept"] print(f"query : {q['text']} -> concept {k} , class {q['label']}") print(f" {k} in document-side form '{CONCEPTS[k][1]}' , in query-side form '{CONCEPTS[k][2]}'") print(f" in-domain embedding weight of {k} in the concept vector :" f" {_to_concepts(q['text'])[KEYS.index(k)]:.2f}") print(f" out-of-domain embedding weight of {k} in the concept vector :" f" {_to_concepts(q['text'], in_domain=False)[KEYS.index(k)]:.2f}") ``` ``` corpus: 200 documents , 20 concepts , 5 classes , 40 queries , embedding dimension 32 query : weak flow region -> concept pressure , class fault pressure in document-side form 'pressure drop' , in query-side form 'weak flow' in-domain embedding weight of pressure in the concept vector : 1.00 out-of-domain embedding weight of pressure in the concept vector : 0.05 ``` The last two lines carry the whole measure. For the `pressure` concept the weight is 1.00 in the in-domain embedding and 0.05 in the out-of-domain embedding. The second is not a concept signal but the base noise that the common words distribute equally to all concepts: the out-of-domain embedding puts the query at a point **equidistant from twenty concepts**. #### Two Embeddings at the Same Dimension The block below runs five setups on the same forty queries: the keyword search baseline, the two models of the in-domain embedding and the two models of the out-of-domain embedding. All of them run at thirty-two dimensions and store the same number of coordinates. The last column counts how many **distinct** top-five sets the forty queries produce; whether a ranker changes from query to query is read from there. The last two lines write separately the difference between two models that stay in the same domain and the loss of falling out of domain. ```python K = 5 RELEVANT = {k: sum(1 for d in DOCUMENTS if d["main"] == k) for k in KEYS} TOKENS = [set(tokenize(doc_text(d))) for d in DOCUMENTS] # SECOND embedding model of the same family: only the projection seed differs (20260219). PROJECTION2 = [] for i in range(len(KEYS)): r = rng((SEED + 1) ^ (i + 1) * 2654435761) PROJECTION2.append([r() * 2 - 1 for _ in range(64)]) def embed2(m, dim=DIM, in_domain=True): k = _to_concepts(m, in_domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION2[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def measure(rank): """Four numbers and the number of DISTINCT top-five sets the forty queries produce.""" precision = recall = concept = label = 0.0 top_sets = set() for q in QUERIES: top = rank(q)[:K] top_sets.add(tuple(sorted(top))) t = sum(DOCUMENTS[i]["main"] == q["concept"] for i in top) precision += t / K recall += t / RELEVANT[q["concept"]] concept += DOCUMENTS[top[0]]["main"] == q["concept"] label += DOCUMENTS[top[0]]["label"] == q["label"] n = len(QUERIES) return precision / n, recall / n, concept / n, label / n, len(top_sets) def keyword_rank(q): qb = set(tokenize(q["text"])) return sorted(range(len(DOCUMENTS)), key=lambda i: (-len(qb & TOKENS[i]), i)) def embedding_rank(embedder, in_domain, dim=DIM): vecs = [embedder(doc_text(d), dim, in_domain) for d in DOCUMENTS] def rank(q): qv = embedder(q["text"], dim, in_domain) return sorted(range(len(DOCUMENTS)), key=lambda i: (-cosine(qv, vecs[i]), i)) return rank print("setup precision recall concept class sets") results = {} for name, rank in (("keyword search", keyword_rank), ("in-domain , model 1", embedding_rank(embed, True)), ("in-domain , model 2", embedding_rank(embed2, True)), ("out-of-domain , model 1", embedding_rank(embed, False)), ("out-of-domain , model 2", embedding_rank(embed2, False))): results[name] = measure(rank) print(f"{name:<23} {results[name][0]:9.4f} " + " ".join(f"{x:7.4f}" for x in results[name][1:4]) + f" {results[name][4]:5d}") in1, in2 = results["in-domain , model 1"], results["in-domain , model 2"] out1 = results["out-of-domain , model 1"] print("\ndifference between the two models (in-domain) :", " ".join(f"{abs(a - b):7.4f}" for a, b in zip(in1[:4], in2[:4]))) print("loss from falling out of domain (model 1) :", " ".join(f"{a - b:7.4f}" for a, b in zip(in1[:4], out1[:4]))) print(f"stored values , same in every setup : {len(DOCUMENTS) * DIM}") ``` ``` setup precision recall concept class sets keyword search 0.0800 0.0363 0.0250 0.1000 7 in-domain , model 1 0.4700 0.2258 0.5000 1.0000 17 in-domain , model 2 0.4050 0.2035 0.5250 1.0000 17 out-of-domain , model 1 0.0650 0.0316 0.0500 0.2500 1 out-of-domain , model 2 0.0450 0.0247 0.0250 0.2500 1 difference between the two models (in-domain) : 0.0650 0.0223 0.0250 0.0000 loss from falling out of domain (model 1) : 0.4050 0.1942 0.4500 0.7500 stored values , same in every setup : 6400 ``` The most striking part of the table is the concept and class columns of the out-of-domain embedding: **0.0500** and **0.2500**. The keyword search baseline gives **0.0250** and **0.1000**; the out-of-domain embedding rises only a step above it, and stays a full **0.4500** below the in-domain embedding on concept hit rate. It produces vectors, computes cosine similarity, ranks, and gets from the first result barely more than a word match does. Dimension, storage and comparison are the same; almost all of what the scaffolding was built for has been wiped out. Looking at the retrieved set, the situation is worse. The out-of-domain embedding's retrieval precision is 0.0650 and its recall 0.0316; the keyword search baseline has 0.0800 and 0.0363. That is, the five-document set is **worse** than when no embedding was used at all. The added scaffolding did not raise the number, it lowered it. The reason is written in the last column: the out-of-domain setup produces **a single** top-five set on forty queries. Since all queries fall on the same point, the same five documents go to all of them. The keyword search baseline produces seven distinct sets, the in-domain embedding seventeen. The last two lines decide whether the loss counts as measured. The difference between the two models is 0.0650 in precision, 0.0223 in recall, 0.0250 in concept hit rate and 0.0000 in class hit rate. The loss from falling out of domain is 0.4050, 0.1942, 0.4500 and 0.7500; in concept hit rate, **eighteen times** the model difference. In the Building Applications with Model APIs course, the loss from halving the dimension came out the same size as the model difference and was counted as **unmeasured**. The loss here is far above that threshold: **domain fit is not a setting to be put on the same scale as dimension.** A language list of an embedding model does not take the place of this measure. A language being on the list means the model has seen the **document** texts in that language; it does not mean it has seen the words with which the user asks about the same concept. #### Where Dimension Dies What happens if domain fit is fixed and the dimension is swept. The block below runs the in-domain setup at five dimensions with two models. ```python print(" dim stored values model 1 concept / class model 2 concept / class") for d in (64, 32, 16, 8, 4): a = measure(embedding_rank(embed, True, d)) b = measure(embedding_rank(embed2, True, d)) print(f"{d:>5} {len(DOCUMENTS) * d:>13} {a[2]:>16.4f} {a[3]:>7.4f}" f" {b[2]:>15.4f} {b[3]:>7.4f}") ``` ``` dim stored values model 1 concept / class model 2 concept / class 64 12800 0.4250 1.0000 0.4500 1.0000 32 6400 0.5000 1.0000 0.5250 1.0000 16 3200 0.4500 1.0000 0.4000 0.9500 8 1600 0.6500 1.0000 0.4750 0.8250 4 800 0.3250 0.6000 0.3500 0.7000 ``` The concept columns are **not monotonic** between 64 and 8. The first model gives 0.4250, 0.5000, 0.4500, 0.6500 and gets its highest value at the smallest dimension; the second model gives 0.4500, 0.5250, 0.4000, 0.4750 and gets its highest value at thirty-two. The two models **do not agree on which dimension is better**, and the difference at eight dimensions is 0.1750, seven times the model difference at thirty-two. In this range the dimension ranking is **unmeasured**; going down to eight by looking at the sweep of a single model would be relying on a draw. The class column is more stable, and it carries the real information. In the first model it does not go below 1.0000 down to eight dimensions; in the second it goes down to 0.9500 at sixteen and to 0.8250 at eight. At four dimensions both collapse together: 0.6000 and 0.7000. **The cliff lies between eight and four dimensions**, and its cause can be counted: when twenty concepts are squeezed into four coordinates, two concepts falling in the same direction is unavoidable. The storage side completes the reading. Going from sixty-four dimensions to eight lowers the stored value count from 12800 to 1600 and leaves no measured loss in class hit rate in the first model; going down to four only halves the storage again while cutting the class hit rate to 0.6000. The shape of the dimension decision is in the same direction as what was seen in the Building Applications with Model APIs course: free over a long range, then suddenly expensive. #### Summary - Domain fit is whether an embedding model ties the query-side surface form to the same concept as the document-side surface form; language coverage is another name for the same question. - At the same thirty-two dimensions and the same 6400 stored values, the in-domain embedding gives 0.4700 precision and 0.5000 concept hit rate, and the out-of-domain embedding gives 0.0650 and 0.0500. - The out-of-domain embedding's concept and class hit rate (0.0500 / 0.2500) are only a step above the keyword search baseline's (0.0250 / 0.1000); on the retrieved set it falls below the baseline, because it gives a single top-five set to forty queries. - The difference between the two models is 0.0250 in concept hit rate and the loss from falling out of domain is 0.4500; the first is an unmeasured difference, the second a measured one. - In the dimension sweep the two models do not agree between 64 and 8; in that range there is no measured order. - The cliff lies between 8 and 4 dimensions and the threshold comes from the number of concepts: twenty concepts do not fit in four. #### Next Step In these two lessons the similarity was computed with a single procedure: corresponding coordinates were multiplied and summed, and before that the vectors were brought to unit length. The second step is a silent decision. If normalization is not done, the same product produces a different ranking and long documents start to get higher scores. The next lesson puts cosine similarity and the dot product side by side on the same forty queries: in how many queries the first three results diverge, which measure is biased toward length, and in which direction this bias moves the correctness of the retrieved chunk. ### Similarity Metrics Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/similarity-metrics Ranking the same vectors with two separate measures and how much the choice changes the retrieved chunk. The dot product is the cosine similarity multiplied by the norms of the two vectors; for one query the query norm is constant, so the entire ranking difference comes from the norm of the document vector, and this norm is 4.6737 in the shortest quarter and 5.4708 in the longest. In thirty of the forty queries the first three results diverge; cosine similarity gives a retrieval precision of 0.4700 and a concept hit rate of 0.5000 while the dot product stays at 0.4100 and 0.4250, and the same direction appears in a second embedding model. The length bias is in the dot product: the mean of the retrieved set is 66.2 tokens, 57.7 under cosine similarity and 64.1 in the corpus; since the mean of the relevant documents is also 64.7, length carries no relevance. The previous two lessons always computed similarity with the same procedure: the vectors were brought to unit length, then the corresponding coordinates were multiplied and summed. The first step was a silent decision and was never discussed. If the vectors are not brought to unit length, the same product gives a different number and produces a different ranking. The two measures are the names of these two cases. **Cosine similarity** measures the angle between two vectors and ignores their lengths. The **dot product** sums the coordinate products directly and carries the length inside it. The relation between them is a single line: $$\text{dot product}(u, v) = \text{cosine similarity}(u, v) \cdot \lVert u \rVert \cdot \lVert v \rVert$$ When a query is fixed, $\lVert u \rVert$ is the same for all documents. The conclusion that follows is the axis of this lesson: the **entire ranking difference** between the two measures comes from the value $\lVert v \rVert$, that is, from the **norm of the document vector**. - **EV18.** The code is a **simulator**, the corpus and the embedding procedure are **fictional**, the seed is **20260218**. The dimension is thirty-two, the number of retrieved chunks is five, the definition of a relevant document is the same as in the previous two lessons, and the embedding is in-domain. - **EV19.** **One vector set, two measures.** The shared embedding procedure brings the vector to unit length, the raw procedure does not; cosine similarity is the dot product of unit vectors. - **EV20.** The relation above is **checked with numbers** in the block. - **EV21.** Document length is measured **in tokens**; in the corpus it varies between thirty-two and one hundred seven. - **EV22.** **Divergence** is counted over the **set** of the first **three** results: if the two measures retrieve the same three documents, they count as not diverged even if their order is different. - **EV23.** **The length bias is read against two baselines**: the mean tokens of the retrieved set against the corpus mean, and the share of the longest quarter in the first five against that quarter's share in the corpus. - **EV24.** The run-independent measure is **a second embedding model of the same family** (projection seed 20260219). If the ranking does not come out in the same direction in the two models, it **does not count as measured**. #### One Vector Set, Two Measures The first block builds the corpus, adds the procedure that produces the unnormalized vector, and shows two things: how the norm of the document vector grows with length, and whether the relation holds. ```python # SIMULATOR -- FICTIONAL. No real embedding endpoint is called. The corpus, # concept axis, embedding procedure and query set are FICTIONAL; none is a real # institution, product or document. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) # ---- FICTIONAL embedding: the surface form is first mapped to a concept, then the concept is reduced to fewer dimensions def _to_concepts(m, in_domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc_form, query_form = CONCEPTS[k][1], CONCEPTS[k][2] if doc_form in dm: v[i] += 1.0 if query_form in dm: v[i] += 1.0 if in_domain else 0.0 # an out-of-domain embedding does not recognize the query-side form if not in_domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, in_domain=True): k = _to_concepts(m, in_domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query-side form is separate from the document-side form QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # Two measures, one vector set: the shared embed() vector is brought to unit length, # raw() is not. Cosine similarity is the dot product of unit vectors. def raw(m, dim=DIM, in_domain=True): k = _to_concepts(m, in_domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] return v def length(v): return sum(x * x for x in v) ** 0.5 TOKEN_COUNTS = [len(tokenize(doc_text(d))) for d in DOCUMENTS] NORMS = [length(raw(doc_text(d))) for d in DOCUMENTS] ordered = sorted(range(len(DOCUMENTS)), key=lambda i: TOKEN_COUNTS[i]) quarter = len(DOCUMENTS) // 4 shortest, longest = ordered[:quarter], ordered[-quarter:] print(f"corpus: mean tokens {sum(TOKEN_COUNTS) / len(DOCUMENTS):.1f} ," f" shortest {TOKEN_COUNTS[ordered[0]]} , longest {TOKEN_COUNTS[ordered[-1]]}") print(f" shortest quarter : mean {sum(TOKEN_COUNTS[i] for i in shortest) / quarter:.1f} tokens ," f" raw vector length {sum(NORMS[i] for i in shortest) / quarter:.4f}") print(f" longest quarter : mean {sum(TOKEN_COUNTS[i] for i in longest) / quarter:.1f} tokens ," f" raw vector length {sum(NORMS[i] for i in longest) / quarter:.4f}") q, d = QUERIES[0], DOCUMENTS[0] raw_q, raw_d = raw(q["text"]), raw(doc_text(d)) print(f"\nidentity check , query 1 with {d['code']}:") print(f" dot product {cosine(raw_q, raw_d):.6f}") print(f" cosine similarity {cosine(embed(q['text']), embed(doc_text(d))):.6f}") print(f" cosine * both vector lengths " f"{cosine(embed(q['text']), embed(doc_text(d))) * length(raw_q) * length(raw_d):.6f}") ``` ``` corpus: mean tokens 64.1 , shortest 32 , longest 107 shortest quarter : mean 39.7 tokens , raw vector length 4.6737 longest quarter : mean 89.4 tokens , raw vector length 5.4708 identity check , query 1 with BK001: dot product -0.218248 cosine similarity -0.011168 cosine * both vector lengths -0.218248 ``` The quarter lines give the mechanism. In the shortest quarter a document has a mean of 39.7 tokens and a raw vector norm of 4.6737; in the longest quarter 89.4 tokens and 5.4708. While the document grows to two and a quarter times, the norm grows only seventeen percent, but because the dot product is **multiplied** by that norm, this seventeen percent enters the score directly. The reason is written in the procedure: the more concepts a document mentions, the more weight falls on the concept vector. The identity check shows that the relation holds: when the dot product is computed directly and when the cosine similarity is multiplied by the two norms, the same number comes out to six digits. The two measures are not separate procedures; **one is the other multiplied by the norms**. #### How Much the Ranking Changes When the Measure Changes The block below runs the same forty queries with two measures and repeats this in two embedding models. For each model it prints first the number of queries whose first three results diverge, then the four retrieval measures and the mean tokens of the retrieved set. ```python K = 5 RELEVANT = {k: sum(1 for d in DOCUMENTS if d["main"] == k) for k in KEYS} # SECOND embedding model of the same family: only the projection seed differs (20260219). PROJECTION2 = [] for i in range(len(KEYS)): r = rng((SEED + 1) ^ (i + 1) * 2654435761) PROJECTION2.append([r() * 2 - 1 for _ in range(64)]) def raw2(m, dim=DIM): k = _to_concepts(m) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION2[i][j % 64] return v def unit(v): n = length(v) return [x / n for x in v] if n else v def rank(qv, vecs): return sorted(range(len(DOCUMENTS)), key=lambda i: (-cosine(qv, vecs[i]), i)) def measure(embedder): raw_vecs = [embedder(doc_text(d)) for d in DOCUMENTS] unit_vecs = [unit(v) for v in raw_vecs] counts = {"cosine": [0.0] * 5, "dot product": [0.0] * 5} diverged = 0 for q in QUERIES: raw_q = embedder(q["text"]) a, b = rank(unit(raw_q), unit_vecs), rank(raw_q, raw_vecs) diverged += set(a[:3]) != set(b[:3]) for name, top in (("cosine", a[:K]), ("dot product", b[:K])): t = sum(DOCUMENTS[i]["main"] == q["concept"] for i in top) row = counts[name] row[0] += t / K row[1] += t / RELEVANT[q["concept"]] row[2] += DOCUMENTS[top[0]]["main"] == q["concept"] row[3] += DOCUMENTS[top[0]]["label"] == q["label"] row[4] += sum(TOKEN_COUNTS[i] for i in top) / K n = len(QUERIES) return diverged, {name: [x / n for x in row] for name, row in counts.items()} for model_name, embedder in (("model 1", raw), ("model 2", raw2)): diverged, results = measure(embedder) print(f"{model_name} : queries whose first three results diverge {diverged}/{len(QUERIES)}") print(" measure precision recall concept class mean tokens") for name, row in results.items(): print(f" {name:<11} {row[0]:9.4f} " + " ".join(f"{x:7.4f}" for x in row[1:4]) + f" {row[4]:11.1f}") ``` ``` model 1 : queries whose first three results diverge 30/40 measure precision recall concept class mean tokens cosine 0.4700 0.2258 0.5000 1.0000 57.7 dot product 0.4100 0.1951 0.4250 1.0000 66.2 model 2 : queries whose first three results diverge 25/40 measure precision recall concept class mean tokens cosine 0.4050 0.2035 0.5250 1.0000 54.6 dot product 0.3600 0.1751 0.3500 1.0000 58.3 ``` The first line says the choice of measure is not a detail: in **thirty** of the forty queries the two measures retrieve a different top three. The same vectors, the same corpus, the same queries; the only thing that changes is whether the score is read from normalized vectors or from raw vectors. Cosine similarity is ahead in three of the four measures and tied in the fourth: precision 0.4700 against 0.4100, recall 0.2258 against 0.1951, concept hit rate 0.5000 against 0.4250; class hit rate is 1.0000 for both. In the second embedding model the numbers change but **the direction does not**: 0.4050 against 0.3600 and 0.5250 against 0.3500. The ranking of the measures is not the draw of a single projection. The last column shows where the difference comes from. The mean of the five documents retrieved by cosine similarity is 57.7 tokens, that of the dot product 66.2. The 8.5-token shift between them is not a byproduct; it is a direct consequence of the definition of the measure. #### Looking at Length Where Length Carries No Relevance For a shift to count as bias, it must be shown that length **carries no relevance**. The block below prints two baselines: the mean length of the corpus against the mean length of the relevant documents, and the share of the longest quarter in the first five of the two measures. ```python quarter = len(DOCUMENTS) // 4 longest = set(sorted(range(len(DOCUMENTS)), key=lambda i: -TOKEN_COUNTS[i])[:quarter]) relevant_docs = {k: [i for i, d in enumerate(DOCUMENTS) if d["main"] == k] for k in KEYS} print(f"corpus mean {sum(TOKEN_COUNTS) / len(DOCUMENTS):.1f} tokens ," f" longest quarter share {quarter / len(DOCUMENTS):.4f}") print("mean of relevant documents %.1f tokens" % (sum(sum(TOKEN_COUNTS[i] for i in relevant_docs[q["concept"]]) / len(relevant_docs[q["concept"]]) for q in QUERIES) / len(QUERIES))) raw_vecs = [raw(doc_text(d)) for d in DOCUMENTS] unit_vecs = [unit(v) for v in raw_vecs] print("\nmeasure longest-quarter share in first five") for name, vecs, query_vec in (("cosine", unit_vecs, lambda q: unit(raw(q["text"]))), ("dot product", raw_vecs, lambda q: raw(q["text"]))): p = sum(sum(1 for i in rank(query_vec(q), vecs)[:K] if i in longest) / K for q in QUERIES) / len(QUERIES) print(f"{name:<11} {p:35.4f}") ``` ``` corpus mean 64.1 tokens , longest quarter share 0.2500 mean of relevant documents 64.7 tokens measure longest-quarter share in first five cosine 0.1500 dot product 0.2950 ``` The first two lines give the verdict. The corpus mean is 64.1 tokens; the mean of the relevant documents is also 64.7. There is no link between length and relevance in this corpus; that is, a long document has no greater chance of answering the query than a short one. The lower table writes how this shows up in the ranking. The longest quarter makes up 0.2500 of the corpus. In the dot product's first five the share of this quarter is **0.2950**, above the baseline rate; in the cosine similarity's first five it is **0.1500**, below the baseline rate. The two measures stand in two different places against length: the dot product keeps long documents the dot product keeps long documents above their share in the corpus, cosine similarity holds them below it. Why the suppression works can be read from the definition of the norm. A long document mentions more concepts; its raw vector is longer but **more scattered**. Since cosine similarity throws the norm away by dividing, the remaining question is not "how many concepts are mentioned" but "which concept is given what share"; since the query targets a single concept, the winner is the concentrated document. The conclusion that follows is independent of the names of the measures. **If all vectors are of unit length, the two measures give the same ranking**; when the norms equal one, the product changes nothing. The real decision is not "which measure" but **whether the vectors are normalized when they are stored**. If normalization is done, the dot product saves a division and the result is the same; if it is not, the choice of measure changes the retrieved chunk in thirty of the forty queries. #### Summary - The dot product is the cosine similarity multiplied by the norms of the two vectors; for a fixed query the entire ranking difference comes from the norm of the document vector. - The norm of the document vector grows with length: 4.6737 in the shortest quarter, 5.4708 in the longest. - In thirty of the forty queries the two measures retrieve a different top three. - Cosine similarity is ahead in three measures and tied in class hit rate (0.4700 / 0.2258 / 0.5000 / 1.0000 against 0.4100 / 0.1951 / 0.4250 / 1.0000) and the direction stays the same in the second embedding model. - The length bias is in the dot product: the mean of the retrieved set is 66.2 tokens, 57.7 under cosine similarity, 64.1 in the corpus; the share of the longest quarter in the first five is 0.2950 against 0.1500. - The mean of the relevant documents is also 64.7 tokens, so length carries no relevance; if the vectors are stored at unit length, the two measures give the same ranking. #### Next Step In three lessons three decisions of retrieval were measured: words or meaning, which embedding model, which measure. There was one thing common to all three that was never asked: every query was compared with **the whole corpus**, 8000 comparisons were made for forty queries and two hundred documents, and the full ranking was built from scratch each time. With two hundred documents this is not a problem; when the corpus grows to hundreds of thousands of chunks, touching all the vectors for every query is not sustainable. The next lesson puts the vectors into an **index** and asks, taking the exhaustive search as the baseline: how much does the set retrieved without touching all the documents deviate from the set retrieved by touching them, and what recall is given in return for the comparison count. ### Vector Databases Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/vector-databases Removing the assumption that a query is compared with the whole corpus and measuring the price of approximation. On two hundred fictional documents, exhaustive search needs 200 comparisons per query; the bucket-based index holds recall at 0.9500 with 16 buckets and 2 probes and brings the comparisons down to 47.2, and with 4 probes recall is 1.0000 and comparisons are 69.2. With 1 probe recall falls to 0.7200 and the concept precision of the first five falls from 0.4700 to 0.3800: the index damages not only the candidate set but also the answer. The neighborhood graph at degree 8 can reach only 27 documents from the entry node and its recall is 0.0800; at degree 48 it stores 9600 edges and, with 84.9 comparisons, stays at 0.7650, that is, both more expensive and worse than the bucket-based index. The difference between two index builds is 0.0100, and this is the threshold of this workload. The previous three lessons built the embedding as a search key, measured how much of the domain fit it carries, and counted where two similarity measures diverge on the same query. All three shared a silent assumption: when a query arrived, it was compared with **every document** in the corpus. With two hundred documents this assumption is cheap. When the corpus grows it stops being cheap, because the comparison count grows in direct proportion to the document count and every query has to read the entire corpus. The vector store reduces this load with an **index**: when a query arrives, only a part of the corpus is looked at. Looking at a part means the right neighbor may remain in the part that was not looked at; for this reason indexed search is called **approximate nearest neighbor search** and it does not have to give the answer that exhaustive search gives. What this lesson measures is that difference: **what the approximation costs**. Against the baseline of exhaustive search, the **recall** and the **comparison count** of two index structures are printed side by side, and it is shown that one of them stays both below the baseline and expensive even though it multiplies the storage. - **EV31.** The corpus, the embedding procedure and the query set are **fictional**; no real embedding endpoint or vector store is called. The seed is **20260218**. - **EV32.** The simulator surface first reduces the surface form to a concept and then the concept to thirty-two dimensions; the document side and the query side share no token. All measurements in this lesson are made with the in-domain embedding. - **EV33.** The baseline is **exhaustive search**: every query is compared with all two hundred documents and the first five results are taken. - **EV34. Retrieval recall** is read against the index in this lesson: the size of the intersection of the index's first five and the first five of exhaustive search, divided by five. A recall of 1.0000 means the index gave the **same** answer as the baseline, not that it gave the correct answer. - **EV35. Concept precision** is the share of the five retrieved documents that carry the target concept of the query, and it shows the quality of the baseline itself. - **EV36. The comparison count** is the number of cosine similarities computed in a single query. No time is written; the build cost of the index is counted separately as **stored edges** or as the **representative count**. - **EV37.** Index structures are named by their **behavior**: bucket-based index and neighborhood graph. No vector store, library or method name is written. - **EV38.** The build of the bucket-based index **depends on the seed**. The two runs of this lesson measure the same query set on two separate index builds; the difference between them is the **threshold** of this workload, and differences below it count as unmeasured. The nearest neighbor procedure itself was built and measured in the Unsupervised Learning course of the Machine Learning and Data Science curriculum; it is not repeated here. The only thing added is the cost of **not running the whole** of that procedure. #### The Baseline of Exhaustive Search The block below builds the fictional corpus and the embedding simulator, then prints the baseline. The baseline gives two numbers: the result set that the index will be measured against for what it misses, and the quality of that set **itself**. ```python # FICTIONAL corpus and FICTIONAL embedding simulator. No real endpoint is called. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) def _to_concepts(m): # the surface form is first mapped to a concept v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): if CONCEPTS[k][1] in dm: v[i] += 1.0 if CONCEPTS[k][2] in dm: v[i] += 1.0 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM): # the concept is then reduced to fewer dimensions v = [0.0] * dim for i, a in enumerate(_to_concepts(m)): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) QUERIES = [] # FICTIONAL query set: the query side is separate from the document side for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) K = 5 VEC = [embed(doc_text(d)) for d in DOCUMENTS] QVECS = [embed(q["text"]) for q in QUERIES] BASELINE = [sorted(range(len(VEC)), key=lambda i: -cosine(qv, VEC[i]))[:K] for qv in QVECS] def concept_precision(retrieved): hits = sum(DOCUMENTS[i]["main"] == QUERIES[n]["concept"] for n, g in enumerate(retrieved) for i in g) return hits / (len(retrieved) * K) print(f"corpus {len(DOCUMENTS)} documents , distinct vectors " f"{len({tuple(v) for v in VEC})} , queries {len(QUERIES)}") print(f"exhaustive search: {len(VEC)} comparisons per query , " f"total {len(VEC) * len(QUERIES)}") print(f"concept precision of the first {K} results {concept_precision(BASELINE):.4f}") ``` ``` corpus 200 documents , distinct vectors 52 , queries 40 exhaustive search: 200 comparisons per query , total 8000 concept precision of the first 5 results 0.4700 ``` The second line is the whole justification for an index: eight thousand cosine similarity computations for forty queries. The third line measures the baseline itself. Exhaustive search carries the target concept of the query in only 0.4700 of the five documents it retrieves. The baseline is **not perfect**; the index cannot rise above it, it can only approach it or move away from it. The last number of the first line will matter in the next section: there are two hundred documents but only **52 distinct vectors**. Documents share concept sets, so they overlap in the vector space. #### Bucket-Based Index The bucket-based index partitions the corpus in advance. A handful of documents chosen by the seed become **representatives**, and every document goes into the bucket of the representative it most resembles. When a query arrives it is first compared with the representatives, and then only the nearest **probe** count of buckets is looked into. ```python def build_buckets(buckets, seed): """FICTIONAL bucket-based index: a single-pass assignment to representatives chosen by the seed.""" r, representatives = rng(seed), [] while len(representatives) < buckets: i = int(r() * len(VEC)) if i not in representatives: representatives.append(i) members = [[] for _ in representatives] for i in range(len(VEC)): members[max(range(buckets), key=lambda c: cosine(VEC[i], VEC[representatives[c]]))].append(i) return representatives, members def bucket_search(qv, representatives, members, probes, k=K): order = sorted(range(len(representatives)), key=lambda c: -cosine(qv, VEC[representatives[c]])) candidates = [i for c in order[:probes] for i in members[c]] return sorted(candidates, key=lambda i: -cosine(qv, VEC[i]))[:k], len(representatives) + len(candidates) def recall(found, base): return len(set(found) & set(base)) / len(base) def bucket_measure(buckets, probes, seed=SEED): representatives, members = build_buckets(buckets, seed) a = comps = 0 retrieved = [] for n, qv in enumerate(QVECS): b, c = bucket_search(qv, representatives, members, probes) retrieved.append(b) a += recall(b, BASELINE[n]) comps += c return a / len(QVECS), comps / len(QVECS), concept_precision(retrieved) print("buckets probes recall comparisons concept precision") for buckets, probes in ((8, 1), (16, 1), (16, 2), (16, 4), (25, 2), (25, 4)): a, c, kk = bucket_measure(buckets, probes) print(f"{buckets:>7} {probes:>6} {a:>6.4f} {c:>11.1f} {kk:>17.4f}") u = [bucket_measure(16, 2, SEED + t)[0] for t in (0, 1)] print(f"two builds buckets=16 probes=2: {u[0]:.4f} , {u[1]:.4f}" f" -- difference {abs(u[0] - u[1]):.4f}") ``` ``` buckets probes recall comparisons concept precision 8 1 0.5800 39.7 0.4250 16 1 0.7200 34.9 0.3800 16 2 0.9500 47.2 0.4700 16 4 1.0000 69.2 0.4700 25 2 0.8900 45.0 0.4500 25 4 1.0000 61.7 0.4700 two builds buckets=16 probes=2: 0.9500 , 0.9600 -- difference 0.0100 ``` The last line gives the threshold of this lesson. The same query set, the same setting, only the seed that chooses the representatives is different: 0.9500 and 0.9600. The build difference carries **0.0100**, so differences of this size in the table do not count as measured. The bucket-based index is not a fixed structure; **which documents are the representatives** moves the result. The main reading of the table is in four numbers. With 16 buckets and 4 probes recall is **1.0000**: the index gives all five of the five documents that exhaustive search gives, and it does so with **69.2** comparisons instead of 200. Going down to 2 probes, recall falls to 0.9500 and the comparisons to 47.2. The 0.0500 in between is five times the threshold, so it is a real loss; in return a third of the comparisons is won. At 1 probe the table breaks: recall is 0.7200 and the comparisons are 34.9. Reading only one bucket fewer wins a quarter of the comparisons and takes away a quarter of the recall. The last column says what this means in the answer. The concept precision of exhaustive search was 0.4700; with 16 buckets and 1 probe it drops to **0.3800**. The index does not only narrow the candidate set, it also lowers the **correctness of the retrieved documents**. The reason the loss looks smaller than the 0.2800 drop in the recall column is that the baseline already retrieves less than half correctly: while continuing to miss what the baseline misses, the index also misses some of what the baseline catches. This is a cost that is not read in one measure, and the rule of the course asks for two columns for this reason. #### Neighborhood Graph and Reachability The second structure does not partition the corpus, it **connects** it. Every document is linked by an edge to a given number of documents it most resembles; the search starts from an entry node and moves to a neighbor if one of its neighbors is closer to the query. Comparison is made only along the path walked. ```python def build_graph(degree): """FICTIONAL neighborhood graph: every document is linked to the 'degree' most similar documents.""" g = [] for i in range(len(VEC)): s = sorted(range(len(VEC)), key=lambda j: -cosine(VEC[i], VEC[j])) g.append([j for j in s if j != i][:degree]) return g def reachable(graph, entry=0): seen, stack = {entry}, [entry] while stack: for j in graph[stack.pop()]: if j not in seen: seen.add(j) stack.append(j) return len(seen) def graph_search(qv, graph, entry=0, k=K): score, comps, expanded = {entry: cosine(qv, VEC[entry])}, 1, set() while True: pending = [i for i in score if i not in expanded] if not pending or max(score[i] for i in pending) < max(score.values()): break i = max(pending, key=lambda x: score[x]) expanded.add(i) for j in graph[i]: if j not in score: score[j] = cosine(qv, VEC[j]) comps += 1 return sorted(score, key=lambda x: -score[x])[:k], comps print("degree edges reachable recall comparisons") for degree in (8, 16, 32, 48): graph = build_graph(degree) a = comps = 0 for n, qv in enumerate(QVECS): b, c = graph_search(qv, graph) a += recall(b, BASELINE[n]) comps += c print(f"{degree:>6} {degree * len(VEC):>5} {reachable(graph):>9}" f" {a / len(QVECS):>6.4f} {comps / len(QVECS):>11.1f}") ``` ``` degree edges reachable recall comparisons 8 1600 27 0.0800 14.0 16 3200 31 0.1750 26.0 32 6400 200 0.5250 59.1 48 9600 200 0.7650 84.9 ``` The reachable column is the explanation of this table and does not depend on any query: it counts **how many documents can be reached** by following the edges from the entry node. At degree 8 this number is **27**, not 200. The graph is fragmented; the component that holds the entry node is smaller than a seventh of the corpus. However much budget the search is given, it cannot look at the remaining 173 documents, so recall hits its ceiling at 0.0800. This is a number of the **structure**, not of the search, and it is independent of the run. The cause of the fragmentation is the number 52 in the first block. Two hundred documents sit on only 52 distinct vectors, so the nearest neighbors of every document are documents with the **same** vector as its own. At degree 8 all eight neighbors of a node are its own twins and the edges do not leave that islet. At degree 32 the number of twins is exceeded, the edges reach other concepts and reachability suddenly becomes 200. Even the connected graph cannot catch the baseline. At degree 48 recall is **0.7650** and the comparisons are **84.9**. On the same query set the bucket-based index, with 16 buckets and 2 probes, gave a recall of 0.9500 with **47.2** comparisons. The graph is both less accurate and more expensive, and on top of that it stores 9600 edges; what the bucket-based index stores is 16 representatives and 200 membership records, that is, 216 values. The added scaffolding does not **improve** the work here, it worsens it on both axes at once. Which structure approximate nearest neighbor search is built with is not a matter of taste; the distribution of the corpus in the vector space makes one structure work and the other not. In a corpus with many repeated documents the graph clogs, the partitioning does not. #### Summary - Exhaustive search needs 200 comparisons per query and carries the target concept in 0.4700 of the five documents it retrieves; the baseline is not perfect and the index cannot rise above it. - The bucket-based index with 16 buckets and 4 probes holds recall at 1.0000 and brings the comparisons down to 69.2, with 2 probes to 47.2 at a recall of 0.9500; with 1 probe recall falls to 0.7200. - The index damages not only the candidate set but also the answer: with 16 buckets and 1 probe the concept precision of the first five falls from 0.4700 to 0.3800. - In the neighborhood graph the ceiling of recall is **reachability**; at degree 8 only 27 documents can be reached from the entry node, so recall stays at 0.0800, and this number is independent of the run. - Even a connected graph stays below the bucket-based index: at degree 48 there are 9600 edges, 84.9 comparisons and a recall of 0.7650; the added scaffolding loses on both axes at once here. - The difference between two index builds is 0.0100; on this workload smaller differences count as unmeasured. #### Next Step In all the measurements of this lesson the query was directed at the **whole** corpus; the index only restricted where in that corpus to look. A real retrieval request often does not arrive like this: only the documents of a certain class, of a certain period or belonging to a certain region are wanted. Is this restriction placed before or after the vector search, and what is the difference between the two. The next lesson puts a metadata filter next to the bucket-based index and counts what happens to recall as the filter selectivity increases: because the index is built over the whole corpus, as the filter narrows, the number of candidates left standing inside the buckets that are looked at collapses, and from some point on search without an index becomes both more accurate and cheaper than the indexed one. ### Metadata Filtering Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/metadata-filtering Combining vector search with a metadata constraint and measuring how the neighborhood breaks down as the filter narrows. Two hundred fictional documents are distributed over sixteen regions; the unfiltered bucket-based index gives a recall of 0.9500 with 47.2 comparisons. When the filter is applied after the search, recall falls to 0.4450 and to 0.0900, and forty of the forty queries return fewer than five results; retrieval raised to fifty pulls this up to 0.8900 and 0.4750 but cannot exceed the ceiling of in-index filtering. In in-index filtering, as the filter width goes from 16 down to 1, the candidate count falls from 31.2 to 2.8 and recall from 0.9500 to 0.4750; the comparisons fall from 47.2 to 18.8 while the exhaustive search of the filtered set falls from 200 to 18. At width 1 the index gives 0.4750 with 18.8 comparisons and exhaustive search gives 1.0000 with 18 comparisons. The difference between two index builds is 0.0100 without a filter and between 0.0850 and 0.1300 with one. The previous lesson directed the query at the whole corpus, and the index only restricted where in that corpus to look. A retrieval request often does not arrive like this. The field crew asks for the records of a certain region, the help desk for the documents of a certain class, the audit for a certain period. This constraint is not a similarity question; it is an exact condition defined over **metadata**, and whether it is satisfied or not admits no degree. There is no single correct order for combining the two procedures. If the filter is applied **after** the vector search, the search runs over the whole index and some of the results are eliminated. If it is applied **before**, the search runs only over the matching documents, but the index was built over the whole corpus and what it promises in the narrowed space is unclear. This lesson measures both orders and puts a third number beside them: the cost of applying the filter and **not using the index at all**. As the filter narrows, there comes a point at which this third option becomes both more accurate and cheaper. - **EV39.** The corpus, the embedding procedure, the query set and the metadata layer are **fictional**; no real vector store or embedding endpoint is called. The seed is **20260218**. - **EV40. The metadata** is a single field in this lesson: every document is given a **region code** (from 1 to 16) derived from the seed. The filter is the condition `region <= width`; as the width gets smaller, the filter becomes more selective. - **EV41.** The filter is **exact**: a document that does not match cannot enter the result at any similarity value. This is a different constraint from a similarity threshold. - **EV42. The correct answer** at every width is **the exhaustive search of the filtered set**: comparison with all the matching documents and the first five. Recall is the ratio of the intersection of the procedure's retrieved set with this five; if the number of matching documents is fewer than five, the denominator is that number. - **EV43. Post-filtering** takes the first `fetch` results from the index and discards the ones that do not match. **In-index filtering** looks into the same buckets but **never takes** the non-matching documents **into the candidate set**. - **EV44.** The index is built with the setting measured in the previous lesson: bucket-based index, 16 buckets, 2 probes. This setting gave a recall of 0.9500 without a filter and is the comparison baseline of this lesson. - **EV45. The comparison count** is the number of cosine similarities computed in a single query: the representative count plus the number of candidates looked at. The filter itself is an equality test and is not included in this number. - **EV46. The candidate count** is the number of documents that pass the filter inside the probed buckets; the breakdown of the neighborhood shows in this column. - **EV47.** The two runs measure the same query set on **two separate index builds**. In the previous lesson this difference came out as 0.0100 without a filter; one finding of this lesson is that the difference grows together with the filter. The filtered query itself was built and measured in the Advanced SQL and Search Engines and Text Retrieval courses of the Databases curriculum; the filtering procedure is not repeated here. The only thing added is what the filter does to the **vector neighborhood**. #### The Space the Filter Narrows The block below builds the corpus and the embedding simulator, adds the metadata layer on top, and prints the unfiltered baseline. ```python # FICTIONAL corpus, FICTIONAL embedding simulator and FICTIONAL metadata. No real endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) def _to_concepts(m): # the surface form is first mapped to a concept v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): if CONCEPTS[k][1] in dm: v[i] += 1.0 if CONCEPTS[k][2] in dm: v[i] += 1.0 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM): # the concept is then reduced to fewer dimensions v = [0.0] * dim for i, a in enumerate(_to_concepts(m)): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) QUERIES = [] # FICTIONAL query set: the query side is separate from the document side for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) K, PROBES = 5, 2 WIDTH = (16, 8, 4, 2, 1) VEC = [embed(doc_text(d)) for d in DOCUMENTS] QVECS = [embed(q["text"]) for q in QUERIES] REGION = [1 + int(rng(SEED ^ (i + 1) * 40503)() * 16) for i in range(len(DOCUMENTS))] def matching(g): return [i for i in range(len(VEC)) if REGION[i] <= g] def truth(qv, g, k=K): # correct answer: exhaustive search of the filtered set return sorted(matching(g), key=lambda i: -cosine(qv, VEC[i]))[:k] def build_buckets(buckets, seed): r, representatives = rng(seed), [] while len(representatives) < buckets: i = int(r() * len(VEC)) if i not in representatives: representatives.append(i) members = [[] for _ in representatives] for i in range(len(VEC)): members[max(range(buckets), key=lambda c: cosine(VEC[i], VEC[representatives[c]]))].append(i) return representatives, members def probed(qv, representatives, members): order = sorted(range(len(representatives)), key=lambda c: -cosine(qv, VEC[representatives[c]])) return [i for c in order[:PROBES] for i in members[c]] def recall(found, correct): return len(set(found) & set(correct)) / max(1, len(correct)) REPS, MEMBERS = build_buckets(16, SEED) full = sum(recall(sorted(probed(qv, REPS, MEMBERS), key=lambda i: -cosine(qv, VEC[i]))[:K], truth(qv, 16)) for qv in QVECS) print(f"corpus {len(DOCUMENTS)} documents , 16 regions , queries {len(QUERIES)}") print("filter width " + " / ".join(str(g) for g in WIDTH) + " -> matching " + " / ".join(str(len(matching(g))) for g in WIDTH)) print(f"unfiltered baseline: bucket-based index (buckets 16 , probes {PROBES}) recall {full / len(QVECS):.4f} , " f"{len(REPS) + sum(len(probed(qv, REPS, MEMBERS)) for qv in QVECS) / len(QVECS):.1f}" f" comparisons , exhaustive search {len(VEC)}") ``` ``` corpus 200 documents , 16 regions , queries 40 filter width 16 / 8 / 4 / 2 / 1 -> matching 200 / 101 / 52 / 27 / 18 unfiltered baseline: bucket-based index (buckets 16 , probes 2) recall 0.9500 , 47.2 comparisons , exhaustive search 200 ``` The five widths narrow the corpus from 200 down to 18 documents. The unfiltered baseline is the line of the previous lesson: the index retrieves 0.9500 of the five that exhaustive search gives, with 47.2 comparisons. The question to ask is this: does this number still hold when a filter is added. #### Applying the Filter Afterward The combination that asks for the least work is to filter the result without changing the search at all. The index gives as many documents as the first `fetch`, the ones that do not match are discarded, and the first five of what remains are returned. ```python def post_filter(qv, g, fetch): upper = sorted(probed(qv, REPS, MEMBERS), key=lambda i: -cosine(qv, VEC[i]))[:fetch] return [i for i in upper if REGION[i] <= g][:K] print("width recall(fetch 5) missing results recall(fetch 15) recall(fetch 50)") for g in WIDTH[1:]: s, missing = [0.0, 0.0, 0.0], 0 for qv in QVECS: d = truth(qv, g) for n, fetch in enumerate((5, 15, 50)): s[n] += recall(post_filter(qv, g, fetch), d) missing += len(post_filter(qv, g, 5)) < len(d) print(f"{g:>5} {s[0] / len(QVECS):>15.4f} {missing:>15} {s[1] / len(QVECS):>16.4f}" f" {s[2] / len(QVECS):>16.4f}") ``` ``` width recall(fetch 5) missing results recall(fetch 15) recall(fetch 50) 8 0.4450 37 0.8900 0.8900 4 0.1950 40 0.6800 0.7600 2 0.1050 40 0.3850 0.6250 1 0.0900 40 0.2600 0.4750 ``` The first column started from 0.9500 without a filter; it falls to **0.4450** at width 8 and to **0.0900** at width 1. The third column gives the reason: below width 4, **forty** of the forty queries return fewer results than the correct answer. The index gives five documents, the filter discards four or five of them, and either one result or none is left. There is no ranking problem here; the result box **empties**. The known remedy is over-retrieval: taking fifteen or fifty documents instead of five and filtering after that. The last two columns show how much this helps. At width 8, going up to fifteen carries 0.4450 to 0.8900 and going up to fifty **adds nothing**. At width 1, retrieving fifty documents gives only 0.4750. Over-retrieval hits a ceiling, and that ceiling is the first column of the next section: the number of documents that pass the filter inside the probed buckets. Asking for fifty documents does not retrieve fifty documents if there are not that many suitable documents in the buckets. #### Applying the Filter Inside the Index The second procedure never takes the non-matching document into the candidate set. The same buckets are looked at, but only the ones that pass the filter are counted and ranked. ```python def in_index(qv, g, representatives, members): candidates = [i for i in probed(qv, representatives, members) if REGION[i] <= g] return sorted(candidates, key=lambda i: -cosine(qv, VEC[i]))[:K], len(candidates) SECOND = build_buckets(16, SEED + 1) print("width candidates recall comparisons exhaustive search two-build difference") for g in WIDTH: a = b = 0.0 total = 0 for qv in QVECS: d = truth(qv, g) r1, n1 = in_index(qv, g, REPS, MEMBERS) a += recall(r1, d) b += recall(in_index(qv, g, *SECOND)[0], d) total += n1 print(f"{g:>5} {total / len(QVECS):>10.1f} {a / len(QVECS):>6.4f} " f"{len(REPS) + total / len(QVECS):>11.1f} {len(matching(g)):>17}" f" {abs(a - b) / len(QVECS):>20.4f}") ``` ``` width candidates recall comparisons exhaustive search two-build difference 16 31.2 0.9500 47.2 200 0.0100 8 15.7 0.8900 31.6 101 0.1300 4 7.8 0.7600 23.9 52 0.0850 2 4.0 0.6250 20.0 27 0.1250 1 2.8 0.4750 18.8 18 0.1100 ``` The candidates column is the breakdown of the neighborhood. Without a filter the index weighs 31.2 documents per query; at width 1 it is **2.8**. The buckets were built over the whole corpus, and a bucket was filled by similarity, not by region. As the filter narrows, the number of documents left standing inside a bucket falls, and the index comes to the point of choosing among three documents where it should choose five results. This is why recall falls from 0.9500 to **0.4750**. The real decision is read in the last three columns. While the comparisons fall from 47.2 to 18.8, **the exhaustive search column** falls from 200 to 18. Without a filter the index did a quarter of the work of exhaustive search; at width 1 it does **more** work. The width 1 row says these two things side by side: the index gives 0.4750 with 18.8 comparisons, and the exhaustive search of the filtered set gives 1.0000 with 18 comparisons. Approximate search is both expensive and wrong here. The crossover is between width 2 and 1: at width 2 the index is still seven comparisons cheaper but leaves a recall of 0.6250, and at width 1 its cheapness is gone too. Below this point, skipping the index and scanning the matching documents from start to finish wins on both axes. The added scaffolding worsening the work is the second example of this lesson, and the cause is not that the index was built badly: the index was built **unaware of the constraint the query would ask**. The last column adds an instability as well. Without a filter the difference between two index builds was 0.0100, that is, the threshold of the previous lesson. With a filter the difference is between 0.0850 and 0.1300, eight to thirteen times the threshold. The same corpus, the same query, the same setting; only which documents are the representatives changes. The recall measured in a setup that works with a narrow filter is the number of that setup and does not carry over when it is built again. #### Summary - The metadata filter is an exact constraint and is different from a similarity threshold; a document that does not match cannot enter the result at any similarity value. - Applying the filter after the search empties the result box: at width 4 and below, forty of the forty queries return fewer than five results and recall falls from 0.9500 to 0.0900. - Over-retrieval closes this loss partly but hits a ceiling; at width 8, retrieving fifty documents does not exceed the 0.8900 that retrieving fifteen documents gives. - In in-index filtering the candidate count falls from 31.2 to 2.8 and recall from 0.9500 to 0.4750; because the buckets are filled by similarity, the filter empties the inside of the bucket. - At width 1 the index gives 0.4750 with 18.8 comparisons and the exhaustive search of the filtered set gives 1.0000 with 18 comparisons; the threshold is around width 2, and below it the index loses on both axes at once. - The difference between two index builds is 0.0100 without a filter and between 0.0850 and 0.1300 with one; the recall measured with a narrow filter depends on the build. #### Next Step In all the measurements of this lesson, the link between the query and the document was built from a single place: the angle of the embedding vectors. The filter narrowed this link but did not change it. Yet the corpus also contains documents that carry the words of the query exactly, and finding them needs no embedding at all; when a document code, a date or an error number is searched, semantic similarity passes by the side of what is searched. The next lesson measures keyword matching and embedding search separately on the same query set, then fuses the ranks of the two and counts what the fusion adds. The expected thing is that the whole is better than its parts; the measured thing is that on one query class the hybrid stays worse than **both of them**. ### Hybrid Search Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/hybrid-search Measuring keyword matching and embedding search separately and together. Three setups run side by side on three fictional query classes: in subscriber language keyword search is 0.0250 and embedding is 0.5000; in document language both are 0.5000; on the document code query keyword search is 1.0000 and embedding is 0.0000. Rank fusion gives 0.4000 in the document language class, that is, worse than both; this holds at constants of 5, 20 and 60. In the code class fusion drops the share of keeping the target document in the first five from 1.0000 to 0.0500. The reason is the number of documents shared by the first five of the two lists: 0.12 in subscriber language, 1.05 in document language. A document that stands second in both lists passes a document that is first in one and absent from the other. Fusion raises recall up to rank twenty from 0.7122 to 0.7214; the place it gains is depth, the place it loses is the first rank. The previous lesson built the link between the query and the document from a single place: the angle of the embedding vectors. The filter narrowed this link, it did not change it. Yet what is searched in a support corpus is not always a meaning. When a document code, an error number or the exact name of a procedure is searched, the question asked is not "what is similar to this" but "where exactly does this occur". Semantic similarity passes by the side of this question. **Hybrid search** is running two rankings together for the same query and gathering their results into a single list. The expectation is that the whole is better than its parts: let keyword search catch the exact match that the embedding cannot see, and let the embedding catch the synonym that keyword search cannot see. This lesson measures three setups (keyword only, embedding only, combined) side by side on three query classes. One of the results does not meet the expectation: in one query class the combined setup stays worse than **both procedures**. - **EV48.** The corpus, the embedding procedure, the tokenizer and the query classes are **fictional**; no real search engine, vector store or embedding endpoint is called. The seed is **20260218**. - **EV49.** The **searchable text** of the document is kept together with its code. Because the code is not a concept surface form, it does not change the embedding vector; this is shown in the block. - **EV50. The three query classes** are derived from the same documents and differ only in their surface forms: **subscriber language** carries the query-side form, **document language** carries the document-side form, and the **code query** carries the document code. All three are forty queries. - **EV51. The keyword ranking** is the number of overlapping tokens. Relevance scoring was built and measured in the Search Engines and Text Retrieval course of the Databases curriculum; it is not repeated here. In this lesson the only function of the keyword side is to produce **a second ranking**. - **EV52. Fusion is rank-based**: the score of a document is the sum of the inverses of its ranks in the two lists, `1 / (constant + rank)`. The constant is 20 by default and is also tested at 5 and 60. - **EV53. The top-one hit rate** is the share of the first result that carries the target concept, **retrieval precision** is the same share for the first five results, and **retrieval recall** is the share of the documents carrying the target concept that are caught in the first twenty. The **target document** column is the share of the document from which the query was derived being in the first five. - **EV54.** Every concept is carried by more than one document; for this reason precision and recall are read at different depths and are not reduced to a single number. - **EV55.** All three rankings are **deterministic**: the same query gives the same list in two runs. The run-independent measure of this lesson is **the number of documents shared by the first five of the two lists**, and it is a property of the pair of procedures, independent of the query. - **EV56.** A query that carries no concept surface form produces a **zero vector**; all similarities become zero and the order falls to the storage order. This is not a defect but the embedding leaving its domain of definition, and it is measured in the code class. - **EV57.** This lesson is the close of the topic; what the retrieved chunk does to the model is not measured and is left to the next topic. #### Three Query Classes ```python # FICTIONAL corpus, FICTIONAL embedding simulator, FICTIONAL query classes. No real endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, pieces of at most five characters b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL semantic axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier threshold", "which band am i in"), "unit_rate": ("tariff", "unit price", "fee per cubic metre"), "subscriber_type": ("tariff", "subscriber type", "home or shop"), "switch": ("tariff", "tariff switch", "changing plan"), "fault_code": ("fault", "fault code", "error id"), "leak": ("fault", "line leakage", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "who will be sent"), "index_value": ("meter", "index value", "what the dial shows"), "reading": ("meter", "reading procedure", "how it gets saved"), "replacement": ("meter", "meter replacement", "new hardware"), "verify": ("meter", "reading verification", "typed in wrong"), "planned": ("outage", "planned outage", "announced stoppage"), "notice": ("outage", "notice channel", "where i hear about it"), "duration": ("outage", "outage duration", "how long it lasts"), "restoration": ("outage", "service restoration", "when it comes back"), "periodic": ("maintenance", "periodic maintenance", "regular checkup"), "valve": ("maintenance", "valve inspection", "tap shutoff routine"), "filter_swap": ("maintenance", "filter swap", "cleaning part"), "calibration": ("maintenance", "calibration record", "gauge accuracy"), } KEYS = list(CONCEPTS) COMMON = ["entry", "region", "term", "step", "form", "approval", "note"] DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paragraphs, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paragraphs.append(f"{CONCEPTS[y][1]} for {o} is kept and written together with {CONCEPTS[main][1]}") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paragraphs": paragraphs}) def doc_text(d): return d["title"] + " " + " ".join(d["paragraphs"]) def _to_concepts(m): # the surface form is first mapped to a concept v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): if CONCEPTS[k][1] in dm: v[i] += 1.0 if CONCEPTS[k][2] in dm: v[i] += 1.0 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM): # the concept is then reduced to fewer dimensions v = [0.0] * dim for i, a in enumerate(_to_concepts(m)): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) K, CONSTANT = 5, 20 SEARCHABLE = [d["code"] + " " + doc_text(d) for d in DOCUMENTS] # the code enters the searchable text of the document VEC = [embed(a) for a in SEARCHABLE] TOKEN_SETS = [set(tokenize(a)) for a in SEARCHABLE] def queries(mode): # three separate surface forms from the same documents q = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] o = COMMON[int(r() * len(COMMON))] q.append({"text": {"subscriber": f"{CONCEPTS[d['main']][2]} {o}", "document": f"{CONCEPTS[d['main']][1]} {o}", "code": f"{d['code']} numbered document"}[mode], "concept": d["main"], "code": d["code"]}) return q QUERY_CLASSES = {k: queries(k) for k in ("subscriber", "document", "code")} def keyword_rank(m): # by the number of overlapping tokens qb = set(tokenize(m)) return sorted(range(len(DOCUMENTS)), key=lambda i: -len(qb & TOKEN_SETS[i])) def embedding_rank(m): qv = embed(m) return sorted(range(len(DOCUMENTS)), key=lambda i: -cosine(qv, VEC[i])) def hybrid_rank(m, constant=CONSTANT): # rank fusion pk = {i: 1.0 / (constant + n) for n, i in enumerate(keyword_rank(m))} pe = {i: 1.0 / (constant + n) for n, i in enumerate(embedding_rank(m))} return sorted(range(len(DOCUMENTS)), key=lambda i: -(pk[i] + pe[i])) for k, s in QUERY_CLASSES.items(): print(f"{k:10s} example query: {s[0]['text']}") print(f"document code does not change the embedding: " f"{embed(SEARCHABLE[0]) == embed(doc_text(DOCUMENTS[0]))} , code query vector length " f"{sum(x * x for x in embed(QUERY_CLASSES['code'][0]['text'])):.1f}") ``` ``` subscriber example query: how it gets saved form document example query: reading procedure form code example query: SY194 numbered document document code does not change the embedding: True , code query vector length 0.0 ``` The three queries derived from the same document and ask the same thing; the only place they differ is the surface form. The last line measures two things. Adding the document code to the searchable text **does not change** the embedding vector: the code is not a concept surface form, so the embedding does not see it. The second is harsher: the vector length of the code query is **0.0**. Because there is no recognized concept in the query, the vector is zero, all the cosine similarities come out zero and the ranking falls to the storage order. The embedding is not mistaken on this query, it stays **outside its domain of definition**. #### The Table of the Three Setups ```python def measure(qset, rank): first = prec = rec = target = 0.0 for q in qset: relevant = {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]} s = rank(q["text"]) first += s[0] in relevant prec += len(set(s[:K]) & relevant) / K rec += len(set(s[:20]) & relevant) / len(relevant) target += any(DOCUMENTS[i]["code"] == q["code"] for i in s[:K]) return [x / len(qset) for x in (first, prec, rec, target)] SETUPS = (("keyword", keyword_rank), ("embedding", embedding_rank), ("hybrid", hybrid_rank)) print("class setup top1 precision@5 recall@20 target document@5") for mode, qset in QUERY_CLASSES.items(): for name, f in SETUPS: d = measure(qset, f) print(f"{mode:10s} {name:9s} {d[0]:>6.4f} {d[1]:>11.4f} {d[2]:>9.4f} {d[3]:>17.4f}") print("hybrid top1 in the document class , constant 5 / 20 / 60: " + " / ".join(f"{measure(QUERY_CLASSES['document'], lambda m, c=c: hybrid_rank(m, c))[0]:.4f}" for c in (5, 20, 60))) ``` ``` class setup top1 precision@5 recall@20 target document@5 subscriber keyword 0.0250 0.0800 0.0967 0.0250 subscriber embedding 0.5000 0.4700 0.7122 0.1500 subscriber hybrid 0.4250 0.2900 0.4616 0.1250 document keyword 0.5000 0.3650 0.6616 0.1750 document embedding 0.5000 0.4700 0.7122 0.1500 document hybrid 0.4000 0.4250 0.7214 0.1250 code keyword 1.0000 0.2500 0.1950 1.0000 code embedding 0.0000 0.0800 0.1078 0.0250 code hybrid 0.0000 0.0850 0.1950 0.0500 hybrid top1 in the document class , constant 5 / 20 / 60: 0.4500 / 0.4000 / 0.4000 ``` The first two triples build the justification for hybrid search. In subscriber language keyword search is 0.0250 and embedding is 0.5000; on the code query keyword search is **1.0000** and embedding is **0.0000**. The two procedures work in exactly opposite places on the same corpus, and choosing a single procedure means sacrificing one of the query classes entirely. The hybrid rows show that this sacrifice does not go away. In subscriber language the hybrid is 0.4250, a little below the embedding; this is the expected cost. In document language, while both procedures give 0.5000, the hybrid gives **0.4000**. The total is worse than **both** of its parts. The last line says that this is not the game of one choice of constant: at 5, 20 and 60 it is 0.4500 / 0.4000 / 0.4000, and all three are below 0.5000. The code class is the most expensive row. Keyword search puts the target document in the first five in forty of the forty queries; the hybrid drops this to **0.0500**. In the query class that is the reason hybrid search exists, fusion sinks the working procedure together with the one that does not work. The cause is in the table itself: the embedding side works with a zero vector in this class and the order it produces carries no information, but the fusion does not know this and gives that order **equal weight**. One column looks in the opposite direction. In document language recall@20 is 0.6616 for keyword search, 0.7122 for the embedding, and **0.7214** for the hybrid. Up to rank twenty the fusion really does gain. In the code class too, recall@20 rises from 0.1078 to 0.1950. The place where fusion gains is **depth**, the place where it loses is the **first rank**, and in a retrieval pipeline what enters the prompt is the first few chunks. #### What the Fusion Loses ```python print("class shared in first five both correct hybrid wrong") for mode, qset in QUERY_CLASSES.items(): shared = lost = 0 for q in qset: relevant = {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]} kw, em, hy = (f(q["text"]) for _, f in SETUPS) shared += len(set(kw[:K]) & set(em[:K])) lost += kw[0] in relevant and em[0] in relevant and hy[0] not in relevant print(f"{mode:10s} {shared / len(qset):>20.2f} {lost:>25}") def relevant_docs(q): return {i for i, d in enumerate(DOCUMENTS) if d["main"] == q["concept"]} q = next(x for x in QUERY_CLASSES["document"] if keyword_rank(x["text"])[0] in relevant_docs(x) and embedding_rank(x["text"])[0] in relevant_docs(x) and hybrid_rank(x["text"])[0] not in relevant_docs(x)) print(f"\nquery: {q['text']} -- target concept: {q['concept']}") for name, f in SETUPS: print(f" {name:9s}", [f"{DOCUMENTS[i]['code']}/{DOCUMENTS[i]['main']}" for i in f(q["text"])[:3]]) ``` ``` class shared in first five both correct hybrid wrong subscriber 0.12 0 document 1.05 3 code 4.03 0 query: pressure drop region -- target concept: pressure keyword ['AR002/pressure', 'AR015/crew', 'AR018/pressure'] embedding ['AR079/pressure', 'AR088/pressure', 'AR101/crew'] hybrid ['AR033/crew', 'AR015/crew', 'AR018/pressure'] ``` The shared documents column is a number that belongs not to any query but to the pair of procedures itself, and it is independent of the run. In subscriber language the first five of the two lists share **0.12** documents on average: the procedures almost never agree. In document language it is 1.05, that is, one in five. The example shows the consequence of this number. Keyword search puts AR002 and the embedding puts AR079 at the first rank; both carry the target concept, so **both procedures are correct**. The hybrid puts AR033 at the first rank, and that document belongs to the `crew` concept. AR002 is first in one list and far behind in the other; its score comes from a single large term. AR033 is high in both lists but first in neither, and the sum of two medium terms passes one large term. Rank fusion rewards **the document of compromise**; when the two sides do not agree, the second choice of both wins. In the document language class this happens in three of the forty queries and never in the opposite direction: there is not a single example in which the hybrid finds the right one on a query where both procedures were wrong. The counterpart of the three-query drop in the first rank is zero gain. The 4.03 in the code row is the trap. The high overlap here is not agreement: because the embedding side works with a zero vector, its list is the storage order, and the tail of the keyword side after the exact match is the same order. The two lists share **the same uninformed tail**. Fusion cannot tell apart the two lists being the same from both being empty; if it could, it would give keyword search full weight in this class. Hybrid search asks not for an average but for a **decision**: which side has the say on which query. #### Summary - On three query classes the two procedures work in opposite places: in subscriber language keyword search is 0.0250 and embedding is 0.5000, on the code query keyword search is 1.0000 and embedding is 0.0000. - A query that carries no concept surface form produces a zero vector; the embedding is not mistaken, it leaves its domain of definition and its ranking falls to the storage order. - In the document language class both procedures give 0.5000 while rank fusion gives 0.4000; the combined setup is worse than both and this holds at constants of 5, 20 and 60. - The cause is that the first five of the two lists share 1.05 documents: the document of compromise that stands second in both lists passes the document that is first in one and absent from the other. - In the code class fusion drops the share of the target document being in the first five from 1.0000 to 0.0500; giving equal weight to an uninformed list sinks the working procedure as well. - Fusion gains in depth: in document language recall@20 rises from 0.7122 to 0.7214. The gain is at rank twenty, the loss is at the first rank, and what enters the prompt is the first rank. #### Next Step Through six lessons a single question was asked and answered better each time: which chunk of the corpus belongs to this query. The embedding became a search key, the domain fit was measured, the similarity measure was chosen, what the index loses was counted, the point where the filter breaks the neighborhood was found and the fusion of two rankings was weighed. All of these lessons are on the **retrieval** side, and the measure of all of them is the retrieved chunk itself. What is not measured is what comes after. The found chunk is put into a prompt and given to a model; from that point on, the right chunk producing the right answer is an assumption and was never tested in this topic. A wrong answer can be produced even though the right chunk was retrieved, the model can give the right answer by chance even though the wrong chunk was retrieved, and worst of all, the retrieved chunk can lead the model to produce a **worse** answer than its own answer without retrieval. The next topic builds the retrieval pipeline from end to end and separates two columns in its first lesson: was the right chunk retrieved, and was the right answer produced with what was retrieved. ## Retrieval Pipeline Counting what a retrieved chunk does to the model and what each step of the pipeline costs on its own: comparing three runs, no retrieval, retrieval and retrieval with wrong chunks, and the run with wrong chunks falling below the baseline; what document preparation adds to retrieval recall and what it throws away; recall that rises and chunk and token counts that multiply as chunk size and overlap grow; the stale chunk rate that grows and the number of re-embedded chunks as the refresh interval lengthens; what one-step query transformation adds in calls and what it gains on a short query and loses on a long one; what reranking adds to the precision of the top k against the growing candidate count and latency class; the share of answers changed by the number of chunks that fit the context limit and by giving the same chunks in a different order; the four-cell contingency of retrieval and generation of the right chunk and the right answer and how a single number hides all four; what retrieval and fine-tuning add on the tasks of knowledge freshness, format compliance and domain language; what row-level and block-level chunking of a source carrying tables does to recall. ### What Is Retrieval-Augmented Generation Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/what-is-retrieval-augmented-generation Running the same forty queries in three setups over a fictional corpus of two hundred documents and printing retrieval and generation as two separate columns: the no-retrieval baseline scores 0.2000 on generation, while the pipeline built with an in-domain embedding raises retrieval recall to 0.9500 and generation to 0.9250, and the same pipeline built with an out-of-domain embedding leaves recall at 0.1750 and drops generation to 0.0500, below the baseline. The cost is written side by side: from 3033 tokens to 4848 tokens, from zero retrieval steps to twenty-seven thousand seven hundred twenty steps, from 379.1 tokens per correct answer to 131.0 tokens, and to 2457.5 tokens in the wrong-chunk run. The run of six seeds gives this workload a threshold of 0.0750, and one of the thirty-eight queries that receive the correct chunk is still answered wrongly. The six lessons of the previous topic did a single job: given a query, find a chunk in the corpus. The embedding was set up as a search key, the loss of domain fit was measured, a similarity measure was chosen, an indexed search replaced the exhaustive search, a metadata filter was added, and two rankings were merged. In all six the measure sat in the same place: was the retrieved chunk correct. The thing nobody asked is what that chunk does once it is handed to the model. **Retrieval-augmented generation** means putting chunks selected from the corpus into the prompt and calling the model with those chunks. The rule this lesson sets up holds for sixteen lessons: **retrieval and generation are measured separately.** When the two are reduced to one number, it is not visible which stage broke things. - **RP1.** The code is a **simulator**; no real embedding or model endpoint is called. The corpus, the concept axis and the queries are **fictional**, seed **20260218**. - **RP2.** The definition of an embedding, cosine similarity and neighborhood were measured in the Natural Language Processing and Computer Vision course and are **not repeated**. Here the embedding is only a **search key**. - **RP3.** Every concept has a **document surface** (`tier limit`) and a different **query surface** (`which band am i in`), and the two share no token: the word the subscriber asks with is not the word the record is written in. - **RP4.** The simulator's internal trace recognizes the **document surface**, and recognizes only a seven-concept part of the query surface. This is the only source of the no-retrieval baseline. - **RP5.** In this lesson the retrieval unit is **the source's own paragraph boundary**, and it is called a **chunk**; that the boundary is a decision is measured in the third lesson. - **RP6.** **Retrieval recall** is the share of queries for which at least one chunk carrying the target concept is in the top three; **retrieval precision** is the proportion of target-concept chunks in the top three; the **class hit rate** is the proportion of the top three that come from the target class. The class hit rate is the easy measure, the concept hit rate is the hard one. - **RP7.** The **generation measure** is the answer naming the target concept, and it is not summed into one column with the retrieval measure. The rate limit depends on the call count: the thirty-seventh of forty calls returns empty, and because it falls in the same place in all three setups it does not distort the comparison. #### Corpus and Unit of Measure The block below sets up the fictional corpus, the concept axis, the embedding simulator and the query set. Each of the twenty concepts is tied to a class and is written with two surfaces. ```python # FICTIONAL. The corpus, concept axis and embedding below are a simulator; no real # embedding endpoint is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- chunk: in this lesson the unit is the source's own paragraph boundary CHUNKS = [{"code": d["code"], "label": d["label"], "text": p, "concepts": [k for k in KEYS if CONCEPTS[k][1] in p]} for d in DOCUMENTS for p in d["paras"]] print("documents %d chunks %d concepts %d queries %d tokens per chunk %.1f" % (len(DOCUMENTS), len(CHUNKS), len(KEYS), len(QUERIES), sum(len(tokenize(x["text"])) for x in CHUNKS) / len(CHUNKS))) print("labels", {s: sum(d["label"] == s for d in DOCUMENTS) for s in CLASSES}) ``` ``` documents 200 chunks 693 concepts 20 queries 40 tokens per chunk 13.0 labels {'tariff': 52, 'fault': 46, 'meter': 39, 'outage': 32, 'maintenance': 31} ``` The scale reads off from here: six hundred ninety-three chunks, thirteen tokens per chunk. The answer to a query sits inside a stack of about nine thousand tokens, and the context window of 240 tokens holds only a small part of it. #### Three Setups The second block sets up the language model simulator and runs the same forty queries in three setups. The simulator comes from the family built in the Introduction to AI Engineering course: it counts tokens, carries a context window and a stopping condition, can hit a rate limit, and gives the same response when it is called with the same prompt and the same seed. The only adaptation is the response space: the model writes one of twenty concepts. The **no-retrieval baseline** calls the model with the query alone. The **retrieval** setup puts the three nearest chunks, found with the in-domain embedding, into the prompt. The **wrong-chunk run** builds the same pipeline with an out-of-domain embedding, and it is not a hand-broken scaffolding: it is the pipeline that results when an embedding that does not recognize the query language is chosen. ```python import math # ---- FICTIONAL language model simulator: the response space is twenty concepts WINDOW, MAX_OUTPUT = 240, 24 PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"} FREE_FORM = ["request linked to {k} concept and logged", "{k} record created and passed to the relevant department", "your report was received and the relevant unit will reply"] COUNTER = [0] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def call(prompt, seed=SEED, temperature=0.6, max_output=MAX_OUTPUT): COUNTER[0] += 1 b = tokenize(prompt) if len(b) > WINDOW: return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: # the rate limit depends on the call count return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) dm = " " + prompt.lower() + " " p = {k: 0.15 for k in KEYS} for k in KEYS: p[k] += 2.0 * dm.count(CONCEPTS[k][1]) # the model recognizes the document language if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm: p[k] += 2.0 # the known share of the query language if temperature <= 0.0: k = max(sorted(p), key=lambda a: p[a]) else: w = [math.exp(p[a] / temperature) for a in sorted(p)] k = sorted(p)[pick(r(), [x / sum(w) for x in w])] y = k if "one" in b else FREE_FORM[int(r() * 3)].format(k=k) cb = tokenize(y) if len(cb) > max_output: cb, y = cb[:max_output], " ".join(cb[:max_output]) return {"status": "ok", "input": len(b), "output": len(cb), "response": y} def read_concept(y): return next((s for s in y.split() if s in KEYS), None) INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) + " . Reply with one concept . ") VEC = {True: [embed(x["text"]) for x in CHUNKS], False: [embed(x["text"], domain=False) for x in CHUNKS]} def run(mode, seed=SEED, k=3): COUNTER[0] = 0 recall = precision = label_hit = correct = inp = out = 0 for s in QUERIES: if mode == "no retrieval": prompt = INSTRUCTION + "Request: " + s["text"] else: domain = mode == "retrieval" v = embed(s["text"], domain=domain) top = sorted(range(len(CHUNKS)), key=lambda i: -cosine(v, VEC[domain][i]))[:k] prompt = (INSTRUCTION + "Source: " + " . ".join(CHUNKS[i]["text"] for i in top) + " . Request: " + s["text"]) h = [s["concept"] in CHUNKS[i]["concepts"] for i in top] recall += any(h) precision += sum(h) / k label_hit += sum(CHUNKS[i]["label"] == s["label"] for i in top) / k y = call(prompt, seed) inp, out = inp + y["input"], out + y["output"] correct += read_concept(y["response"]) == s["concept"] n = len(QUERIES) return {"recall": recall / n, "precision": precision / n, "label_hit": label_hit / n, "generation": correct / n, "tokens": inp + out, "chunks": 0 if mode == "no retrieval" else k * n, "steps": 0 if mode == "no retrieval" else len(CHUNKS) * n, "correct": correct} print("setup RETRIEVAL: recall precision label GENERATION COST: tokens chunks" " steps tokens/correct") for mode in ("no retrieval", "retrieval", "wrong chunks"): a = run("wrong" if mode.startswith("wrong") else mode) e = (" - - -" if mode == "no retrieval" else " %.4f %.4f %.4f" % (a["recall"], a["precision"], a["label_hit"])) print("%-13s%s %.4f %6d %5d %6d %14.1f" % (mode, e, a["generation"], a["tokens"], a["chunks"], a["steps"], a["tokens"] / a["correct"])) ``` ``` setup RETRIEVAL: recall precision label GENERATION COST: tokens chunks steps tokens/correct no retrieval - - - 0.2000 3033 0 0 379.1 retrieval 0.9500 0.9500 0.9500 0.9250 4848 120 27720 131.0 wrong chunks 0.1750 0.1750 0.2500 0.0500 4915 120 27720 2457.5 ``` #### Three Readings of the Table The first reading is what retrieval adds. The no-retrieval baseline is **0.2000**, and the queries it answers correctly are the ones that happen to fall on concepts the simulator recognizes from the query language. The pipeline built with the in-domain embedding raises generation to **0.9250**, against that baseline of 0.2000. The second reading is the cost. Total tokens rise **from 3033 to 4848**, three chunks are retrieved for each query and the whole corpus is scanned: six hundred ninety-three comparisons per query, **27720 retrieval steps** over forty queries. A step is not a model call, but it does not come free either. Read per correct answer, the arithmetic reverses: **from 379.1 tokens to 131.0 tokens**. The third reading is the second claim of this course: **retrieval also breaks things.** The wrong-chunk run spends **more** tokens than the baseline (4915) and drops generation to **0.0500**, that is, below the baseline's **0.2000**. The retrieved chunks are written in the document language and the model recognizes the document language, so it reads the wrong concepts **with confidence**; the no-retrieval model's uncertain guess turns into the confident error of a wrong source. The cost per correct answer rises to **2457.5 tokens**. The retrieval column is also read with two numbers: in the out-of-domain run the class hit rate is **0.2500**, which is the level of a random pick among five classes, and the concept hit rate is **0.1750**. Had a single hit number been printed, it would have stayed unclear which measure was being discussed. #### Threshold and Separating the Two Columns A difference counts as a difference only if it is larger than the run-to-run variability. This threshold was measured on its own workload in the Building Applications with Model APIs course; this course does **not inherit** it. The third block runs the same three setups with six seeds and then counts where the retrieval and generation columns come apart. ```python print("setup generation hits of six seeds spread") threshold = 0.0 for mode in ("no retrieval", "retrieval", "wrong"): v = [run(mode, SEED + i)["generation"] for i in range(6)] threshold = max(threshold, max(v) - min(v)) print("%-13s %s %.4f" % (mode, " ".join("%.4f" % x for x in v), max(v) - min(v))) print("threshold of this course %.4f (%d of the forty queries)" % (threshold, round(threshold * 40))) COUNTER[0] = 0 gc = gw = 0 for s in QUERIES: v = embed(s["text"]) top = sorted(range(len(CHUNKS)), key=lambda i: -cosine(v, VEC[True][i]))[:3] if not any(s["concept"] in CHUNKS[i]["concepts"] for i in top): continue prompt = (INSTRUCTION + "Source: " + " . ".join(CHUNKS[i]["text"] for i in top) + " . Request: " + s["text"]) d = read_concept(call(prompt)["response"]) == s["concept"] gc, gw = gc + d, gw + (not d) print("%d queries with the correct chunk: %d correct, %d wrong" % (gc + gw, gc, gw)) ``` ``` setup generation hits of six seeds spread no retrieval 0.2000 0.2000 0.2250 0.1500 0.2250 0.1750 0.0750 retrieval 0.9250 0.9250 0.9250 0.9250 0.9250 0.9250 0.0000 wrong 0.0500 0.0500 0.1000 0.0750 0.1250 0.0750 0.0750 threshold of this course 0.0750 (3 of the forty queries) 38 queries with the correct chunk: 37 correct, 1 wrong ``` The threshold of this course is **0.0750**: three of the forty queries. A generation difference smaller than this is **unmeasured** on this workload. The spread also changes from setup to setup: 0.0750 for the no-retrieval baseline, **0.0000** for the retrieval pipeline. The output of a model that was given the source does not depend on the seed, because the correct concept scores far above the others. Uncertainty grows where knowledge is absent; this is the less-told second gain of retrieval. The retrieval measure, on the other hand, is independent of the seed, because the embedding and the ranking are deterministic. For this reason the threshold applies only to the **generation** column. The payoff of keeping the two columns apart is the last line: of the **thirty-eight** queries that receive the correct chunk, thirty-seven are answered correctly and **one** wrongly. For that query the failure is in the generation stage; had a single number been printed, that query would have fallen into the same box as the two queries whose chunk could not be retrieved. #### Summary - Retrieval-augmented generation is putting chunks selected from the corpus into the prompt and calling the model with those chunks; the rule of the course is that retrieval and generation are **measured separately**. - On the same forty queries the no-retrieval baseline gives 0.2000 generation, and the pipeline built with the in-domain embedding gives 0.9250. - The gain is not read without the cost written: total tokens rise from 3033 to 4848, three chunks and 693 comparisons are added per query; tokens per correct answer, however, fall from 379.1 to 131.0. - The same pipeline built with an out-of-domain embedding drops generation to 0.0500, that is, **below** the no-retrieval baseline, and does so while spending more tokens: retrieval also breaks things. - The measurement threshold of this workload comes out as 0.0750 from the six-seed run; the spread of the retrieval pipeline is 0.0000. - One of the thirty-eight queries that receive the correct chunk is still answered wrongly. #### Next Step In this lesson the corpus arrived ready: clean paragraphs, boundaries in the right places, code and class fields kept apart. A real source does not arrive like this; field records carry form headers, page footers, sentences split in the middle of a line, and code fields embedded in the text. What these leftovers do to retrieval was never asked: which preparation raises recall, and **what the preparation itself throws away**. The next lesson starts from the raw form of the same corpus, measures three levels of preparation on the same query set, and counts the lines that a crude cleaning rule throws away. ### Document Preparation Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/document-preparation Measuring four preparation levels on the same forty queries, starting from the raw form of the same fictional corpus: the source left as one block per document pushes retrieval recall to the highest value, 1.0000, but every request overflows the context window and generation falls to 0.0000, so had a single column been printed the worst setup would have won. Crude line-based splitting stores 1805 chunks and gives 0.9250 on generation, the same as structure-aware preparation with 693 chunks, which reproduces exactly where the previous lesson started; what separates them is the stored vector count, not the hit rate. The repetition-based boilerplate filter drops 798 lines and 7021 tokens while destroying 96 of 568 concept-document links and erasing the evidence of the main concept in eight documents; when the threshold is lowered to two, the loss rises to 283 links and sixty documents. The retrieval and generation columns show none of this loss. The previous lesson found the corpus ready: two hundred documents, six hundred ninety-three clean paragraphs, code and class fields kept apart. That preparation did not happen by itself. A field team's records carry a form header, a code line, sentences split in the middle of a line and a page footer; all of it enters the retrieval pipeline. This lesson asks a single question: **what does preparation add to retrieval recall, and what does the preparation itself throw away.** The second half matters as much as the first, because cleaning is an act of selection and every line not selected is deleted from the corpus. The **procedure** of parsing, cleaning and metadata extraction was measured in the Data Collection and Preparation course and is **not repeated** here; the only thing added here is that procedure's counterpart in the retrieval pipeline. - **RP8.** The code is a **simulator**; the corpus, the raw source format and the queries are **fictional**, seed **20260218**. The embedding and the language model are the definitions of the previous lesson. - **RP9.** The raw source carries four leftovers: a **form header** repeated in every document, a **code and class field** embedded in the text (`document code SY003 class meter`), a split in the middle of a line in **0.45** of the paragraphs, and a **page footer** repeated in every document. The continuation line is indented by two spaces. - **RP10.** The split point falls after the first, fifth or seventh word of the paragraph. The first and seventh points **cut a document surface in the middle** and make that surface invisible to search; the fifth point leaves both surfaces intact. - **RP11.** The **unsegmented raw block** leaves one chunk per document. **Line-based splitting** counts every raw line as a separate chunk. The **repetition-based boilerplate filter** drops a line that appears in more than three documents. **Structure-aware preparation** merges the continuation lines, drops the known header and footer, and takes the code and class field out of the text and writes it into a field. - **RP12.** The measures are the definitions of the previous lesson and the top three chunks are retrieved; the threshold is **0.0750**. The cost has two units: the **stored chunk count** (each chunk holds one vector, thirty-two numbers per vector) and the **stored tokens**. - **RP13.** A **concept-document link** is the occurrence of a concept's document surface in a document's text. This number is **independent** of the query set and of the run. #### The State of the Raw Source The first block sets up the fictional corpus, applies the raw source format on top of it and prints the raw form of one document. The printed document has three splits, and all three are at different points. ```python # FICTIONAL. The corpus, concept axis, embedding and raw source format are a simulator; # no real embedding endpoint is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- FICTIONAL raw source: form header, code field, split paragraphs, page footer HEADER, FOOTER = "regional directorate field record sheet", "for internal use only do not distribute" RAW = [] for i, d in enumerate(DOCUMENTS): r = rng(SEED + 3011 * i) s = [HEADER, f"document code {d['code']} class {d['label']}", d["title"]] for p in d["paras"]: w = p.split() if r() < 0.45: k = [1, 5, 7][pick(r(), [0.4, 0.25, 0.35])] s += [" ".join(w[:k]), " " + " ".join(w[k:])] # continuation line indented by two spaces else: s.append(p) RAW.append(s + [FOOTER]) j = next(i for i, s in enumerate(RAW) if sum(x.startswith(" ") for x in s) >= 2 and len(s) <= 10) print("raw source %s, %d lines:" % (DOCUMENTS[j]["code"], len(RAW[j]))) for x in RAW[j]: print(" |" + x) print("total raw lines %d, ready paragraphs %d" % (sum(len(s) for s in RAW), sum(len(d["paras"]) for d in DOCUMENTS))) ``` ``` raw source SY003, 10 lines: |regional directorate field record sheet |document code SY003 class meter |reading routine procedure |reading check for note goes | with reading routine and is filed |meter swap for form goes with reading | routine and is filed |reading | routine for task goes with reading routine and is filed |for internal use only do not distribute total raw lines 1805, ready paragraphs 693 ``` The eighth line is a single word: `reading`. Its continuation is one line below. In this document the surface `reading routine` appears four times, but because the line boundary cuts the surface in the middle, two of those occurrences count as **absent** on the search side. The sixth line does the same thing. The split is not a loss; it is a loss **unless preparation is done**. #### Four Preparation Levels The second block sets up the language model simulator, produces the four preparation levels and measures all four on the same forty queries. Because the instruction forces a one-concept answer, the model writes the label directly. ```python import math # ---- FICTIONAL language model simulator: the previous lesson's simulator, response space is twenty concepts WINDOW = 240 PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"} COUNTER = [0] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def call(prompt, seed=SEED, temperature=0.6): COUNTER[0] += 1 b = tokenize(prompt) if len(b) > WINDOW: return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) dm = " " + prompt.lower() + " " p = {k: 0.15 for k in KEYS} for k in KEYS: p[k] += 2.0 * dm.count(CONCEPTS[k][1]) # the model recognizes the document language if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm: p[k] += 2.0 w = [math.exp(p[a] / temperature) for a in sorted(p)] y = sorted(p)[pick(r(), [x / sum(w) for x in w])] return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y} INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) + " . Reply with one concept . ") def merge_lines(s): c = [] for x in s: if x.startswith(" ") and c: c[-1] += " " + x.strip() # the continuation line is appended to the previous one else: c.append(x) return c COUNTS = {} for s in RAW: for x in set(merge_lines(s)): COUNTS[x] = COUNTS.get(x, 0) + 1 def prepare(mode, threshold=3): p, dropped = [], [] for i, s in enumerate(RAW): if mode == "raw block": c = [" ".join(s)] elif mode == "line-based": c = [x.strip() for x in s] elif mode == "repeat filter": c = [x for x in merge_lines(s) if COUNTS[x] < threshold] dropped += [x for x in merge_lines(s) if COUNTS[x] >= threshold] else: c = [x for x in merge_lines(s) if x not in (HEADER, FOOTER, DOCUMENTS[i]["title"]) and not x.startswith("document code")] p += [{"code": DOCUMENTS[i]["code"], "label": DOCUMENTS[i]["label"], "text": x, "concepts": [k for k in KEYS if CONCEPTS[k][1] in x]} for x in c] return p, dropped def measure(p, k=3): COUNTER[0] = 0 v = [embed(x["text"]) for x in p] a = [0] * 6 for s in QUERIES: q = embed(s["text"]) top = sorted(range(len(p)), key=lambda i: -cosine(q, v[i]))[:k] h = [s["concept"] in p[i]["concepts"] for i in top] a[0] += any(h) a[1] += sum(h) / k a[2] += sum(p[i]["label"] == s["label"] for i in top) / k y = call(INSTRUCTION + "Source: " + " . ".join(p[i]["text"] for i in top) + " . Request: " + s["text"]) a[3] += y["status"] == "context_overflow" a[4] += y["input"] + y["output"] a[5] += y["response"] == s["concept"] return [x / len(QUERIES) for x in a[:4]] + [a[4], a[5] / len(QUERIES)] print("preparation chunks stored recall precision label generation overflow tokens") for mode in ("raw block", "line-based", "repeat filter", "structure-aware"): p, at = prepare(mode) m = measure(p) print("%-16s%6d %8d %6.4f %9.4f %6.4f %10.4f %8.2f %8d" % (mode, len(p), sum(len(tokenize(x["text"])) for x in p), m[0], m[1], m[2], m[5], m[3], m[4])) ``` ``` preparation chunks stored recall precision label generation overflow tokens raw block 200 14925 1.0000 0.9667 0.9667 0.0000 1.00 11839 line-based 1805 14925 0.9500 0.9500 0.9500 0.9250 0.00 4152 repeat filter 695 7904 0.9500 0.9500 0.9500 0.9250 0.00 4850 structure-aware 693 8985 0.9500 0.9500 0.9500 0.9250 0.00 4848 ``` The first row is the most important number of this lesson. The unprepared source is **the best of the retrieval column**: recall **1.0000**, precision and class hit rate **0.9667**. The reason is clear: when one chunk is kept per document, that chunk carries every concept in the document, and it becomes harder to miss the target. The generation hit rate on the same row is **0.0000**. The context overflow rate is **1.00**: all forty requests failed to fit the window, because three documents at once were put into the prompt. **Had a single column been printed, this setup would have won.** Line-based splitting is the other extreme. Recall falls to **0.9500**, because the split surfaces can no longer be searched; on the other hand, the tokens per request are at their lowest (**4152**), since the retrieved lines are short. The cost is paid in the stored chunk count: **1805** vectors, that is, **2.6 times** that of structure-aware preparation, and with the stored tokens staying the same. Generation is **0.9250**, the same as structure-aware preparation's **0.9250**: there is no generation difference to measure between the two. The measured difference between these two setups is not in the hit rate but in the **stored vector count**. The last row is where the previous lesson started: 693 chunks, 0.9500 recall, 0.9250 generation, 4848 tokens. The output of structure-aware preparation is the input of the first lesson. #### What Preparation Throws Away The third row looks cheap: it lowers the stored tokens **from 8985 to 7904** and moves no column of the hit rate. The repetition-based boilerplate filter rests on a guess: if the same text appears in many documents, it is leftover. That is true for the form header and the page footer. Because record language is **formulaic**, real paragraphs repeat too. ```python def links(p): return {(x["code"], k) for x in p for k in x["concepts"]} full = links(prepare("structure-aware")[0]) print("concept-document links built by structure-aware preparation:", len(full)) print("threshold chunks dropped lines dropped tokens lost links no-main docs recall generation") for e in (2, 3, 4): p, at = prepare("repeat filter", e) b = links(p) m = measure(p) print("%9d %6d %13d %16d %13d %13d %6.4f %10.4f" % (e, len(p), len(at), sum(len(tokenize(x)) for x in at), len(full - b), sum((d["code"], d["main"]) not in b for d in DOCUMENTS), m[0], m[5])) ``` ``` concept-document links built by structure-aware preparation: 568 threshold chunks dropped lines dropped tokens lost links no-main docs recall generation 2 427 1066 10525 283 60 0.9500 0.9250 3 695 798 7021 96 8 0.9500 0.9250 4 835 658 5282 17 0 0.9500 0.9250 ``` At all three thresholds recall is **0.9500** and generation **0.9250**. The query set sees nothing. Yet at threshold three the filter drops **798 lines** and **7021 tokens**; **96** of the **568** concept-document links the corpus built vanish, and in **eight documents** no evidence of that document's own main concept remains. When the threshold is lowered to two, the loss rises to **283 links** and **sixty documents**, two thirds of the stored tokens are gone, and the two measure columns still do not budge. The real lesson here is about measurement. Forty queries do not sample the twenty concepts of two hundred documents; the deletion of the evidence for a concept nobody asked about is **invisible** in this measurement. The number of concept-document links, on the other hand, is independent of the run and of the query set: it says how much of the corpus can still be searched, without running a single query. This is the number to look at when choosing a cleaning rule. #### Summary - Preparation's counterpart in the retrieval pipeline is two separate questions: **what it adds to recall** and **what it throws away**. The second may be invisible in the retrieval and generation columns. - The unprepared source is the best of the retrieval column (recall 1.0000) and the worst of the generation column (0.0000); all forty requests exceed the context window. - Crude line-based splitting stores 1805 vectors, structure-aware preparation 693; the generation is the same, 0.9250, in both. The measured gain is not in the hit rate but in the stored vector count. - Structure-aware preparation gives 693 chunks, 0.9500 recall and 0.9250 generation; this is where the previous lesson started. - At threshold three the repetition-based boilerplate filter drops 798 lines and 7021 tokens while deleting 96 of 568 concept-document links; at threshold two the loss is 283 links and sixty documents. - The number of concept-document links is a measure independent of the query set, and it is the real criterion for a cleaning decision. #### Next Step Structure-aware preparation accepted one boundary as **given**: the source's own paragraph boundary. The continuation lines were merged and the header and footer dropped, but where a paragraph ends was left as the source's decision. That boundary is a decision, and it changes everything the retrieval pipeline measures. As the chunk shrinks, precision rises and surfaces on the boundary are lost; as it grows, recall rises and the context window overflows. The next lesson takes that boundary in hand: it sweeps chunk size and overlap, prints each setup's recall, stored chunk count and stored tokens side by side, and counts the point where what overlap buys stops paying for its cost. ### Chunking Strategies Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/chunking-strategies Treating the chunk boundary as a decision and sweeping size and overlap on the same fictional corpus: eight-token chunks keep only 0.0765 of the source's 693 relation occurrences whole, eighty-token chunks keep 0.9971 of them but 0.15 of the requests exceed the context window and generation falls from 0.9250 to 0.5500. In a twenty-token chunk, raising overlap from zero to twelve lifts relation recall from 0.6392 to 1.0000 while the stored tokens climb from 8985 to 17206, that is 1.91 times; the first step gains nothing, and the cost per relation gained jumps from 23 to 138 after the eighth token, which is where the crossover point lies. Splitting along the source's own paragraph boundary gives the same 1.0000 relation recall with 693 chunks and 8985 tokens, roughly half the space of the overlap. Retrieval recall and generation do not move enough to cross the threshold in any row of the overlap sweep. The previous lesson carried preparation up to the boundary and stopped there: continuation lines were merged, the header and footer dropped, the code and class field separated, but where a paragraph ends was left as the **source's decision**. In the retrieval pipeline that boundary is not the source's decision; it is the decision of whoever builds the pipeline. A chunk is both the unit that is indexed and the piece of text handed to the model; its size determines both ends at once. What this lesson measures is the price of the boundary. As the chunk shrinks, the stored vector count rises and the **relations** inside the text are cut; as it grows, the context window fills. Overlap takes back the cut relation and pays for what it takes back in tokens. Where these three cross is a number that can be measured. - **RP14.** The code is a **simulator**; the corpus, the embedding and the queries are **fictional**, seed **20260218**. The embedding and the language model are the definitions of the previous lessons. - **RP15.** Chunk boundaries are drawn on **words**, and chunk size is given in **tokens**: the window grows by adding words until the token budget is full. No split is made in the middle of a token, because a split token matches no surface in search. - **RP16.** **Overlap** is also in tokens: the next window starts by re-taking the last overlap-many tokens of the previous one. - **RP17.** A **relation occurrence** is the span in a paragraph that carries two concept surfaces together, and it is the source's real unit of information: which record is kept together with which procedure. The corpus has **693** relation occurrences. - **RP18.** **Relation recall** is the share of these occurrences that **stay whole inside a single chunk**. It is **independent** of the query set and of the run; it says what the chunking decision did to the source without running a single query. - **RP19.** Retrieval recall, generation hit rate and context overflow rate are the definitions of the previous lessons; the top three chunks are retrieved and the threshold is **0.0750**. - **RP20.** The stored tokens are the token total of all chunks. In a split without overlap this number equals the size of the source itself (**8985**); how many times overlap multiplies it is written alongside. #### Taking the Boundary in Hand The first block sets up the corpus, puts a token-budgeted chunker on top of it and marks the relation occurrences. The language model simulator follows, and the measurement function is built. ```python # FICTIONAL. The corpus, concept axis, embedding and language model are a simulator; no real # endpoint is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- chunking: chunk boundaries fall on words, size is measured in tokens BODY = [" ".join(d["paras"]).split() for d in DOCUMENTS] RELATIONS = [(i, 11 * t, 11 * t + 8) # the span carrying both surfaces of a paragraph together for i, d in enumerate(DOCUMENTS) for t in range(len(d["paras"]))] def windows(w, size, overlap): tc = [len(tokenize(x)) for x in w] spans, i = [], 0 while True: j, t = i, 0 while j < len(w) and t + tc[j] <= size: t += tc[j] j += 1 j = max(j, i + 1) spans.append((i, j)) if j >= len(w): return spans new, g = j, 0 while new > i + 1 and g + tc[new - 1] <= overlap: # rewinds by the overlap new -= 1 g += tc[new] i = new def split(size, overlap, structure=False): p, spans = [], {} for i, d in enumerate(DOCUMENTS): w = BODY[i] spans[i] = ([(11 * t, 11 * t + 11) for t in range(len(d["paras"]))] if structure else windows(w, size, overlap)) for a, b in spans[i]: m = " ".join(w[a:b]) p.append({"code": d["code"], "label": d["label"], "text": m, "concepts": [k for k in KEYS if CONCEPTS[k][1] in m]}) kept = sum(any(a <= s and e <= b for a, b in spans[i]) for i, s, e in RELATIONS) return p, kept / len(RELATIONS) import math # ---- FICTIONAL language model simulator: the previous lessons' simulator WINDOW = 240 PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"} COUNTER = [0] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def call(prompt, seed=SEED, temperature=0.6): COUNTER[0] += 1 b = tokenize(prompt) if len(b) > WINDOW: return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) dm = " " + prompt.lower() + " " p = {k: 0.15 for k in KEYS} for k in KEYS: p[k] += 2.0 * dm.count(CONCEPTS[k][1]) if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm: p[k] += 2.0 w = [math.exp(p[a] / temperature) for a in sorted(p)] y = sorted(p)[pick(r(), [x / sum(w) for x in w])] return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y} INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) + " . Reply with one concept . ") def measure(p, k=3): COUNTER[0] = 0 v = [embed(x["text"]) for x in p] a = [0] * 4 for s in QUERIES: q = embed(s["text"]) top = sorted(range(len(p)), key=lambda i: -cosine(q, v[i]))[:k] a[0] += any(s["concept"] in p[i]["concepts"] for i in top) y = call(INSTRUCTION + "Source: " + " . ".join(p[i]["text"] for i in top) + " . Request: " + s["text"]) a[1] += y["response"] == s["concept"] a[2] += y["input"] + y["output"] a[3] += y["status"] == "context_overflow" return (sum(len(tokenize(x["text"])) for x in p), a[0] / 40, a[1] / 40, a[2], a[3] / 40) print(" size chunks stored relation recall generation request tokens overflow") for size in (8, 12, 20, 32, 48, 80): p, rel = split(size, 0) m = measure(p) print("%5d %6d %8d %8.4f %7.4f %10.4f %14d %8.2f" % (size, len(p), m[0], rel, m[1], m[2], m[3], m[4])) ``` ``` size chunks stored relation recall generation request tokens overflow 8 1217 8985 0.0765 0.9500 0.9250 4164 0.00 12 838 8985 0.5440 0.9500 0.9250 4576 0.00 20 555 8985 0.6392 0.9500 0.9250 4913 0.00 32 369 8985 0.8846 1.0000 0.9000 5582 0.00 48 276 8985 0.9351 1.0000 0.8500 5860 0.00 80 203 8985 0.9971 1.0000 0.5500 8271 0.15 ``` The size sweep shows two separate degradations at two separate ends. Eight-token chunks keep **almost none of the source's 693 relation occurrences** whole (**0.0765**), because a relation takes about twelve tokens and rarely fits in the window. These chunks do not lower retrieval recall, since they preserve the concept surfaces: **0.9500**. What is cut is not the concept but the tie between concepts. The other end breaks visibly. Eighty-token chunks keep **0.9971** of the relations and retrieval recall rises to **1.0000**, but when three chunks are put into the prompt, **0.15** of the forty requests exceed the context window. Generation falls **from 0.9250 to 0.5500**. The drop is far larger than the threshold of 0.0750: it is **a measured degradation**. The row where the retrieval column is best is the row where the generation column is worst. At forty-eight tokens generation is 0.8500, and its gap to the 0.9250 of the smaller chunks is no larger than the threshold: whether this row lowers the hit rate **cannot be known** from this measurement. The usable band is in the middle: between twenty and thirty-two tokens there is no context overflow, retrieval recall is at its highest level and relation recall runs between 0.6392 and 0.8846. The way to lift this band is not to enlarge the size; it is overlap. #### The Price of Overlap The second block sweeps overlap from zero to twelve in the twenty-token chunk and writes, at each step, how many relations were gained and how many tokens were stored in return. The last row uses the source's own paragraph boundary instead of overlap. ```python print("overlap chunks stored fold relation gain extra tokens per relation recall generation") SOURCE_TOKENS = sum(len(tokenize(x["text"])) for x in split(0, 0, structure=True)[0]) prev = None for overlap in (0, 2, 4, 6, 8, 10, 12): p, rel = split(20, overlap) m = measure(p) extra = "" if prev: k = round((rel - prev[0]) * len(RELATIONS)) extra = "%7d %12d %12d" % (k, m[0] - prev[1], (m[0] - prev[1]) // k) print("%7d %6d %8d %5.2f %8.4f%s %7.4f %10.4f" % (overlap, len(p), m[0], m[0] / SOURCE_TOKENS, rel, extra or " " * 33, m[1], m[2])) prev = (rel, m[0]) p, rel = split(0, 0, structure=True) m = measure(p) print("structure-aware boundary: chunks %d, stored %d, relation %.4f, recall %.4f, generation %.4f" % (len(p), m[0], rel, m[1], m[2])) ``` ``` overlap chunks stored fold relation gain extra tokens per relation recall generation 0 555 8985 1.00 0.6392 0.9500 0.9250 2 587 9697 1.08 0.6335 -4 712 -178 0.9500 0.9250 4 605 10517 1.17 0.7157 57 820 14 0.9500 0.9250 6 640 11538 1.28 0.8730 109 1021 9 0.9500 0.9250 8 718 13025 1.45 0.9639 63 1487 23 0.9500 0.9250 10 790 14825 1.65 0.9827 13 1800 138 0.9500 0.9250 12 897 17206 1.91 1.0000 12 2381 198 0.9500 0.9250 structure-aware boundary: chunks 693, stored 8985, relation 1.0000, recall 0.9500, generation 0.9250 ``` Relation recall rises with overlap, **from 0.6392 to 1.0000**, though not at every step: the first step, from zero to two, gains nothing. Relation recall slips to 0.6335 (the gain column reads -4) while 712 tokens are added, because shifting the windows moves boundaries onto relations that were whole. The cost does not rise in a steady way; it **multiplies**: the stored tokens go from 8985 to **17206** (1.91 times), the stored chunks from 555 to 897. Because every chunk holds one vector, the stored vector count grows by a comparable ratio. The crossover point is read in the last column. The three steps after the first are cheap: for every relation gained, **14, 9 and 23** tokens are stored respectively. At the fifth step, as overlap goes from eight to ten, the cost per relation jumps to **138 tokens**. It does not recover afterward either: **198**. Overlap up to eight rescues most of the relations that were cut (relation recall 0.9639); the rest are the few occurrences that happen to fall on a window boundary in the most unlucky way, and rescuing them requires storing the whole corpus again. **Overlap is eight tokens**; on this workload, what comes after that buys not relations but copies. Looking at the retrieval recall and generation columns, the whole sweep is flat: recall is 0.9500 in every row, and generation is **0.9250** in all seven rows. What overlap **adds to the hit rate has not been measured** on this query set; what it adds to the stored tokens has been measured. A mechanism whose cost is not written down counts as unmeasured, and the only thing measured in these rows is the cost. The last row closes the argument. Splitting along the source's own paragraph boundary makes relation recall **1.0000**: the place overlap twelve reaches. In return it stores **693 chunks** and **8985 tokens**, that is, **roughly half as much** as the setup with overlap. Overlap is the insurance paid when the structure of the source is unknown; when the structure is known, its price is wasted. #### Summary - The chunk boundary is the decision of whoever builds the pipeline, not of the source; size determines both the stored vector count and the text that enters the prompt. - Relation recall measures, without running a query, how many of the source's units of information stayed whole in a chunk, and it is the real criterion of the chunking decision. - A small chunk cuts the relation but keeps the concept surface: at eight tokens relation recall is 0.0765 while retrieval recall is 0.9500. - A large chunk keeps the relation but fills the window: at eighty tokens 0.15 of the requests overflow the context and generation falls from 0.9250 to 0.5500. - Overlap lifts relation recall from 0.6392 to 1.0000, takes the stored tokens to 1.91 times, and the cost per relation jumps from 23 to 138 after eight: the crossover point is overlap eight. - Structure-aware splitting gives the same 1.0000 relation recall with roughly half the space of the overlap. #### Next Step In three lessons the corpus was taken as **static**. The documents were prepared once, chunked once, embedded once, and the index was never touched again. Field records do not behave that way: a procedure is updated, a threshold changes, a record is withdrawn. The index does not learn of it at that moment but at the next refresh, and in the time that passes in between it keeps bringing the model **the right chunk**, but the information inside the chunk is no longer valid. The next lesson sweeps the refresh interval: it measures how the stale chunk rate, the wrong answer rate and the number of chunks re-embedded in a refresh change as the interval grows, and shows that the retrieval column shows none of this degradation. ### Indexing and Updating Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/indexing-and-updating Measuring that the index is a snapshot and goes stale as the source changes: in a fictional twelve-period change stream, when the refresh interval goes from one to twelve the stale chunk rate rises from 0.0000 to 0.2562 and the wrong answer rate from 0.0750 to 0.3875. The retrieval column shows none of this degradation: retrieval recall is 0.9500 and concept hit rate 0.9250 at all six intervals; the only column that falls is the share of answers whose copied validity record is up to date. The cost of refreshing reads differently under the two policies: full reindexing embeds 8316 chunks at interval one, incremental refresh embeds 881. Multiplying the interval by twelve lowers the re-embedded chunks per up-to-date answer under incremental refresh only from 1.98 to 1.77, a gain of eleven percent; the price of the same decision is 0.3125 in the up-to-date answer rate. The difference between the two runs is 0.0000 in every row. The previous three lessons took the corpus as **static**. The documents were prepared once, chunked once, embedded once; after the index was built it was never touched again. Field records do not behave that way. A procedure is republished, a threshold is updated, a record is withdrawn. When the source changes, the index does not; the index is a **snapshot**, and unless it is written down when it was taken, what it shows is not known. The degradation this lesson measures is of a different kind from the earlier ones. Retrieval does not bring the wrong chunk: the retrieved chunk is still from the right document and the right concept. What degrades is the chunk's **content**. That is why the retrieval column stays silent and the fault shows only in the generation column. - **RP21.** The code is a **simulator**; the corpus, the change stream and the queries are **fictional**, seed **20260218**. The embedding and the language model are the definitions of the previous lessons. - **RP22.** Each document carries a **validity record** (the string `validity record` in the data). When the document is republished the record goes up by one; when a chunk is served from the index it is given together with its record. The change stream is twelve periods, and in each period **0.10** of the documents are republished. - **RP23.** A republication changes the chunk's record, not its **body**. This is an assumption that simplifies the measurement: because the embedding does not move, the retrieval ranking stays the same at all intervals, and it is certain that the column that falls falls **only** from staleness. - **RP24.** One behavior is added to the simulator: the model **copies as is the validity record** in the first source chunk that carries the surface of the concept it chose. This is what a quoting model does, and it is the place where staleness passes into the answer. - **RP25.** The **stale chunk rate** is the rate at which the records of the served chunks disagree with the source's actual record in that period. The **up-to-date answer rate** is the rate of answers whose concept is correct **and** whose copied record equals the source's current record. - **RP26.** Two refresh policies are measured: **full reindexing** re-embeds all chunks at every refresh, **incremental refresh** only the chunks of documents that changed in that interval. A document that changed twice in one interval is embedded once. - **RP27.** The measurement is taken in **all** twelve periods; forty queries are asked in each period, that is, 480 requests for each interval. The stale chunk rate and the number of re-embedded chunks are independent of the run. #### Where Staleness Passes into the Answer The first block sets up the corpus, puts the change stream and the validity record on top of it, defines the simulator together with the copying behavior and sweeps the refresh interval. ```python # FICTIONAL. The corpus, concept axis, embedding, change stream and language model are a simulator; # no real endpoint is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) import math # ---- FICTIONAL change stream: twelve periods, a fraction of the documents is republished each period PERIODS = 12 CHANGES = [] for t in range(1, PERIODS + 1): r = rng(SEED + 6007 * t) CHANGES.append([i for i in range(len(DOCUMENTS)) if r() < 0.10]) def stamp(i, t): """The validity record of document i at the end of period t.""" return "Y%02d" % sum(i in CHANGES[u] for u in range(t)) CHUNKS = [{"doc": i, "code": d["code"], "label": d["label"], "body": p, "concepts": [k for k in KEYS if CONCEPTS[k][1] in p]} for i, d in enumerate(DOCUMENTS) for p in d["paras"]] VEC = [embed(x["body"]) for x in CHUNKS] # the stamp does not change the embedding TOP = [] for s in QUERIES: q = embed(s["text"]) TOP.append(sorted(range(len(CHUNKS)), key=lambda i: -cosine(q, VEC[i]))[:3]) # ---- FICTIONAL language model simulator: the previous lessons' simulator, one addition # copying the validity record it sees in the context WINDOW = 240 PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"} COUNTER = [0] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def call(prompt, seed=SEED, temperature=0.6): COUNTER[0] += 1 b = tokenize(prompt) if len(b) > WINDOW: return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) dm = " " + prompt.lower() + " " p = {k: 0.15 for k in KEYS} for k in KEYS: p[k] += 2.0 * dm.count(CONCEPTS[k][1]) if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm: p[k] += 2.0 w = [math.exp(p[a] / temperature) for a in sorted(p)] y = sorted(p)[pick(r(), [x / sum(w) for x in w])] for x in prompt.split(" . "): # copies the record in the source as is if CONCEPTS[y][1] in x and "validity record" in x: y += " " + x.split("validity record ")[1].split()[0] break return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y} INSTRUCTION = ("Link the request to one of these concepts and write down the validity entry of the source: " + " , ".join(KEYS) + " . Reply with one concept and the entry . ") def run(interval, seed=SEED): COUNTER[0] = 0 stale = concept = current = recall = n = 0 embedded = full = 0 for t in range(1, PERIODS + 1): last = (t // interval) * interval # the last period the index saw if t % interval == 0: # a refresh occurs this period changed = {i for u in range(t - interval, t) for i in CHANGES[u]} embedded += sum(x["doc"] in changed for x in CHUNKS) full += len(CHUNKS) for s, top in zip(QUERIES, TOP): texts = {i: "%s validity record %s" % (CHUNKS[i]["body"], stamp(CHUNKS[i]["doc"], last)) for i in top} stale += sum(stamp(CHUNKS[i]["doc"], last) != stamp(CHUNKS[i]["doc"], t) for i in top) / 3 recall += any(s["concept"] in CHUNKS[i]["concepts"] for i in top) y = call(INSTRUCTION + "Source: " + " . ".join(texts[i] for i in top) + " . Request: " + s["text"], seed) kv, _, rv = y["response"].partition(" ") rec = next((i for i in top if CONCEPTS[kv][1] in texts[i]), None) if kv in KEYS \ else None concept += kv == s["concept"] current += (kv == s["concept"] and rec is not None and rv == stamp(CHUNKS[rec]["doc"], t)) n += 1 return stale / n, recall / n, concept / n, current / n, embedded, full print("periods %d, total republications %d, documents never changed %d" % (PERIODS, sum(len(x) for x in CHANGES), sum(1 for i in range(len(DOCUMENTS)) if all(i not in d for d in CHANGES)))) print("interval refreshes embedded(incremental) embedded(full) stale chunks recall concept current") for a in (1, 2, 3, 4, 6, 12): st, rc, cn, cu, em, fl = run(a) print("%8d %9d %21d %14d %12.4f %7.4f %8.4f %8.4f" % (a, PERIODS // a, em, fl, st, rc, cn, cu)) ``` ``` periods 12, total republications 254, documents never changed 51 interval refreshes embedded(incremental) embedded(full) stale chunks recall concept current 1 12 881 8316 0.0000 0.9500 0.9250 0.9250 2 6 839 4158 0.0347 0.9500 0.9250 0.8917 3 4 774 2772 0.0625 0.9500 0.9250 0.8771 4 3 741 2079 0.0937 0.9500 0.9250 0.8208 6 2 662 1386 0.1278 0.9500 0.9250 0.7708 12 1 519 693 0.2562 0.9500 0.9250 0.6125 ``` The first reading of the table is the clearest reason for keeping the two columns apart. **Retrieval recall is 0.9500 in all six rows, and the concept hit rate is 0.9250 in all six rows.** As the refresh interval goes from one period to twelve, **nothing** changes in the retrieval column, because the index keeps bringing the right documents. **0.2562** of the chunks it brings now show a text that is no longer in the source, and none of the retrieval measures says so. The only column that falls is the up-to-date answer rate: **from 0.9250 to 0.6125**. The 0.3125 between them is more than four times the threshold. The user-visible counterpart of this is the following: the model finds the right procedure, names the right concept and writes beside it a record that has been **withdrawn**. This answer is not a hallucination; it is faithful to the source. The source itself is stale. Grounding in the source does not guarantee the freshness of the source. The stale chunk rate and the wrong answer rate do not coincide one to one either. At interval twelve **0.2562** of the chunks are stale but **0.3875** of the answers are wrong. The difference comes from the concept hit rate already being 0.9250, and from the probability that a stale chunk is the chunk carrying the chosen concept carrying a distribution of its own. Staleness is an **input** measure, the wrong answer is an **output** measure; they are not the same number and cannot be written in place of each other. #### The Price of Refreshing The embedding cost behaves entirely differently under the two policies. **Full reindexing** re-embeds six hundred ninety-three chunks at every refresh: **8316** at interval one, **693** at interval twelve. Under this policy the interval is a lever that divides the cost directly. **Incremental refresh** embeds only the chunks of changed documents: **881** at interval one, that is, **9.4 times** smaller than full reindexing. The second block reads these two policies per correct piece of work and tests the stability of the measurement itself. ```python print("interval wrong answer rate incremental/current full/current 2nd seed current difference") for a in (1, 2, 4, 12): st, rc, cn, cu, em, fl = run(a) c2 = run(a, SEED + 1)[3] n = cu * PERIODS * len(QUERIES) print("%8d %17.4f %19.2f %12.2f %16.4f %10.4f" % (a, 1 - cu, em / n, fl / n, c2, abs(cu - c2))) ``` ``` interval wrong answer rate incremental/current full/current 2nd seed current difference 1 0.0750 1.98 18.73 0.9250 0.0000 2 0.1083 1.96 9.71 0.8917 0.0000 4 0.1792 1.88 5.28 0.8208 0.0000 12 0.3875 1.77 2.36 0.6125 0.0000 ``` The incremental column shows why the interval is the wrong lever. When the interval goes from one to twelve, the embedded chunks fall from 881 to 519, that is, only **forty-one percent**. The reason is that a document republished several times in one interval is embedded only once; as the interval lengthens this merging increases, but the total number of changed documents does not change. Read per up-to-date answer, the gain melts further: **from 1.98 to 1.77**, that is, eleven percent. The price of the same decision is **0.3125** in the up-to-date answer rate. Lengthening the interval under incremental refresh is selling a quarter of the accuracy for next to nothing. In the full reindexing column the table looks the other way round: per up-to-date answer, **from 18.73 to 2.36**. This row does not justify lengthening the interval; it justifies giving up full reindexing. Setting interval one with the incremental policy is both cheaper than setting interval twelve with full reindexing (881 against 693, the same order of magnitude) and 0.3125 more accurate. The last two columns are the stability of the measurement itself. Run with the second seed, the up-to-date answer rate comes out **exactly the same** in all four rows, a difference of **0.0000**. This is the continuation of the first lesson's finding: the output of a call whose source has been given does not depend on the seed. For that reason all the differences in the table are far above the threshold and have been **measured**. #### Summary - The index is a snapshot; when the source changes the index does not, and in the time that passes it serves the right chunk with stale content. - As the refresh interval goes from one to twelve, the stale chunk rate rises from 0.0000 to 0.2562 and the wrong answer rate from 0.0750 to 0.3875. - The retrieval column shows none of this degradation: recall is 0.9500 at all six intervals and the concept hit rate 0.9250. Grounding in the source does not guarantee the freshness of the source. - The stale chunk rate is an input measure, the wrong answer rate an output measure; 0.2562 and 0.3875 are not two names for the same thing. - Full reindexing embeds 8316 chunks at interval one, incremental refresh 881; lengthening the interval reduces the embedded chunks under the incremental policy by only forty-one percent. - The chunks embedded per up-to-date answer fall from 1.98 to 1.77 under the incremental policy: for an eleven percent gain, 0.3125 of accuracy is sold. #### Next Step In four lessons the pipeline was built from the input side: the source was prepared, chunked, embedded, the index refreshed. All four of these decisions were on the **document's** side. The text at the other end of the pipeline, the query the subscriber wrote, was used as it was in all four lessons: short, incomplete, in a language other than that of the record. Yet it is one of the two texts that determine what retrieval finds. Expanding the query, rewriting it, or translating it into the language of the record with a model call can raise recall, and that call itself is added to the cost of the pipeline. The next lesson takes up the query and measures the recall that the transformation adds together with the call it adds. ### Query Transformation Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/query-transformation Translating the query the user typed into the document language with a single model call before it reaches the index, and counting what this step gains and what it loses separately: on a fictional set of forty queries the short phrasing has a baseline of 0.3250 recall and 0.1500 generation, while expansion raises the two to 0.7500 and 0.7000, which is where a query written in the index language leaves off. The same setup drops the long phrasing from a baseline of 0.5500 recall to 0.5000 with expansion and to 0.2750 with rewriting, because the share with which the transformer finds the correct concept falls from 0.9500 to 0.3500. The cost on both sets is one call per query and more tokens: from 4609 to 5804 on the short set and from 5827 to 8553 on the long set; the run-to-run difference was measured at 0.0500 in recall and 0.0500 in generation. The previous four lessons built the document side of the pipeline: the corpus was prepared, split into chunks, indexed and refreshed. All four stayed silent about one assumption: **the query goes to the index in the form the user typed it**. The subscriber writes "which band", and the corpus says "tier limit". One way to close the gap is to give the query to a model before sending it to the index and have it **translated into the document language**; the cost is one call per query. The transformation is **one step**; seeing the result and querying again is the subject of the AI Agents course. - **RP41.** The code is a **simulator**; no real embedding or model endpoint is called. The corpus, the concepts, the queries and the user phrasings are **fictional**, seed **20260218**. - **RP42. The target has two parts.** The target of a query is both a **concept** and a **record type**; a chunk is relevant only if it carries both. **Recall@k** is the share of queries with a relevant chunk in the top k, and **precision@k** is the share of the top k that is relevant. - **RP43.** The pipeline uses the **hybrid search** measured in `embeddings/06`: token overlap is added next to cosine similarity. Its procedure is not repeated, and the weight is fixed. The baseline of this lesson is **retrieval without transformation**; the baseline of the no-retrieval call was set up in `retrieval-pipeline/01`. - **RP44.** The generation simulator picks one of the chunks in the context and cites its code; the chunk's **position** changes the weight. That weight is the subject of `retrieval-pipeline/07` and is fixed here. - **RP45.** Two phrasings of the same forty queries are measured: the **short phrasing** is the abbreviated form the subscriber types, and the **long phrasing** is the same question extended with context sentences. The target is the same in both. - **RP46.** If the transformation hits the rate limit, the pipeline **falls back to the raw query**; the tokens of the failed call are still counted. #### The Pipeline and the Target The pipeline works at chunk level. The block below sets up the corpus, the embedding simulator, hybrid retrieval and the generation simulator; then it prints where the pipeline stands when the query is written **in the form the index recognizes**: the upper bound that transformation can reach. ```python # SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus, # concepts and queries are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def text_of(d): return d["title"] + " " + " ".join(d["paras"]) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- pipeline: chunks, hybrid retrieval, target definition and generation CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]] for p in CHUNKS: p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"])) for s in QUERIES: s["record"] = s["text"].split()[-1] def retrieve(q, k=5, w=0.5): qv, qb = embed(q), set(tokenize(q)) s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i) for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1])) return [CHUNKS[i] for _, i in s[:k]] def relevant(s, p): return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h WINDOW = 240 COUNTER = [0] ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . " def answer(query, chunks, seed=SEED, temperature=0.12): """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight.""" COUNTER[0] += 1 prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None} r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks) score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb)) for i, x in enumerate(chunks)] w = [math.exp(a / temperature) for a in score] chosen = chunks[pick(r(), [a / sum(w) for a in w])] return {"status": "ok", "input": len(toks), "chunk": chosen, "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))} rec = gen = 0 for s in QUERIES: g = retrieve(s["text"], 5) rec += any(relevant(s, p) for p in g) gen += relevant(s, answer(s["text"], g)["chunk"]) print("corpus %d documents %d chunks ; average chunk %.1f tokens ; example query %r" % (len(DOCUMENTS), len(CHUNKS), sum(len(p["bag"]) for p in CHUNKS) / len(CHUNKS), QUERIES[0]["text"])) print("target chunk set average %.1f chunks" % (sum(len([p for p in CHUNKS if relevant(s, p)]) for s in QUERIES) / 40)) print("query written in the index language k=5 recall %.4f generation %.4f" % (rec / 40, gen / 40)) ``` ``` corpus 200 documents 693 chunks ; average chunk 12.0 tokens ; example query 'how it gets written down form' target chunk set average 8.9 chunks query written in the index language k=5 recall 0.8000 generation 0.6750 ``` The target set averages **8.9** chunks, a little above one percent of six hundred ninety-three. When the query is written in the index language, the top five chunks bring one from this set with a share of **0.8000**, and generation turns **0.6750** of them into an answer. #### Two Phrasings and the Transformer The subscriber does not write these sentences. "how it gets written down form" is an index query; the subscriber writes "is it written form" or describes the problem at length. The block below sets up the two phrasings of the same forty targets and runs three setups on both: the **baseline** (the raw query), **expansion** (a term in the document language is appended to it) and **rewriting** (the raw query is discarded and a sentence in the document language takes its place). ```python # FICTIONAL: the short form the user types -- it does not carry the surface the index recognizes SHORT = {"tier": "which band", "unit": "costs per ton", "account": "home or office", "switch": "moving plans", "faultcode": "error code", "leak": "might be escaping", "pressure": "flow weak", "crew": "technician never came", "index": "dial digits", "read": "is it written", "swap": "hardware changed", "verify": "typed wrong", "planned": "announced ahead", "notice": "where to look", "duration": "hours it will last", "restore": "will it show up again", "periodic": "regular upkeep", "valve": "tap shutoff", "filter": "cleaning filter", "calib": "accuracy calibration"} PREFIX = ["previous week", "for a few days", "since this morning", "yesterday evening"] CONNECTOR = ["one more thing i wanted to ask", "also i wrote to the relevant department", "by the way i was curious", "i had also called before"] SHORT_QUERIES, LONG_QUERIES = [], [] for i, s in enumerate(QUERIES): r = rng(SEED + 31337 * i) y, o2 = KEYS[int(r() * 20)], COMMON[int(r() * 7)] z = KEYS[int(r() * 20)] SHORT_QUERIES.append(dict(s, text=f"{SHORT[s['concept']]} {s['record']}")) LONG_QUERIES.append(dict(s, text=f"{PREFIX[int(r() * 4)]} {CONCEPTS[y][2]} {o2} " f"{CONNECTOR[int(r() * 4)]} {s['text']} {CONCEPTS[z][2]}")) QUERY_TRACE = {} # query language -> concept for k in KEYS: for t in tokenize(CONCEPTS[k][2]): if len(t) > 3: QUERY_TRACE.setdefault(t, k) TRANSFORM_INSTRUCTION = "Translate the query into the document language . Query: " def transform(q, mode, seed=SEED, temperature=0.5): """FICTIONAL transformer: a one-step model call, no loop and no planning.""" COUNTER[0] += 1 toks = tokenize(TRANSFORM_INSTRUCTION + q) if COUNTER[0] % 37 == 0: # rate limit: the raw query return {"status": "rate_limit", "input": len(toks), "output": 0, "text": q, "concept": None} r, score = rng(seed ^ trace_id(q)), {} for t in toks: if t in QUERY_TRACE: score[QUERY_TRACE[t]] = score.get(QUERY_TRACE[t], 0.0) + 1.0 if not score: return {"status": "empty", "input": len(toks), "output": 0, "text": q, "concept": None} keys = sorted(score) w = [math.exp(score[x] / temperature) for x in keys] key = keys[pick(r(), [x / sum(w) for x in w])] common = next((t for t in toks if t in COMMON), "") out = f"{CONCEPTS[key][1]} for {common} goes" if mode == "rewriting" else f"{q} {CONCEPTS[key][1]}" return {"status": "ok", "input": len(toks), "output": len(tokenize(out)), "text": out, "concept": key} def measure(qset, mode, seed=SEED, k=5): COUNTER[0] = 0 rec = gen = calls = toks = failed = right = 0 for s in qset: q = s["text"] if mode != "baseline": d = transform(q, mode, seed) q, calls, toks = d["text"], calls + 1, toks + d["input"] + d["output"] failed, right = failed + (d["status"] != "ok"), right + (d["concept"] == s["concept"]) g = retrieve(q, k) rec += any(relevant(s, p) for p in g) y = answer(s["text"], g, seed) calls, toks = calls + 1, toks + y["input"] + y["output"] gen += y["chunk"] is not None and relevant(s, y["chunk"]) return rec / 40, gen / 40, calls, toks, failed, right / 40 print("average query: short %.1f long %.1f tokens" % (sum(len(tokenize(s["text"])) for s in SHORT_QUERIES) / 40, sum(len(tokenize(s["text"])) for s in LONG_QUERIES) / 40)) print("query setup recall@5 generation calls tokens correct concept failed") gap = [0.0, 0.0] for name, qset in (("short", SHORT_QUERIES), ("long ", LONG_QUERIES)): for mode in ("baseline", "expansion", "rewriting"): a, u, c, b, failed, right = measure(qset, mode) a2, u2 = measure(qset, mode, SEED + 1)[:2] gap = [max(gap[0], abs(a - a2)), max(gap[1], abs(u - u2))] print("%s %-14s %8.4f %10.4f %5d %8d %15s %6d" % (name, mode, a, u, c, b, "-" if mode == "baseline" else "%.4f" % right, failed)) print("run-to-run difference (seed 20260219): recall at most %.4f generation at most %.4f" % (gap[0], gap[1])) ``` ``` average query: short 4.5 long 26.5 tokens query setup recall@5 generation calls tokens correct concept failed short baseline 0.3250 0.1500 40 4609 - 0 short expansion 0.7500 0.7000 80 5804 0.9500 1 short rewriting 0.7500 0.7000 80 5749 0.9500 1 long baseline 0.5500 0.2250 40 5827 - 0 long expansion 0.5000 0.1250 80 8553 0.3500 1 long rewriting 0.2750 0.1000 80 7647 0.3500 1 run-to-run difference (seed 20260219): recall at most 0.0500 generation at most 0.0500 ``` #### The Gain on the Short Phrasing The baseline of the short phrasing is **0.3250** recall and **0.1500** generation. The gap to the upper bound is not a defect of the pipeline but **the query not touching the index**: "which band" appears in no chunk, and all that is left is the overlap of the record type. Transformation closes the gap. Expansion raises recall to **0.7500** and generation to **0.7000**; both stand above the run-to-run difference (0.0500 and 0.0500), and generation reaches the upper bound within that difference. The reason is written in the table: the transformer finds the correct concept with a share of **0.9500**. That expansion and rewriting give the **same** result here is not a coincidence: keeping the raw query has neither a cost nor a gain, because there is no signal to be thrown away. #### The Loss on the Long Phrasing The baseline of the long phrasing is **0.5500**, higher than that of the short phrasing: while lengthening the problem, the subscriber also writes the phrase the index recognizes. Transformation **lowers** the baseline, to **0.5000** with expansion and to **0.2750** with rewriting. Rewriting stands far above the run-to-run difference, so it is a measured loss. Expansion is a smaller story: its recall (0.5000) sits at the edge of the run-to-run difference, and only its generation, which goes from 0.2250 to 0.1250, clearly leaves it behind. The cause is again in the correct concept column: the hit rate falls to **0.3500**. The long query has three concept surfaces and two record types; a one-step call has to choose which one is the real question, and it is wrong most of the time. When it is wrong, **expansion** appends a wrong term but the raw query stays in place; **rewriting** deletes the one thing that was right, the phrase the subscriber wrote, and generation falls to **0.1000**. This is the instance, in this lesson, of the course's second claim: the added machinery made the job worse. #### The Bill for Transformation **Calls** go from one to two per query, from forty to eighty. **Tokens** go from 4609 to 5804 on the short phrasing with expansion and from 5827 to 8553 on the long one; having the model read the long query as well is the most expensive line of the pipeline, and one of the forty calls hits the rate limit. Latency is counted as a **class**: the pipeline turns from a single-call step into a **two-stage chain**. While the extra tokens on the short phrasing raise recall from 0.3250 to 0.7500, the larger extra tokens on the long phrasing lower it from 0.5500 to 0.5000 with expansion and to 0.2750 with rewriting. #### Summary - The target of a query is both a concept and a record type; the share with which a chunk carrying both is found in the top five is retrieval, and the share with which that chunk turns into the answer is the generation measure. - Because the short form the user typed does not touch the index, the baseline is 0.3250 / 0.1500; the same query in the index language gives 0.8000 / 0.6750. - Expansion raises recall to 0.7500 and generation to 0.7000 on the short phrasing; the share of finding the correct concept is 0.9500. - On the long phrasing the same machinery drops the baseline recall from 0.5500 to 0.5000, and rewriting to 0.2750; the correct concept share falls to 0.3500, and rewriting also deletes the phrase the subscriber wrote. - The cost is one call per query and more tokens (5804 and 8553 against 4609 and 5827); the pipeline turns into a chain. #### Next Step The measurements of this lesson count the top five chunks **together**: if a relevant chunk is among them, recall is written, and the order is not asked. Yet precision@5 says most of the top five are irrelevant. The next lesson splits retrieval in two: a cheap stage that gathers a wide candidate set and an expensive stage that reads every candidate together with the query. ### Reranking Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/reranking Splitting retrieval into two stages and counting side by side what the second stage gains and what it makes you pay: in the single-stage pipeline the precision of the top three stays at 0.4167, because a single similarity score cannot search for the concept and the record type of the query at the same time. The second stage, which reads the candidates together with the query, raises precision to 0.8250 with twelve candidates and to 0.9000 with twenty-four, and takes generation from 0.7000 to 1.0000. With forty-eight candidates precision rises by only 0.0083, that is, it stays below the run-to-run difference of 0.0250; in return the recall of the top three falls from 1.0000 to 0.9500, the call count goes from forty to one thousand nine hundred sixty, and tokens go from 3654 to 71797. The previous lesson corrected the query before sending it to the index and counted how good the top five chunks are **together**. What remained is the distribution inside those five: why the **head** of the retrieved set is not full, and what it costs to fix the head. Half of the answer lies in the score itself. The first stage compresses the query into a vector **without seeing the chunk**; this vector can say "search for this concept", but it cannot say "search for this concept **and** this record type together", because a single similarity number does not carry the intersection of two conditions. The **second stage** reads the query and the chunk **together** and can search for the two conditions separately. The cost is plain: one call for every candidate. - **RP47.** The code is a **simulator**; no real embedding, ranker or model endpoint is called. The corpus, the concepts and the queries are **fictional**, seed **20260218**. - **RP48.** The definition of the target continues from `retrieval-pipeline/05`: the target of a query is both a **concept** and a **record type**, and a chunk is relevant only if it carries both. The pipeline is the same too. - **RP49.** This lesson **closes the transformation step** and takes the query set in the form in which it was defined; the only setup measured is reranking. The cost of transformation was measured in `05`. - **RP50.** The reranker is **one model call per candidate** and sees the query and the chunk in a single prompt. Its score is the sum of two conditions: concept match and record type match. - **RP51.** The reranker is also a model, and its score is **noisy**. The amplitude of the noise exceeds the weight of one condition, so it cannot always see a difference of a single condition. - **RP52.** The number of chunks that enter the context is **fixed at three**; k itself and the context limit are the subject of `retrieval-pipeline/07`. - **RP53.** Latency is written as a **class**. Because the candidates are scored independently of one another they can be called in parallel, but the order of the stages is mandatory: the second cannot start before the first finishes. #### The Limit of a Single Stage The block below sets up the pipeline and prints what recall and precision do as k grows in single-stage retrieval. The two columns go in **opposite directions**; the tension between them is the reason for this lesson. ```python # SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus, # concepts and queries are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def text_of(d): return d["title"] + " " + " ".join(d["paras"]) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- pipeline: chunks, hybrid retrieval, target definition and generation CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]] for p in CHUNKS: p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"])) for s in QUERIES: s["record"] = s["text"].split()[-1] def retrieve(q, k=5, w=0.5): qv, qb = embed(q), set(tokenize(q)) s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i) for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1])) return [CHUNKS[i] for _, i in s[:k]] def relevant(s, p): return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h WINDOW = 240 COUNTER = [0] ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . " def answer(query, chunks, seed=SEED, temperature=0.12): """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight.""" COUNTER[0] += 1 prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None} r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks) score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb)) for i, x in enumerate(chunks)] w = [math.exp(a / temperature) for a in score] chosen = chunks[pick(r(), [a / sum(w) for a in w])] return {"status": "ok", "input": len(toks), "chunk": chosen, "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))} print("corpus %d documents %d chunks ; target chunk set average %.1f chunks" % (len(DOCUMENTS), len(CHUNKS), sum(len([p for p in CHUNKS if relevant(s, p)]) for s in QUERIES) / 40)) print("single-stage retrieval: k recall@k precision@k") for k in (1, 3, 5, 10, 20, 40): a = sum(any(relevant(s, p) for p in retrieve(s["text"], k)) for s in QUERIES) / 40 e = sum(sum(relevant(s, p) for p in retrieve(s["text"], k)) / k for s in QUERIES) / 40 print(" %2d %8.4f %.4f" % (k, a, e)) ``` ``` corpus 200 documents 693 chunks ; target chunk set average 8.9 chunks single-stage retrieval: k recall@k precision@k 1 0.7750 0.7750 3 0.7750 0.4167 5 0.8000 0.2700 10 0.9750 0.2425 20 1.0000 0.2425 40 1.0000 0.1831 ``` Recall needs **twenty** chunks to catch all forty queries. Precision says the opposite: when k is twenty, only **0.2425** of the top twenty is relevant, that is, about three wrong chunks enter the context next to every correct chunk. At k of three precision is **0.4167**, but recall stays at **0.7750** and a quarter of the queries never see a correct chunk. There is no way to improve these two columns at the same time in a single-stage pipeline, because the order comes out of a single score. Reranking suggests this: **take recall from a wide candidate set, take precision from a second scorer.** #### The Second Stage The block below defines the ranker and runs the same forty queries five times, raising the candidate count from three to forty-eight. The number of chunks that enter the context is always three; the only thing that changes is **how many candidates** those three are chosen from. ```python RERANK_INSTRUCTION = "Is the query about the same record as the chunk . Chunk: " TRACE = {} # the tokens of both languages are tied to the same concept for k in KEYS: for m in (CONCEPTS[k][1], CONCEPTS[k][2]): for t in tokenize(m): if len(t) > 3: TRACE.setdefault(t, k) def score_pair(q, p, seed=SEED, noise=1.4): """FICTIONAL second stage: it sees the query and the chunk TOGETHER and searches for the two conditions separately.""" COUNTER[0] += 1 toks = tokenize(RERANK_INSTRUCTION + p["text"] + " Query: " + q) r, qb = rng(seed ^ trace_id(q + " | " + p["text"])), set(tokenize(q)) concept_match = bool({TRACE[t] for t in qb if t in TRACE} & {TRACE[t] for t in p["bag"] if t in TRACE}) record_match = bool(qb & p["bag"] & set(COMMON)) return 1.0 * concept_match + 1.0 * record_match + noise * (r() - 0.5), len(toks) + 1 def measure(n, seed=SEED, k=3): COUNTER[0] = 0 rec = prec = gen = calls = toks = 0 for s in QUERIES: candidates = retrieve(s["text"], max(n, k)) if n > k: scores = [] for p in candidates: v, b = score_pair(s["text"], p, seed) scores.append(v) calls, toks = calls + 1, toks + b candidates = [p for _, p in sorted(zip(scores, candidates), key=lambda t: -t[0])] top = candidates[:k] rec += any(relevant(s, p) for p in top) prec += sum(relevant(s, p) for p in top) / k y = answer(s["text"], top, seed) calls, toks = calls + 1, toks + y["input"] + y["output"] gen += relevant(s, y["chunk"]) return rec / 40, prec / 40, gen / 40, calls, toks print("candidates recall@3 precision@3 generation calls tokens latency class") previous, gap = None, [0.0, 0.0, 0.0] for n in (3, 6, 12, 24, 48): a, e, u, c, b = measure(n) i = measure(n, SEED + 1) gap = [max(f, abs(x - y)) for f, x, y in zip(gap, (a, e, u), i[:3])] print("%10d %8.4f %11.4f %10.4f %5d %8d %s" % (n, a, e, u, c, b, "single stage" if n == 3 else "two stages, %d extra calls" % n)) print("run-to-run difference (seed 20260219): recall %.4f precision %.4f generation %.4f" % tuple(gap)) ``` ``` candidates recall@3 precision@3 generation calls tokens latency class 3 0.7750 0.4167 0.7000 40 3654 single stage 6 0.8000 0.4917 0.7250 280 12226 two stages, 6 extra calls 12 0.9750 0.8250 0.9250 520 20796 two stages, 12 extra calls 24 1.0000 0.9000 1.0000 1000 38003 two stages, 24 extra calls 48 0.9500 0.9083 0.9500 1960 71797 two stages, 48 extra calls run-to-run difference (seed 20260219): recall 0.0000 precision 0.0250 generation 0.1000 ``` #### As the Candidate Count Grows The first row is the baseline: no reranking, the first three go straight into the context, precision is **0.4167**, generation is **0.7000**, forty calls. With six candidates precision rises to **0.4917**. The gain is 0.0750 and stands above the run-to-run difference (0.0250), so it is real; but in return calls go **from forty to two hundred eighty** and tokens from 3654 to 12226. Generation rises to **0.7250** in this row; the gain is 0.0250, and because the run-to-run difference of generation is 0.1000, this row cannot be read as "it raised generation"; it is read as **this measurement cannot tell**. With twelve candidates the table changes: precision **0.8250**, recall **0.9750**, generation **0.9250**. Against the baseline there is a gain of 0.4083 in precision and 0.2250 in generation, and both are far above the threshold. With twenty-four candidates recall becomes **1.0000**, precision **0.9000** and generation **1.0000**; this setup puts the correct chunk into the context and into the source of the answer for all forty queries. On the cost side the same row writes **one thousand calls** and **38003 tokens**. The baseline was forty calls; that is twenty-five times. The latency class has changed too: the pipeline went from a single stage to **two stages**, and the second stage scales with the candidate count. The candidates can be scored in parallel, but none of them can start before the first stage finishes, that is, while the length of the chain stays fixed at two steps, its **width** goes up to twenty-four. #### The Gain That Cannot Be Measured and the Loss That Is Measured The last row is the real finding of this lesson. When the candidate count is raised from twenty-four to forty-eight, precision goes from 0.9000 to **0.9083**. The gain is **0.0083**, **a third** of the run-to-run difference. This row cannot be written as "it increased precision"; **whether it increased it cannot be known with this measurement**. What is paid in return, on the other hand, has been measured: calls go from 1000 to **1960**, tokens from 38003 to **71797**. There is one more thing that can be measured in the same row, and its direction is the opposite. The recall of the top three falls **from 1.0000 to 0.9500**; the drop is 0.0500 and the run-to-run difference of recall is 0.0000, so it is **real**. Generation also goes down from 1.0000 to 0.9500, although that drop stays below the run-to-run difference of generation (0.1000). As the candidate set grows, more wrong chunks come in front of the noisy ranker and the chance that it carries one of them into the top three goes up. Because the noise of the ranker can mask a difference of a single condition, a wrong candidate passing a correct one becomes unavoidable in a crowd. This is the instance, in this lesson, of the course's second claim: **enlarging the retrieval machinery made the job worse**. The right reading is not to pick the row with the highest precision but to find the place where the gain falls below the threshold and to stop there. In this workload that place is **twenty-four candidates**. #### Summary - Because a single similarity score compresses the query without seeing the chunk, it cannot search for the concept and the record type together; recall asks for twenty chunks, while precision at the top three stays at 0.4167. - The second stage, which reads the candidates together with the query, raises precision to 0.8250 with twelve candidates and to 0.9000 with twenty-four, and generation from 0.7000 to 1.0000. - The cost is one call per candidate: forty calls and 3654 tokens become 1000 calls and 38003 tokens with twenty-four candidates; the latency class goes from a single stage to two stages. - With forty-eight candidates precision rises by only 0.0083 and stays below the run-to-run difference of 0.0250; this row cannot be read as having gained. - In the same row the recall of the top three falls from 1.0000 to 0.9500, and this drop is above the threshold of 0.0000: enlarging the candidate set presents the noisy ranker with more wrong candidates. #### Next Step In all the setups of this lesson **three** chunks entered the context, and always in the same order, that is, in the order the ranker gave. Two numbers show that this choice has not been questioned: where those three chunks stand against the limit of the context, and whether the **order** of those three changes the answer. The next lesson counts how many chunks fit into the context limit, shows what the pipeline does when the limit is exceeded, and measures the share of change in the answer by giving the same chunks in different orders. ### Context Construction Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/context-assembly Placing the retrieved chunks into the prompt and measuring that this placement is a mechanism in its own right: after the instruction and the query, 13.4 chunks fit into a window of two hundred forty tokens, yet at the eleventh chunk six of the forty queries, at the twelfth and the thirteenth fifteen of them, exceed the limit. When the chunk count is raised from one to ten, retrieval recall rises from 0.7750 to 0.9750 while generation falls from 0.7750 to 0.5750. When the context always holds one relevant chunk with four irrelevant chunks beside it, generation is 0.7750 with the relevant chunk at the head, 0.6250 at the tail and 0.2000 in the middle, and when the order of the same chunks changes, 0.7000 of the answers change in the middle layout; when the seed is changed in the same order, the share of change is 0.4250. The previous two lessons corrected what gets retrieved: the query was translated into the index language, and the candidates were reranked in a second stage. In both, the retrieved chunks were put into the prompt **in the order they were ranked** and **in whatever number was asked for**. This lesson measures that step, because constructing the context is also a mechanism, and it has two constraints. The first is the **limit**: the context window is fixed, and when the chunk count exceeds it the request returns with no answer. The second is the **order**: when the same chunks are placed in different positions in the same prompt, the chunk the model looks at changes. In this lesson every answer also **cites** the code of the chunk it used; grounding makes it readable in which half the error lies. - **RP54.** The code is a **simulator**; no real embedding or model endpoint is called. The corpus, the concepts and the queries are **fictional**, seed **20260218**. - **RP55.** The target definition and the pipeline continue from `retrieval-pipeline/05`. Transformation and reranking are **closed**; the only setup measured is the construction of the context. - **RP56.** The context window is **240 tokens**. The window was defined in the Introduction to AI Engineering course and is not repeated; here it is used as a **constraint**. - **RP57.** Every chunk enters the context **with its code**, and the answer cites the code of the chunk it used. Because the simulator builds the answer from the chunk it picked, **answer faithfulness is complete by definition**; what is measured is **the accuracy of the citation**. - **RP58.** A request that exceeds the window returns with no answer, and that query is counted **wrong**. - **RP59.** The **position weight** of the generation simulator is modeled: the first chunk 1.0, the last chunk 0.8, those in between 0.45. What this lesson measures is the consequence of this weight. - **RP60.** In the controlled experiment the context **always** holds one relevant chunk, and j irrelevant chunks are placed beside it; this fixes retrieval and measures only the context. - **RP61. The share of change** is the share with which **the cited chunk** changes when the same chunks are given in a different order. The comparison baseline is the run in which the order is fixed and the seed changes. #### Context Budget The budget of a prompt consists of three items: the instruction, the query and the chunks. The first two are fixed, and the third grows with k. The block below sets up the pipeline, derives the budget, and prints retrieval and generation side by side while raising k from one to thirteen. ```python # SIMULATOR -- FICTIONAL. No real embedding or model endpoint is called; the corpus, # concepts and queries are fictional, seed 20260218. import math SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def text_of(d): return d["title"] + " " + " ".join(d["paras"]) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- pipeline: chunks, hybrid retrieval, target definition and generation CHUNKS = [{"code": d["code"], "main": d["main"], "text": p} for d in DOCUMENTS for p in d["paras"]] for p in CHUNKS: p["vec"], p["bag"] = embed(p["text"]), set(tokenize(p["text"])) for s in QUERIES: s["record"] = s["text"].split()[-1] def retrieve(q, k=5, w=0.5): qv, qb = embed(q), set(tokenize(q)) s = sorted(((cosine(qv, p["vec"]) + w * len(qb & p["bag"]) / len(qb | p["bag"]), i) for i, p in enumerate(CHUNKS)), key=lambda t: (-t[0], t[1])) return [CHUNKS[i] for _, i in s[:k]] def relevant(s, p): return CONCEPTS[s["concept"]][1] in p["text"] and s["record"] in p["bag"] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h WINDOW = 240 COUNTER = [0] ANSWER_INSTRUCTION = "Answer by looking at the chunks below and write the code of the chunk you used . " def answer(query, chunks, seed=SEED, temperature=0.12): """FICTIONAL generation: one chunk is picked from the context, and the chunk's POSITION changes the weight.""" COUNTER[0] += 1 prompt = ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query toks = tokenize(prompt) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0, "chunk": None} r, qb, n = rng(seed ^ trace_id(prompt)), set(tokenize(query)), len(chunks) score = [(1.0 if i == 0 else 0.8 if i == n - 1 else 0.45) * (0.3 + len(qb & x["bag"]) / len(qb)) for i, x in enumerate(chunks)] w = [math.exp(a / temperature) for a in score] chosen = chunks[pick(r(), [a / sum(w) for a in w])] return {"status": "ok", "input": len(toks), "chunk": chosen, "output": len(tokenize(f"{chosen['code']} : {chosen['text']}"))} def build_prompt(query, chunks): return (ANSWER_INSTRUCTION + " ".join(f"[{p['code']}] {p['text']} ." for p in chunks) + " Question: " + query) per_chunk = sum(len(tokenize(f"[{p['code']}] {p['text']} .")) for p in CHUNKS) / len(CHUNKS) per_query = sum(len(tokenize(s["text"])) for s in QUERIES) / 40 print("window %d ; instruction %d + query %.1f + per chunk %.1f tokens -> chunks that fit %.1f" % (WINDOW, len(tokenize(ANSWER_INSTRUCTION)), per_query, per_chunk, (WINDOW - len(tokenize(ANSWER_INSTRUCTION)) - per_query) / per_chunk)) print(" k context avg largest overflow recall@k generation cited concept cited record") for k in (1, 3, 5, 8, 10, 11, 12, 13): lens, rec, gen, over, cited_concept, cited_record = [], 0, 0, 0, 0, 0 for s in QUERIES: g = retrieve(s["text"], k) lens.append(len(tokenize(build_prompt(s["text"], g)))) rec += any(relevant(s, p) for p in g) y = answer(s["text"], g) if y["chunk"] is None: over += 1 continue cited_concept += CONCEPTS[s["concept"]][1] in y["chunk"]["text"] cited_record += s["record"] in y["chunk"]["bag"] gen += relevant(s, y["chunk"]) print("%2d %11.1f %7d %8d %8.4f %10.4f %13.4f %12.4f" % (k, sum(lens) / 40, max(lens), over, rec / 40, gen / 40, cited_concept / 40, cited_record / 40)) ``` ``` window 240 ; instruction 20 + query 5.5 + per chunk 16.0 tokens -> chunks that fit 13.4 k context avg largest overflow recall@k generation cited concept cited record 1 43.6 48 0 0.7750 0.7750 1.0000 0.7750 3 76.2 88 0 0.7750 0.7000 1.0000 0.7000 5 109.0 127 0 0.8000 0.6750 1.0000 0.6750 8 157.4 186 0 0.8500 0.5500 1.0000 0.5500 10 189.9 226 0 0.9750 0.5750 1.0000 0.5750 11 206.4 246 6 0.9750 0.5500 0.8500 0.5500 12 222.7 266 15 0.9750 0.4500 0.6250 0.4500 13 238.9 286 15 0.9750 0.4500 0.6250 0.4500 ``` #### Recall Rises While Generation Falls The budget calculation says **13.4 chunks** on average, and the table shows that the limit arrives earlier than that: up to ten chunks no request exceeds the limit; at eleven **six** of the forty queries do, at twelve **fifteen**, and at thirteen **fifteen** again. The average context at the thirteenth chunk is 238.9 tokens, and the largest is 286. The limit collides not with the average but with **the largest**; even when the average is 206.4, six requests return with no answer. The real finding is that the two columns go in **opposite directions**. When k goes from one to ten, retrieval recall rises **from 0.7750 to 0.9750**, while generation falls **from 0.7750 to 0.5750**. A plan that looks at the retrieval measure enlarges k and thinks it has improved; the generation measure says the opposite. Why the rule of the course asks for two columns can be seen in these rows. The last two columns say where the error is. The **cited concept** share stays at 1.0000 until k reaches ten: the model almost always cites a chunk that describes the correct concept. The **cited record** share, on the other hand, falls to 0.5750 together with generation. That is, the error is not in the concept but in the **record type**. When the limit is exceeded both fall at once for the overflowing requests, because there is no answer at all. What makes this distinction readable is the citation itself; if the answer were only text, the two kinds of error would stay inside a single number. #### The Share of the Order The k sweep changes two things together: how many chunks are in the context and where the relevant chunk stands. The block below isolates the second. In every query **one relevant chunk** is put into the context, j irrelevant chunks are added beside it, and the same set is given in three different orders. The last column prints, as the baseline, the run in which the order is fixed and only the seed changes. ```python def experiment(j, seed=SEED): """The context always holds one relevant chunk; j irrelevant chunks are placed beside it in three different layouts.""" hits, diff, seed_diff = {"head": 0, "middle": 0, "tail": 0}, {"middle": 0, "tail": 0}, 0 for s in QUERIES: candidates = retrieve(s["text"], 40) good = next(p for p in candidates if relevant(s, p)) bad = [p for p in candidates if not relevant(s, p)][:j] layout = {"head": [good] + bad, "tail": bad + [good], "middle": bad[:j // 2] + [good] + bad[j // 2:]} picked = {} for a in ("head", "middle", "tail"): picked[a] = answer(s["text"], layout[a], seed)["chunk"] hits[a] += picked[a] is not None and relevant(s, picked[a]) for a in ("middle", "tail"): diff[a] += picked[a] is not picked["head"] seed_diff += answer(s["text"], layout["head"], seed + 1)["chunk"] is not picked["head"] return [hits[a] / 40 for a in ("head", "middle", "tail")], diff["middle"] / 40, diff["tail"] / 40, seed_diff / 40 print("irrelevant generation: head middle tail | share of change: middle tail run-to-run") for j in (0, 2, 4, 8): u, dm, dt, sd = experiment(j) print("%10d %.4f %.4f %.4f | %.4f %.4f %10.4f" % (j, u[0], u[1], u[2], dm, dt, sd)) print("second run (seed 20260219), j=4: head %.4f middle %.4f tail %.4f" % tuple(experiment(4, SEED + 1)[0])) ``` ``` irrelevant generation: head middle tail | share of change: middle tail run-to-run 0 1.0000 1.0000 1.0000 | 0.0000 0.0000 0.0000 2 0.8250 0.2250 0.5000 | 0.7000 0.4500 0.3750 4 0.7750 0.2000 0.6250 | 0.7000 0.5250 0.4250 8 0.6500 0.1500 0.6000 | 0.8500 0.5500 0.6000 second run (seed 20260219), j=4: head 0.7500 middle 0.1500 tail 0.5750 ``` #### The Price of an Irrelevant Chunk The first row sets up the control: when the context holds only the relevant chunk, generation is **1.0000** and the order changes nothing. The drop in every later row is the price of the irrelevant chunks that were added. With the relevant chunk at the **head**, eight irrelevant chunks take generation from 1.0000 down to **0.6500**. With the same chunk in the **middle** it goes from 1.0000 down to **0.1500**. The drop has no threshold: even the first two irrelevant chunks take generation down to 0.8250 at the head and **0.2250** in the middle. This is the counterpart, in the retrieval context, of the finding measured in `ai-assisted-development/02`, and the added information here is this: the size of the price depends on **where the relevant chunk stands**. The row with four irrelevant chunks makes the order readable on its own. The same five chunks, the same seed, only the order different: generation is **0.7750 at the head**, **0.6250 at the tail**, **0.2000 in the middle**. The head and the middle are far apart. Whether this difference is real is tested with the last column: in the run where the order is fixed and only the seed changes, **0.4250** of the answers change, while when the order changes **0.7000** of them do in the middle layout. The numbers of the second run in the same row (0.7500 / 0.1500 / 0.5750) give the same ordering too, and the gap between the head and the middle stays as large in both runs; no layout moves by more than 0.0500 from one run to the other. The rule that follows from this is: **the order of the context is a design decision, and without being written down the measurement cannot be repeated.** The same retrieval, the same chunks and the same model swing between 0.7750 and 0.2000 in generation when the arrangement of the chunks changes. The precision that reranking bought with 1000 calls in `retrieval-pipeline/06` is spent if the chunks are put into the context in a bad order. #### Summary - After the instruction and the query, 13.4 chunks fit into a window of two hundred forty tokens; at the eleventh chunk six of the forty queries, at the twelfth and the thirteenth fifteen of them, exceed the limit and return with no answer. - The limit collides not with the average context but with the largest: even when the average is 206.4 tokens, six requests exceed the window. - When the chunk count goes from one to ten, retrieval recall rises from 0.7750 to 0.9750 while generation falls from 0.7750 to 0.5750; the two measures go in opposite directions. - The cited concept share stays at 1.0000 while the cited record share falls to 0.5750: the error is not in the concept but in the record type, and what makes this distinction readable is grounding. - With the relevant chunk at the head eight irrelevant chunks take generation from 1.0000 to 0.6500, and in the middle to 0.1500, and the drop has no threshold. - When the same five chunks are given in a different order, generation swings between 0.7750 and 0.2000 and 0.7000 of the answers change in the middle layout; when the seed changes with the order fixed, the share of change is 0.4250. #### Next Step In seven lessons the pipeline was built from end to end: the corpus was prepared, split, indexed, refreshed, the query was transformed, the candidates were reranked and the context was constructed. Each lesson defended its own mechanism with its own number, but all of these numbers carry the same flaw: a setup being **good** does not mean that the correct chunk arriving and the correct answer being produced happen **in the same query**. The table of this lesson has a row where the cited concept share is 1.0000 while generation is 0.5500; there the model pointed at the right source and read the wrong record. The next lesson counts these four cases separately and shows which one a single number hides. ### Measuring Retrieval and Generation Quality Separately Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/measuring-retrieval-and-generation-quality-separately Showing that a single end-to-end accuracy rate stacks four separate cases on top of one another: on the forty-query fictional set the baseline without retrieval comes out at 0.3750, and the whole of this rate is chance, because the correct chunk is never retrieved. With a three-chunk context the single number is 0.9500, and with a five-chunk context it is also 0.9500; but with three chunks the two failures are retrieval faults, with five chunks the same two failures are generation faults, and the single number does not tell them apart. Retrieval recall is 0.9500 with three chunks and 1.0000 with five; the cost goes from 120 chunks and 1872 tokens to 200 chunks and 2932 tokens. The difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the three-chunk context. The class subgroup shows that both faults are collected in the same class. The previous lesson gave the same chunks in different orders and counted the share of change in the answer. The pipeline is now built: the document was prepared, split into chunks, indexed, the query was transformed in a single step, the candidates were reranked and the context was constructed under the limit. Each of the seven lessons defended its own step with its own number. This lesson stops where those numbers come together. When a retrieval-augmented generation pipeline is run, what is usually left is a single rate: how many questions were answered correctly. The claim of this lesson is that this single rate stacks **four separate cases** on top of one another, and that from inside the stack it cannot be read which stage is broken. The evaluation procedure itself was established in the Model Evaluation, Interpretability and Ethics course, together with its lessons on model evaluation and subgroups, and is not repeated here; the only thing taken over from there is **the baseline rule**, and in this course the baseline is **the call without retrieval**. - **RP71.** The vocabulary, the concept axis, the embedding and the query set are **fictional** and are produced inside the lesson. There is no real institution, product, document, vector store or embedding endpoint. - **RP72.** The concept has one surface in the **document language** (`tier limit`) and another surface in the **query language** (`which band am i in`), and the two share no token. - **RP73.** **The retrieval measure**: if the document surface of the target procedure appears inside the retrieved chunks, retrieval is right. This measure is independent of the seed. - **RP74.** **The generation measure**: if the procedure named in the answer is the target procedure, generation is right. The answer comes from a **language model simulator**; the simulator does not call a real endpoint. - **RP75.** The **prior knowledge of the simulator is at the class level only**. From the query language it infers which class the query belongs to and gives an equal, small base score to the four procedures of that class, but it does not know a procedure name on its own. Every procedure name that appears in the context adds a point. - **RP76.** **Cost** is written in three units: the number of retrieved chunks, the tokens entering the prompt and the retrieval step. Time is not written. #### The Retrieval Column: Did the Evidence Arrive The first column never looks at the model. The only question is whether the retrieved chunks carry the target procedure. Two rates stand side by side: **retrieval recall** is the share of queries for which at least one right chunk arrived, and **retrieval precision** is the share of the retrieved chunks that are right. ```python # FICTIONAL. The corpus, the concept axis, the embedding and the query set are produced inside the # lesson; there is no real institution, product, document or embedding endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) CHUNKS = [{"code": d["code"], "label": d["label"], "text": p} for d in DOCUMENTS for p in d["paras"]] VECS = [embed(p["text"]) for p in CHUNKS] def retrieve(q, k): v = embed(q) s = sorted(range(len(CHUNKS)), key=lambda i: (-cosine(v, VECS[i]), i)) return [CHUNKS[i] for i in s[:k]] def measure_retrieval(k): """The retrieval column only: do the retrieved chunks carry the target procedure.""" recall = precise = 0 for s in QUERIES: surface = CONCEPTS[s["concept"]][1] carriers = [p for p in retrieve(s["text"], k) if surface in p["text"]] recall += 1 if carriers else 0 precise += len(carriers) / k return recall / len(QUERIES), precise / len(QUERIES) print(f"corpus {len(DOCUMENTS)} documents , {len(CHUNKS)} chunks , queries {len(QUERIES)}") print("\n k retrieval recall retrieval precision chunks retrieved") for k in (1, 3, 5): a, ke = measure_retrieval(k) print(f"{k:>3} {a:>16.4f} {ke:>19.4f} {k * len(QUERIES):>16}") ``` ``` corpus 200 documents , 693 chunks , queries 40 k retrieval recall retrieval precision chunks retrieved 1 0.9500 0.9500 40 3 0.9500 0.9500 120 5 1.0000 0.9600 200 ``` Going from one chunk to three does **not move** recall: 0.9500 stays the same, while the number of retrieved chunks triples. Going to five makes recall **1.0000**, that is, the evidence is now inside the context for every query. The retrieval column says this much and cannot say more: it knows that the chunk **arrived**, and it does not know whether it was used. #### The Generation Column and the Four Cases The second column runs the same queries with a **language model simulator**. The simulator does not call a real endpoint; it scores the procedure names that appear in the context, gives a small base score to the four procedures of the class it recognizes from the query, and samples with a temperature. When the two columns intersect, every query falls into one of four boxes. ```python from math import exp def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def generate(q, context, seed=SEED, temperature=0.6): """FICTIONAL language model simulator. It scores the procedure names that appear in the context; its prior knowledge is at the class level only, it does not know the procedure name.""" prompt = q + " || " + " ".join(p["text"] for p in context) dm, sq = " " + prompt.lower() + " ", " " + q.lower() + " " label = next((CONCEPTS[k][0] for k in KEYS if CONCEPTS[k][2] in sq), None) score = {} for k in KEYS: p = (0.15 if CONCEPTS[k][0] == label else 0.0) + dm.count(CONCEPTS[k][1]) if p: score[k] = p if not score: return None, prompt names = list(score) a = [exp(score[k] / temperature) for k in names] t = sum(a) return names[pick(rng(seed ^ trace_id(prompt))(), [x / t for x in a])], prompt def run_setup(k, seed=SEED): d = {"rr": 0, "rw": 0, "wr": 0, "ww": 0, "tok": 0} for s in QUERIES: context = retrieve(s["text"], k) if k else [] reply, prompt = generate(s["text"], context, seed) d["tok"] += len(tokenize(prompt)) got_chunk = any(CONCEPTS[s["concept"]][1] in p["text"] for p in context) got_answer = reply == s["concept"] d["rr" if got_chunk and got_answer else "rw" if got_chunk else "wr" if got_answer else "ww"] += 1 return d n = len(QUERIES) print("setup right+right right+wrong wrong+right wrong+wrong single number tokens") for name, k in (("no retrieval", 0), ("k = 1", 1), ("k = 3", 3), ("k = 5", 5)): d = run_setup(k) print(f"{name:<12} {d['rr']:>11} {d['rw']:>11} {d['wr']:>11} {d['ww']:>11}" f" {(d['rr'] + d['wr']) / n:>13.4f} {d['tok']:>9}") print("\nsecond run (seed 20260219) , single number and run-to-run difference") for name, k in (("no retrieval", 0), ("k = 1", 1), ("k = 3", 3), ("k = 5", 5)): a = (lambda d: (d["rr"] + d["wr"]) / n)(run_setup(k)) b = (lambda d: (d["rr"] + d["wr"]) / n)(run_setup(k, SEED + 1)) print(f"{name:<12} {a:>7.4f} {b:>9.4f} {b - a:>+10.4f}") print("\nclass subgroup (right+right / right+wrong / wrong+right / wrong+wrong)") print(f"{'class':<12} {'k = 3':>16} {'k = 5':>16}") for label in CLASSES: row = [] for k in (3, 5): c = [0, 0, 0, 0] for s in QUERIES: if s["label"] != label: continue context = retrieve(s["text"], k) y, _ = generate(s["text"], context) got_chunk = any(CONCEPTS[s["concept"]][1] in p["text"] for p in context) c[0 if got_chunk and y == s["concept"] else 1 if got_chunk else 2 if y == s["concept"] else 3] += 1 row.append(" / ".join(str(x) for x in c)) print(f"{label:<12} {row[0]:>16} {row[1]:>16}") ``` ``` setup right+right right+wrong wrong+right wrong+wrong single number tokens no retrieval 0 0 15 25 0.3750 258 k = 1 35 3 0 2 0.8750 796 k = 3 38 0 0 2 0.9500 1872 k = 5 38 2 0 0 0.9500 2932 second run (seed 20260219) , single number and run-to-run difference no retrieval 0.3750 0.3250 -0.0500 k = 1 0.8750 0.8000 -0.0750 k = 3 0.9500 0.9500 +0.0000 k = 5 0.9500 0.9500 +0.0000 class subgroup (right+right / right+wrong / wrong+right / wrong+wrong) class k = 3 k = 5 tariff 8 / 0 / 0 / 0 8 / 0 / 0 / 0 fault 7 / 0 / 0 / 2 7 / 2 / 0 / 0 meter 10 / 0 / 0 / 0 10 / 0 / 0 / 0 outage 6 / 0 / 0 / 0 6 / 0 / 0 / 0 maintenance 7 / 0 / 0 / 0 7 / 0 / 0 / 0 ``` #### The Same Number, a Different Composition The first row of the table gives the baseline. The call without retrieval comes out at **0.3750**, and **the whole of this rate comes from the third box**: the right chunk was never retrieved, and fifteen answers came out right from the class-level guess. Someone who looks at the single number says the model knows more than a third of the questions. The four-cell contingency of retrieval and generation says, in the same row, that **no information was relied on**. This distinction prevents mistaking chance for a capability. The real demonstration is in the last two rows. The single number of the three-chunk and the five-chunk setups is **exactly the same: 0.9500**. Their composition is not. With three chunks the two failures are in the **wrong chunk + wrong answer** box, that is, the evidence never arrived; the problem is in retrieval. With five chunks retrieval recall rises to 1.0000 and those two failures move to the **right chunk + wrong answer** box: the evidence is in the context and the model did not use it. This second kind is an **answer faithfulness** fault and its fix is entirely different. The single number never shows this move, but the cost does: from 120 chunks to 200 chunks, from 1872 tokens to **2932 tokens**. The subgroup breakdown writes where which fix is to be made. Both faults are collected in **a single class**, the `fault` class; the other four classes are complete in both setups. Going to five did not rescue the lost queries, it only **changed the name** of the fault. Finally, the difference between the two runs says how much of the single number can be trusted. For the baseline without retrieval the difference is **0.0500**; a number that rests on chance is not fixed by anything. For one chunk, which brings the evidence for only part of the queries, the difference is **0.0750**. For three and five chunks the difference is **0.0000**: because the evidence in the context opens the score gap wide enough, sampling no longer turns the result. What fixes the number is not the apparatus itself but **the evidence that enters the context**. #### Summary - Retrieval and generation are two separate columns: the first measures that the evidence arrived, the second measures that the evidence was used, and one cannot take the place of the other. - The baseline without retrieval comes out at 0.3750 on this workload, and the whole of this rate is in the **wrong chunk + right answer** box, that is, it is chance. - The single number of the three-chunk and the five-chunk setups is also 0.9500, but in the first the two failures are retrieval faults and in the second the same two failures are generation faults. - Going to five chunks takes recall from 0.9500 to 1.0000; the cost is going from 120 chunks to 200 chunks and from 1872 tokens to 2932 tokens, while the end-to-end number does not change. - The class subgroup collects both faults in the `fault` class; the difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the three-chunk context. #### Next Step This lesson separated not when retrieval wins but **where the gain comes from**. One question was left unanswered: the retrieval pipeline is not the only way to give a model information. For the same task the model itself can be changed too; in the Building Applications with Model APIs course the number for fine-tuning was measured, and it was seen that with sixty examples it could not beat the prompt setup. The next lesson puts these two solutions on the same table and measures them side by side on three separate tasks: it counts **which problem** each one solves and on which task **retrieval does not win**. ### Measuring Retrieval and Fine-Tuning Side by Side Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/retrieval-and-fine-tuning-comparison Measuring three separate tasks on the same workload, once with retrieval and once with fine-tuning: on the forty-query fictional set, knowledge freshness comes out at 0.0000 without retrieval, 0.9500 with retrieval alone and 0.6750 with fine-tuning on forty-eight examples, and adding examples does not lift this ceiling, because stale knowledge is not a coverage problem. Format compliance stays at 0.0000 with retrieval and reaches 1.0000 with fine-tuning on twelve examples: this is the task retrieval does not win. Domain language is 0.9500 with retrieval and 1.0000 with fully covering fine-tuning. The cost is extra tokens per request and forty retrieval steps for retrieval, and forty-eight labeled examples for fine-tuning. The difference between two runs is 0.0500 for the baseline without retrieval and 0.0000 for the other three setups. The previous lesson split the single number of the retrieval pipeline into four cases and showed that the single number hides which stage is broken. The split also exposed this: the job of retrieval is to change **the information entering the prompt**. It never touches the model itself. There is a way to touch the model, and it has been measured. In the Building Applications with Model APIs course, fine-tuning was tried on the same support task: the baseline was **0.6000**, the prompt setup with sixty examples was **0.9667**, and fine-tuning with the same sixty examples was **0.9500**. The context per request went from **28.48** tokens to **623.48** tokens. That procedure is not repeated here; those numbers are **the input** of this lesson. The question of this lesson is a different one: given a task, which of the two solves **which problem**, and on which task retrieval gains **nothing at all**. - **RP77.** The vocabulary, the concept axis, the embedding, the query set and the refresh are **fictional** and are produced inside the lesson. There is no real institution, product, document or embedding endpoint. - **RP78.** **The knowledge freshness task**: every procedure has an **validity stamp**. After the fine-tuning snapshot, the stamp changed in a part of the procedures. The right answer is the stamp **in force**. - **RP79.** **The format compliance task**: the answer is required to fit a fixed pattern with two fields (`procedure | stamp`). This pattern is fictional and concerns only the output channel. - **RP80.** **The domain language task**: finding which procedure the surface in the query language (`which band am i in`) corresponds to. This is the very same job that the previous lesson measured. - **RP81.** **Fine-tuning here is a learning simulator**: for a procedure it **has seen** in the training set it resolves the query surface directly, in a class it **has seen** it applies the output pattern, and it gives the validity stamp **from the snapshot**. For a procedure it has not seen it falls back to the guess of the language model simulator. - **RP82.** **The cost is in two separate units and is not added up**: for retrieval, extra tokens per request and the retrieval step; for fine-tuning, **the number of labeled examples**. The two have no common unit. - **RP83.** The measures are taken over forty queries; retrieval brings three chunks. #### Three Tasks, One Workload The fictional refresh changes the validity stamp of a part of the procedures. The fine-tuning snapshot carries the old one, the corpus carries the new one. ```python # FICTIONAL. The corpus, the concept axis, the embedding, the query set and the refresh are # produced inside the lesson; there is no real institution, product, document or embedding endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # FICTIONAL refresh: a part of the procedures changed after the fine-tuning snapshot. STAMP_TRAIN, STAMP_CURRENT = {}, {} for i, k in enumerate(KEYS): STAMP_TRAIN[k] = f"y{i + 1:02d}a" STAMP_CURRENT[k] = (f"y{i + 1:02d}b" if rng(SEED + 31 * (i + 1))() < 0.45 else f"y{i + 1:02d}a") STALE = [k for k in KEYS if STAMP_TRAIN[k] != STAMP_CURRENT[k]] CHUNKS = [{"code": d["code"], "main": d["main"], "text": p + " effective " + STAMP_CURRENT[d["main"]]} for d in DOCUMENTS for p in d["paras"]] VECS = [embed(p["text"]) for p in CHUNKS] TRAINING = [DOCUMENTS[int(rng(SEED + 65537 * (i + 1))() * len(DOCUMENTS))]["main"] for i in range(48)] print(f"corpus {len(DOCUMENTS)} documents , {len(CHUNKS)} chunks , queries {len(QUERIES)}") print(f"refreshed procedures {len(STALE)} / {len(KEYS)} , queries asking about a stale procedure " f"{sum(1 for s in QUERIES if s['concept'] in STALE)} / {len(QUERIES)}") print(f"training set {len(TRAINING)} labeled examples , procedures covered " f"{len(set(TRAINING))} , classes covered {len({CONCEPTS[x][0] for x in set(TRAINING)})}") ``` ``` corpus 200 documents , 693 chunks , queries 40 refreshed procedures 9 / 20 , queries asking about a stale procedure 13 / 40 training set 48 labeled examples , procedures covered 20 , classes covered 5 ``` Nine of the twenty procedures were refreshed, and **thirteen** of the forty queries now ask for an answer other than the one fine-tuning knows. The training set is forty-eight labeled examples and it is fully covering: it sees all twenty of the twenty procedures and all five of the five classes. This is the best condition that can be given to fine-tuning. #### Four Setups The same forty queries are solved in four setups: the baseline without retrieval, retrieval alone, fine-tuning alone and the two together. The three tasks are three separate columns and **are not gathered into a single number**. ```python from math import exp def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def retrieve(q, k=3): v = embed(q) s = sorted(range(len(CHUNKS)), key=lambda i: (-cosine(v, VECS[i]), i)) return [CHUNKS[i] for i in s[:k]] def solve(s, retrieval, tuned, seed=SEED, temperature=0.6): """Retrieval changes the input of the prompt, fine-tuning changes the model itself.""" context = retrieve(s["text"]) if retrieval else [] prompt = s["text"] + " || " + " ".join(p["text"] for p in context) seen = set(TRAINING[:tuned]) sq = " " + s["text"].lower() + " " concept = next((x for x in KEYS if CONCEPTS[x][2] in sq and x in seen), None) if concept is None: # fine-tuning has not seen this procedure: the simulator guesses dm = " " + prompt.lower() + " " label = next((CONCEPTS[x][0] for x in KEYS if CONCEPTS[x][2] in sq), None) sc = {x: (0.15 if CONCEPTS[x][0] == label else 0.0) + dm.count(CONCEPTS[x][1]) for x in KEYS} names = [x for x in KEYS if sc[x]] a = [exp(sc[x] / temperature) for x in names] t = sum(a) concept = names[pick(rng(seed ^ trace_id(prompt))(), [x / t for x in a])] if retrieval: b = context[0]["text"].split() stamp = b[b.index("effective") + 1] else: stamp = STAMP_TRAIN[concept] if tuned else "none" fmt_ok = tuned > 0 and CONCEPTS[concept][0] in {CONCEPTS[x][0] for x in seen} return concept, stamp, fmt_ok, len(tokenize(prompt)) def measure(retrieval, tuned, seed=SEED): n = len(QUERIES) t = d = b = tok = 0 for s in QUERIES: concept, stamp, fmt, nt = solve(s, retrieval, tuned, seed) t += stamp == STAMP_CURRENT[s["concept"]] d += concept == s["concept"] b += fmt tok += nt return t / n, b / n, d / n, tok SETUPS = (("no-retrieval baseline", 0, 0), ("retrieval only", 1, 0), ("fine-tuning only", 0, 48), ("both together", 1, 48)) print("setup freshness format domain language tokens retrieval steps labeled examples") for name, e, i in SETUPS: t, b, d, tok = measure(e, i) print(f"{name:<21} {t:>9.4f} {b:>7.4f} {d:>15.4f} {tok:>9} {e * len(QUERIES):>15} {i:>16}") print("\nfine-tuning example count sweep (no retrieval)") print("examples seen concepts seen classes freshness format domain language") for i in (0, 6, 12, 24, 48): t, b, d, _ = measure(0, i) g = set(TRAINING[:i]) print(f"{i:>8} {len(g):>13} {len({CONCEPTS[x][0] for x in g}):>12}" f" {t:>9.4f} {b:>7.4f} {d:>15.4f}") print("\nsecond run (seed 20260219) , domain language hit rate and run-to-run difference") for name, e, i in SETUPS: a, b = measure(e, i)[2], measure(e, i, SEED + 1)[2] print(f"{name:<21} {a:>7.4f} {b:>8.4f} {b - a:>+9.4f}") ``` ``` setup freshness format domain language tokens retrieval steps labeled examples no-retrieval baseline 0.0000 0.0000 0.3750 258 0 0 retrieval only 0.9500 0.0000 0.9500 2232 40 0 fine-tuning only 0.6750 1.0000 1.0000 258 0 48 both together 0.9500 1.0000 1.0000 2232 40 48 fine-tuning example count sweep (no retrieval) examples seen concepts seen classes freshness format domain language 0 0 0 0.0000 0.0000 0.3750 6 6 4 0.3750 0.7500 0.5750 12 9 5 0.3750 1.0000 0.6250 24 15 5 0.4000 1.0000 0.7250 48 20 5 0.6750 1.0000 1.0000 second run (seed 20260219) , domain language hit rate and run-to-run difference no-retrieval baseline 0.3750 0.3250 -0.0500 retrieval only 0.9500 0.9500 +0.0000 fine-tuning only 1.0000 1.0000 +0.0000 both together 1.0000 1.0000 +0.0000 ``` #### The Task Retrieval Does Not Win The three columns give three separate answers, and none covers another. **On knowledge freshness retrieval wins.** Retrieval alone is **0.9500**, fully covering fine-tuning is **0.6750**. The gap is not chance: what is lost is exactly the thirteen queries that ask about the nine refreshed procedures. The sweep column confirms this. While the example count goes from six to forty-eight, domain language climbs from 0.5750 to 1.0000, but freshness stands still at 0.3750 (0.4000 at twenty-four examples) and tops out at **0.6750** only when the coverage is complete. More examples **do not correct** this, because staleness is not a coverage problem: the model remembers correctly what it learned correctly, and what it learned is no longer in force. **On format compliance retrieval gains nothing.** The baseline without retrieval is 0.0000, retrieval alone is also **0.0000**. The three retrieved chunks enrich the context and do not touch the output pattern. Fine-tuning makes the same task **1.0000 with twelve examples**; what is needed to learn the pattern is not procedure coverage but **class coverage**, and five classes are complete at twelve examples. Retrieval changes **the input** of the prompt, fine-tuning changes **the output behavior** of the model; format is in the second set. **On domain language both win, and with full coverage fine-tuning comes out ahead.** Retrieval is 0.9500, fine-tuning 1.0000. But the 1.0000 of fine-tuning depends on the forty-eight examples that see all twenty procedures; with twenty-four examples the same number is **0.7250**. The 0.9500 of retrieval, on the other hand, asks for no labeled example. Because the costs carry no common unit, they are written separately. Retrieval takes the total tokens **from 258 to 2232** and adds forty **retrieval steps**. The tokens of fine-tuning **stay at 258** and there is no retrieval step, but it asks for forty-eight **labeled examples**, and each time the stamp is refreshed a part of these examples becomes invalid. The two together give the best column on all three tasks and pay **both costs at once**. The difference between two runs is in the last table. The baseline without retrieval falls from 0.3750 to 0.3250 in the second run (**−0.0500**); a number that does not rest on information is the only one that moves. In the other three setups the difference is **0.0000**. Both the evidence in the context and the learned mapping prevent sampling from turning the result. #### Summary - Retrieval changes the input of the prompt, fine-tuning changes the output behavior of the model; this distinction decides which task is solved with which. - On knowledge freshness retrieval makes 0.9500 and fully covering fine-tuning makes 0.6750; what is lost is exactly the thirteen queries that ask about the nine refreshed procedures. - Increasing the example count does not correct freshness: while domain language goes from 0.5750 to 1.0000, freshness stays at 0.3750 and tops out at 0.6750. - On format compliance retrieval adds nothing from 0.0000 to 0.0000; fine-tuning makes 1.0000 with twelve examples, because the pattern needs not procedure but class coverage. - The costs are not added up: retrieval takes 258 tokens to 2232 and adds forty retrieval steps, and fine-tuning keeps the tokens fixed and asks for forty-eight labeled examples. #### Next Step Both solutions share an assumption: that everything in the corpus is **plain text**. The fictional documents were built entirely of sentences, and chunking meant cutting a sequence of sentences. The real procedure documents of the field team are not like that; inside them are **tables** whose rows and columns carry the meaning. When a table is cut like a sentence, the row and the header are separated from each other and the chunk alone says nothing. The next lesson measures this on the same query set: chunking the table row by row and block by block does not give the same recall. ### Chunking Sources Containing Tables and Images Source: https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/chunking-sources-containing-tables-and-images Counting what the chunking decision does to retrieval recall when the same fictional corpus is presented in table form: because the procedure name sits only in the header, row-level chunking drops recall to 0.0000; block-level chunking gives 0.9500 but carries 88.7 tokens per request; storing each row together with its header gives the same 0.9500 recall with 38.5 tokens per request and raises the tokens stored in the index from 5702 to 8634. When the table arrives as an image, the source without a description gives 0.0000, and the source with a one-line description gives 0.9500 recall but 0.0000 answerability: the source is found, and the value inside it cannot be retrieved. This lesson closes the Embeddings and Retrieval-Augmented Generation course. The previous lesson measured retrieval and fine-tuning side by side on three tasks and showed both rest on the same assumption: that everything in the corpus is **plain text**. The fictional documents were built from sentences, and each chunk said what it was about by itself. The procedure documents of the field team are not like that. They contain **tables**, and in a table meaning does not live in the cell but in **its relation to the header**. When a table is split like a sentence, row and header are separated and the row says nothing by itself. In sources that carry images the loss goes one step further: no text exists at all. Multimodal input itself was measured in the Natural Language Processing and Computer Vision course, under computer vision, and the token cost of the input format was counted in the Building Applications with Model APIs course: the fine grid gave a hit rate of **0.8167**, the cropped grid **0.7000**, and the tokens per usable record fell from **224.1** to **60.0**. Those procedures are not repeated here. This lesson measures only **the chunking decision**. - **RP84.** The corpus, the concept axis, the embedding, the query set and the table source are **fictional** and are produced inside the lesson. There is no real institution, product, document, parser or embedding endpoint. - **RP85.** This lesson turns the same two hundred documents into **tables**. **The procedure name stays only in the header**; the rows carry the field name, the requirement and the source columns. What the table is about is not written in the cells. - **RP86.** Five chunking modes are measured: **row** (each row is a chunk, the header is dropped), **block** (the whole table is one chunk), **row + header** (each row is stored together with the header), **visual** (the table arrives as an image, there is no text) and **visual + description** (the image has a one-line description). - **RP87.** Two measures are counted separately. **Retrieval recall**: does the name of the target procedure appear in the retrieved chunks. **Answerability**: do both the procedure name and a **row** of the table stand together in one of the retrieved chunks. An answer can be produced only if both are present. - **RP88.** The cost is written in three units: the number of chunks in the index, the tokens **stored** in the index, and the tokens entering the prompt **per request**. The first two are storage, the third is the cost of the call. - **RP89.** The whole measurement is **deterministic**: there is no sampling, the language model simulator is not called and the numbers do not change between runs, so the difference between two runs is zero by definition. - **RP90.** Retrieval brings three chunks in every mode; recall and answerability are read over these three. #### Five Ways to Chunk a Table The chunk size and overlap decision was measured on plain text in this course's chunking lesson. In a table the decision stands on another axis: **whether the header is stored together with the row**. ```python # FICTIONAL. The corpus, the concept axis, the embedding, the query set and the table source are # produced inside the lesson; there is no real institution, product, document or embedding endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # FICTIONAL table source: each procedure document is turned into a table. # The procedure name is ONLY in the header; the rows carry the field name, the requirement and the source. TABLES = [] for i, d in enumerate(DOCUMENTS): r = rng(SEED + 4093 * (i + 1)) TABLES.append({"code": d["code"], "main": d["main"], "head": f"{CONCEPTS[d['main']][1]} procedure table", "rows": [f"{p.split()[3]} | {'yes' if r() < 0.5 else 'no'}" f" | {'field' if r() < 0.5 else 'central'}" for p in d["paras"]]}) def build_index(mode): chunks = [] for t in TABLES: if mode == "row": chunks += [{"code": t["code"], "text": s} for s in t["rows"]] elif mode == "block": chunks.append({"code": t["code"], "text": t["head"] + " ; " + " ; ".join(t["rows"])}) elif mode == "row + header": chunks += [{"code": t["code"], "text": t["head"] + " ; " + s} for s in t["rows"]] elif mode == "visual": chunks.append({"code": t["code"], "text": ""}) else: chunks.append({"code": t["code"], "text": t["head"]}) return chunks def measure(mode, k=3): chunks = build_index(mode) vecs = [embed(p["text"]) for p in chunks] recall = answerable = tokens = 0 for s in QUERIES: v = embed(s["text"]) top = sorted(range(len(chunks)), key=lambda i: (-cosine(v, vecs[i]), i))[:k] name = CONCEPTS[s["concept"]][1] recall += any(name in chunks[i]["text"] for i in top) answerable += any(name in chunks[i]["text"] and " | " in chunks[i]["text"] for i in top) tokens += sum(len(tokenize(chunks[i]["text"])) for i in top) n = len(QUERIES) return recall / n, answerable / n, len(chunks), sum(len(tokenize(p["text"])) for p in chunks), tokens / n print("mode recall answerable index chunks stored tokens tokens per request") for mode in ("row", "block", "row + header", "visual", "visual + description"): a, c, np_, sb, ib = measure(mode) print(f"{mode:<21} {a:>6.4f} {c:>10.4f} {np_:>12} {sb:>13} {ib:>18.1f}") ``` ``` mode recall answerable index chunks stored tokens tokens per request row 0.0000 0.0000 693 3815 17.0 block 0.9500 0.9500 200 5702 88.7 row + header 0.9500 0.9500 693 8634 38.5 visual 0.0000 0.0000 200 0 0.0 visual + description 0.9500 0.0000 200 1194 18.7 ``` #### The Source Found, the Value Not Retrieved Row-level chunking is the cheapest mode: **17.0** tokens per request and 3815 tokens in the index. And its recall is **0.0000**. The query asks about a procedure, the rows do not contain its name, and the embedding finds no row close to the query. The loss is not a ranking fault; the surface searched for **never entered the index**. The same chunking worked on plain text, because there every sentence repeated its own subject. Block-level chunking brings recall to **0.9500** and answerability with it: the header and the rows are in the same chunk. Its cost is **88.7** tokens per request, over five times the row-level mode. Stored tokens go from 3815 to **5702**, since the headers are stored too. The row + header mode lies between the two. Recall and answerability are **identical** to the block's (0.9500), but the tokens per request fall to **38.5**: the three chunks retrieved are now three rows, not a whole table. The price is in storage. With the header repeated in every row, stored tokens go from 5702 to **8634**, against the block mode. **The two costs move in opposite directions**: the tokens per request fall while the stored tokens rise. The choice depends on the number of requests; it is a question of numbers, not of style. The last two rows show the multimodal source's own loss. When the table arrives as an image with no description, there is no text in the index: recall is **0.0000**, stored tokens are **0**. The source sits in the index, answering no query: dead weight in the retrieval space. The real warning is in the fifth row. With a one-line description, recall jumps to **0.9500** while answerability stays at **0.0000**. The description says **what the table is about**, it does not say **what is written in it**. Read with the four-cell contingency of retrieval and generation from the evaluation lesson, this is the box where the right chunk is retrieved but the answer cannot be produced; whoever reads a single recall number will think this mode **works**. #### Summary - In a table, meaning is not in the cell but in its relation to the header; row-level chunking drops the header, so recall falls to 0.0000, and the loss is in the index, not the ranking. - Block-level chunking gives 0.9500 recall at 88.7 tokens per request; row + header gives the same 0.9500 at 38.5. - Repeating the header costs storage: stored tokens go from 5702 to 8634, against the block mode; the two costs move in opposite directions. - An image source without a description carries no text in the index, its recall is 0.0000 and it is dead weight in the retrieval space. - A one-line description brings recall to 0.9500 but leaves answerability at 0.0000: the source is found, and the value inside it cannot be retrieved. #### Course Wrap-Up | Lesson | No-retrieval baseline | With retrieval | Cost | |---|---|---|---| | `embeddings/01` — first result's concept hit rate | keyword search 0.0250 | embedding similarity 0.5000 (**+0.4750**) | 8000 comparisons, 240 embedding calls, 6400 stored values | | `embeddings/02` — concept hit rate, 32 dimensions | out-of-domain embedding 0.0500, barely above the 0.0250 baseline | in-domain embedding 0.5000 (**+0.4500**) | stored values 6400 in both setups; band between two models 0.0250 | | `embeddings/03` — first result's concept hit rate | dot product 0.4250 | cosine similarity 0.5000 (**+0.0750**) | first three diverge in thirty of forty queries; average of the retrieved set 66.2 → 57.7 tokens | | `embeddings/04` — recall against exhaustive search | exhaustive search 1.0000, 200 comparisons | bucket-based index with 2 probes 0.9500 (**−0.0500**) | 47.2 comparisons; with 1 probe recall 0.7200 and concept precision 0.4700 → 0.3800 | | `embeddings/05` — filtered recall | unfiltered index 0.9500 | in-index filtering, width 1: 0.4750 (**−0.4750**) | candidates 31.2 → 2.8; exhaustive search of the filtered set 1.0000 with 18 comparisons | | `embeddings/06` — first rank in the document language class | embedding alone 0.5000 | rank fusion 0.4000 (**−0.1000**) | in the code class 1.0000 → 0.0500; recall up to the twentieth rank 0.7122 → 0.7214 | | `retrieval-pipeline/01` — generation hit rate | no-retrieval baseline 0.2000 | pipeline with retrieval 0.9250 (**+0.7250**) | 3033 → 4848 tokens, 120 chunks; run with wrong chunks 0.0500 | | `retrieval-pipeline/02` — generation hit rate | single block without preparation 0.0000, its recall 1.0000 | structure-aware preparation 0.9250 (**+0.9250**) | stored chunks 1805 → 693; the filter deletes 96 of 568 concept links | | `retrieval-pipeline/03` — share of chunks carrying a relation occurrence whole | size 8 chunking 0.0765 | size 32 chunking 0.8846 (**+0.8081**) | stored chunks 1217 → 369; at size 80 generation 0.5500, 0.15 of requests exceed the context limit | | `retrieval-pipeline/04` — up-to-date answer share | refresh once in twelve periods 0.6125 | refresh every period 0.9250 (**+0.3125**) | embedded chunks 519 → 881; stale chunk rate 0.2562 → 0.0000, wrong answer rate 0.3875 → 0.0750 | | `retrieval-pipeline/05` — retrieval recall on the short phrasing | without transformation 0.3250 | expansion 0.7500 (**+0.4250**) | one call per query, 4609 → 5804 tokens; on the long phrasing recall 0.5500 → 0.5000 | | `retrieval-pipeline/06` — precision of the top three | single-stage retrieval 0.4167 (generation 0.7000) | reranking 24 candidates 0.9000 (**+0.4833**, generation 1.0000) | calls 40 → 1000, tokens 3654 → 38003; at 48 candidates recall 1.0000 → 0.9500 | | `retrieval-pipeline/07` — generation hit rate / recall | one chunk 0.7750 / 0.7750 | ten chunks 0.5750 (**−0.2000**) / 0.9750 | 13.4 chunks fit the context window; at k=13 fifteen of forty queries break the limit | | `retrieval-pipeline/08` — end-to-end accuracy rate | no-retrieval baseline 0.3750, all of it chance | three chunks 0.9500 (**+0.5750**) | 120 chunks, 1872 tokens; same 0.9500 with five chunks, the fault carries into generation | | `retrieval-pipeline/09` — knowledge freshness | no retrieval 0.0000 | with retrieval 0.9500 (**+0.9500**) | 2232 tokens, 40 retrieval steps; on format compliance 0.0000 → 0.0000 | | `retrieval-pipeline/10` — answerability | row-level chunking 0.0000 | row + header 0.9500 (**+0.9500**) | 17.0 → 38.5 tokens per request; stored in the index 3815 → 8634 | The course's rule is written in the table row by row: **retrieval and generation are measured separately.** No number in the no-retrieval baseline column comes from a bad model; all of them come from the answer being not inside the model but **in the corpus**. Reducing the two columns to a single ratio hides the most instructive rows: in `retrieval-pipeline/02`, the unprepared source is **the best of the retrieval column** (recall 1.0000) and **the worst of the generation column** (0.0000). **The second claim can also be read from the same table: retrieval spoils too.** In `embeddings/02`, the out-of-domain embedding falls **below** the keyword baseline on the retrieved set. In `embeddings/04`, with 1 probe recall falls to 0.7200 and the concept precision of the first five from 0.4700 to 0.3800. In `embeddings/05`, under a narrow filter the index gives 0.4750 with 18.8 comparisons; exhaustive search of the filtered set gives 1.0000 with 18. In `embeddings/06`, fusion is **worse than both** in the document language class and takes 1.0000 to 0.0500 in the code class. In `retrieval-pipeline/01`, the run with wrong chunks drops generation below the no-retrieval baseline. In `retrieval-pipeline/03`, raising the chunk size to 80 carries relation occurrence to 0.9971 and drops generation to **0.5500**, because 0.15 of the requests exceed the context limit. In `retrieval-pipeline/04`, a longer refresh interval **does not move** the retrieval column; all the degradation stays in the generation column: the up-to-date answer share falls from 0.9250 to 0.6125. In `retrieval-pipeline/05`, rewriting takes the baseline recall from 0.5500 to 0.2750 on the long phrasing. In `retrieval-pipeline/06`, going from 24 to 48 candidates adds an unmeasurable 0.0083 to precision, takes recall from 1.0000 to 0.9500 and almost doubles the tokens. In `retrieval-pipeline/07`, ten chunks raise recall to 0.9750 and drop generation to **0.5750**: what is retrieved is right, what is used is wrong. In `retrieval-pipeline/08`, going up to five chunks makes recall 1.0000, **does not change** the end-to-end number and raises the tokens from 1872 to 2932. In `retrieval-pipeline/09`, retrieval adds nothing on format compliance, from 0.0000 to 0.0000. And in this lesson, the visual source with a description takes recall to 0.9500 and leaves answerability at 0.0000: a number that improved, a job that did not. Over sixteen lessons, **which information** enters the model was measured. But **we** always made this choice: how many chunks to retrieve, which criterion to rank by, whether to transform the query and how to build the context were written into the pipeline by hand. The pipeline was also **single-pass**: the query went in, the chunks came, the answer went out. Which tool the model would call by its own decision, how it would judge the result insufficient and retry, and **where** this loop would stop were never asked even once. The next course builds this loop and turns the stopping condition into a measurement object.