--- title: 'Document Preparation' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/document-preparation' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Separates the forms in which a raw source enters the retrieval pipeline: one block per document, crude line-based splitting, the repetition-based boilerplate filter and structure-aware preparation.' - 'Compares the retrieval recall, generation hit rate, stored chunk count and stored tokens of each preparation level on the same query set.' - 'Shows with the context overflow rate that an unprepared source can look best in the retrieval column and worst in the generation column.' - 'Measures the information a cleaning rule throws away by counting concept-document links, and writes that this loss is invisible in the retrieval and generation columns.' --- # Document Preparation Measuring four preparation levels on the same forty queries, starting from the raw form of the same fictional corpus: the source left as one block per document pushes retrieval recall to the highest value, 1.0000, but every request overflows the context window and generation falls to 0.0000, so had a single column been printed the worst setup would have won. Crude line-based splitting stores 1805 chunks and gives 0.9250 on generation, the same as structure-aware preparation with 693 chunks, which reproduces exactly where the previous lesson started; what separates them is the stored vector count, not the hit rate. The repetition-based boilerplate filter drops 798 lines and 7021 tokens while destroying 96 of 568 concept-document links and erasing the evidence of the main concept in eight documents; when the threshold is lowered to two, the loss rises to 283 links and sixty documents. The retrieval and generation columns show none of this loss. The previous lesson found the corpus ready: two hundred documents, six hundred ninety-three clean paragraphs, code and class fields kept apart. That preparation did not happen by itself. A field team's records carry a form header, a code line, sentences split in the middle of a line and a page footer; all of it enters the retrieval pipeline. This lesson asks a single question: **what does preparation add to retrieval recall, and what does the preparation itself throw away.** The second half matters as much as the first, because cleaning is an act of selection and every line not selected is deleted from the corpus. The **procedure** of parsing, cleaning and metadata extraction was measured in the Data Collection and Preparation course and is **not repeated** here; the only thing added here is that procedure's counterpart in the retrieval pipeline. - **RP8.** The code is a **simulator**; the corpus, the raw source format and the queries are **fictional**, seed **20260218**. The embedding and the language model are the definitions of the previous lesson. - **RP9.** The raw source carries four leftovers: a **form header** repeated in every document, a **code and class field** embedded in the text (`document code SY003 class meter`), a split in the middle of a line in **0.45** of the paragraphs, and a **page footer** repeated in every document. The continuation line is indented by two spaces. - **RP10.** The split point falls after the first, fifth or seventh word of the paragraph. The first and seventh points **cut a document surface in the middle** and make that surface invisible to search; the fifth point leaves both surfaces intact. - **RP11.** The **unsegmented raw block** leaves one chunk per document. **Line-based splitting** counts every raw line as a separate chunk. The **repetition-based boilerplate filter** drops a line that appears in more than three documents. **Structure-aware preparation** merges the continuation lines, drops the known header and footer, and takes the code and class field out of the text and writes it into a field. - **RP12.** The measures are the definitions of the previous lesson and the top three chunks are retrieved; the threshold is **0.0750**. The cost has two units: the **stored chunk count** (each chunk holds one vector, thirty-two numbers per vector) and the **stored tokens**. - **RP13.** A **concept-document link** is the occurrence of a concept's document surface in a document's text. This number is **independent** of the query set and of the run. ## The State of the Raw Source The first block sets up the fictional corpus, applies the raw source format on top of it and prints the raw form of one document. The printed document has three splits, and all three are at different points. ```python # FICTIONAL. The corpus, concept axis, embedding and raw source format are a simulator; # no real embedding endpoint is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # ---- FICTIONAL raw source: form header, code field, split paragraphs, page footer HEADER, FOOTER = "regional directorate field record sheet", "for internal use only do not distribute" RAW = [] for i, d in enumerate(DOCUMENTS): r = rng(SEED + 3011 * i) s = [HEADER, f"document code {d['code']} class {d['label']}", d["title"]] for p in d["paras"]: w = p.split() if r() < 0.45: k = [1, 5, 7][pick(r(), [0.4, 0.25, 0.35])] s += [" ".join(w[:k]), " " + " ".join(w[k:])] # continuation line indented by two spaces else: s.append(p) RAW.append(s + [FOOTER]) j = next(i for i, s in enumerate(RAW) if sum(x.startswith(" ") for x in s) >= 2 and len(s) <= 10) print("raw source %s, %d lines:" % (DOCUMENTS[j]["code"], len(RAW[j]))) for x in RAW[j]: print(" |" + x) print("total raw lines %d, ready paragraphs %d" % (sum(len(s) for s in RAW), sum(len(d["paras"]) for d in DOCUMENTS))) ``` ``` raw source SY003, 10 lines: |regional directorate field record sheet |document code SY003 class meter |reading routine procedure |reading check for note goes | with reading routine and is filed |meter swap for form goes with reading | routine and is filed |reading | routine for task goes with reading routine and is filed |for internal use only do not distribute total raw lines 1805, ready paragraphs 693 ``` The eighth line is a single word: `reading`. Its continuation is one line below. In this document the surface `reading routine` appears four times, but because the line boundary cuts the surface in the middle, two of those occurrences count as **absent** on the search side. The sixth line does the same thing. The split is not a loss; it is a loss **unless preparation is done**. ## Four Preparation Levels The second block sets up the language model simulator, produces the four preparation levels and measures all four on the same forty queries. Because the instruction forces a one-concept answer, the model writes the label directly. ```python import math # ---- FICTIONAL language model simulator: the previous lesson's simulator, response space is twenty concepts WINDOW = 240 PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"} COUNTER = [0] def trace_id(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h def call(prompt, seed=SEED, temperature=0.6): COUNTER[0] += 1 b = tokenize(prompt) if len(b) > WINDOW: return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""} if COUNTER[0] % 37 == 0: return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""} r = rng(seed ^ trace_id(prompt)) dm = " " + prompt.lower() + " " p = {k: 0.15 for k in KEYS} for k in KEYS: p[k] += 2.0 * dm.count(CONCEPTS[k][1]) # the model recognizes the document language if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm: p[k] += 2.0 w = [math.exp(p[a] / temperature) for a in sorted(p)] y = sorted(p)[pick(r(), [x / sum(w) for x in w])] return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y} INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) + " . Reply with one concept . ") def merge_lines(s): c = [] for x in s: if x.startswith(" ") and c: c[-1] += " " + x.strip() # the continuation line is appended to the previous one else: c.append(x) return c COUNTS = {} for s in RAW: for x in set(merge_lines(s)): COUNTS[x] = COUNTS.get(x, 0) + 1 def prepare(mode, threshold=3): p, dropped = [], [] for i, s in enumerate(RAW): if mode == "raw block": c = [" ".join(s)] elif mode == "line-based": c = [x.strip() for x in s] elif mode == "repeat filter": c = [x for x in merge_lines(s) if COUNTS[x] < threshold] dropped += [x for x in merge_lines(s) if COUNTS[x] >= threshold] else: c = [x for x in merge_lines(s) if x not in (HEADER, FOOTER, DOCUMENTS[i]["title"]) and not x.startswith("document code")] p += [{"code": DOCUMENTS[i]["code"], "label": DOCUMENTS[i]["label"], "text": x, "concepts": [k for k in KEYS if CONCEPTS[k][1] in x]} for x in c] return p, dropped def measure(p, k=3): COUNTER[0] = 0 v = [embed(x["text"]) for x in p] a = [0] * 6 for s in QUERIES: q = embed(s["text"]) top = sorted(range(len(p)), key=lambda i: -cosine(q, v[i]))[:k] h = [s["concept"] in p[i]["concepts"] for i in top] a[0] += any(h) a[1] += sum(h) / k a[2] += sum(p[i]["label"] == s["label"] for i in top) / k y = call(INSTRUCTION + "Source: " + " . ".join(p[i]["text"] for i in top) + " . Request: " + s["text"]) a[3] += y["status"] == "context_overflow" a[4] += y["input"] + y["output"] a[5] += y["response"] == s["concept"] return [x / len(QUERIES) for x in a[:4]] + [a[4], a[5] / len(QUERIES)] print("preparation chunks stored recall precision label generation overflow tokens") for mode in ("raw block", "line-based", "repeat filter", "structure-aware"): p, at = prepare(mode) m = measure(p) print("%-16s%6d %8d %6.4f %9.4f %6.4f %10.4f %8.2f %8d" % (mode, len(p), sum(len(tokenize(x["text"])) for x in p), m[0], m[1], m[2], m[5], m[3], m[4])) ``` ``` preparation chunks stored recall precision label generation overflow tokens raw block 200 14925 1.0000 0.9667 0.9667 0.0000 1.00 11839 line-based 1805 14925 0.9500 0.9500 0.9500 0.9250 0.00 4152 repeat filter 695 7904 0.9500 0.9500 0.9500 0.9250 0.00 4850 structure-aware 693 8985 0.9500 0.9500 0.9500 0.9250 0.00 4848 ``` The first row is the most important number of this lesson. The unprepared source is **the best of the retrieval column**: recall **1.0000**, precision and class hit rate **0.9667**. The reason is clear: when one chunk is kept per document, that chunk carries every concept in the document, and it becomes harder to miss the target. The generation hit rate on the same row is **0.0000**. The context overflow rate is **1.00**: all forty requests failed to fit the window, because three documents at once were put into the prompt. **Had a single column been printed, this setup would have won.** Line-based splitting is the other extreme. Recall falls to **0.9500**, because the split surfaces can no longer be searched; on the other hand, the tokens per request are at their lowest (**4152**), since the retrieved lines are short. The cost is paid in the stored chunk count: **1805** vectors, that is, **2.6 times** that of structure-aware preparation, and with the stored tokens staying the same. Generation is **0.9250**, the same as structure-aware preparation's **0.9250**: there is no generation difference to measure between the two. The measured difference between these two setups is not in the hit rate but in the **stored vector count**. The last row is where the previous lesson started: 693 chunks, 0.9500 recall, 0.9250 generation, 4848 tokens. The output of structure-aware preparation is the input of the first lesson. ## What Preparation Throws Away The third row looks cheap: it lowers the stored tokens **from 8985 to 7904** and moves no column of the hit rate. The repetition-based boilerplate filter rests on a guess: if the same text appears in many documents, it is leftover. That is true for the form header and the page footer. Because record language is **formulaic**, real paragraphs repeat too. ```python def links(p): return {(x["code"], k) for x in p for k in x["concepts"]} full = links(prepare("structure-aware")[0]) print("concept-document links built by structure-aware preparation:", len(full)) print("threshold chunks dropped lines dropped tokens lost links no-main docs recall generation") for e in (2, 3, 4): p, at = prepare("repeat filter", e) b = links(p) m = measure(p) print("%9d %6d %13d %16d %13d %13d %6.4f %10.4f" % (e, len(p), len(at), sum(len(tokenize(x)) for x in at), len(full - b), sum((d["code"], d["main"]) not in b for d in DOCUMENTS), m[0], m[5])) ``` ``` concept-document links built by structure-aware preparation: 568 threshold chunks dropped lines dropped tokens lost links no-main docs recall generation 2 427 1066 10525 283 60 0.9500 0.9250 3 695 798 7021 96 8 0.9500 0.9250 4 835 658 5282 17 0 0.9500 0.9250 ``` At all three thresholds recall is **0.9500** and generation **0.9250**. The query set sees nothing. Yet at threshold three the filter drops **798 lines** and **7021 tokens**; **96** of the **568** concept-document links the corpus built vanish, and in **eight documents** no evidence of that document's own main concept remains. When the threshold is lowered to two, the loss rises to **283 links** and **sixty documents**, two thirds of the stored tokens are gone, and the two measure columns still do not budge. The real lesson here is about measurement. Forty queries do not sample the twenty concepts of two hundred documents; the deletion of the evidence for a concept nobody asked about is **invisible** in this measurement. The number of concept-document links, on the other hand, is independent of the run and of the query set: it says how much of the corpus can still be searched, without running a single query. This is the number to look at when choosing a cleaning rule. ## Summary - Preparation's counterpart in the retrieval pipeline is two separate questions: **what it adds to recall** and **what it throws away**. The second may be invisible in the retrieval and generation columns. - The unprepared source is the best of the retrieval column (recall 1.0000) and the worst of the generation column (0.0000); all forty requests exceed the context window. - Crude line-based splitting stores 1805 vectors, structure-aware preparation 693; the generation is the same, 0.9250, in both. The measured gain is not in the hit rate but in the stored vector count. - Structure-aware preparation gives 693 chunks, 0.9500 recall and 0.9250 generation; this is where the previous lesson started. - At threshold three the repetition-based boilerplate filter drops 798 lines and 7021 tokens while deleting 96 of 568 concept-document links; at threshold two the loss is 283 links and sixty documents. - The number of concept-document links is a measure independent of the query set, and it is the real criterion for a cleaning decision. ## Next Step Structure-aware preparation accepted one boundary as **given**: the source's own paragraph boundary. The continuation lines were merged and the header and footer dropped, but where a paragraph ends was left as the source's decision. That boundary is a decision, and it changes everything the retrieval pipeline measures. As the chunk shrinks, precision rises and surfaces on the boundary are lost; as it grows, recall rises and the context window overflows. The next lesson takes that boundary in hand: it sweeps chunk size and overlap, prints each setup's recall, stored chunk count and stored tokens side by side, and counts the point where what overlap buys stops paying for its cost.