Document Preparation

The previous lesson found the corpus ready: two hundred documents, six hundred ninety-three clean paragraphs, code and class fields kept apart. That preparation did not happen by itself. A field team’s records carry a form header, a code line, sentences split in the middle of a line and a page footer; all of it enters the retrieval pipeline.

This lesson asks a single question: what does preparation add to retrieval recall, and what does the preparation itself throw away. The second half matters as much as the first, because cleaning is an act of selection and every line not selected is deleted from the corpus. The procedure of parsing, cleaning and metadata extraction was measured in the Data Collection and Preparation course and is not repeated here; the only thing added here is that procedure’s counterpart in the retrieval pipeline.

• RP8. The code is a simulator; the corpus, the raw source format and the queries are fictional, seed 20260218. The embedding and the language model are the definitions of the previous lesson.
• RP9. The raw source carries four leftovers: a form header repeated in every document, a code and class field embedded in the text (document code SY003 class meter), a split in the middle of a line in 0.45 of the paragraphs, and a page footer repeated in every document. The continuation line is indented by two spaces.
• RP10. The split point falls after the first, fifth or seventh word of the paragraph. The first and seventh points cut a document surface in the middle and make that surface invisible to search; the fifth point leaves both surfaces intact.
• RP11. The unsegmented raw block leaves one chunk per document. Line-based splitting counts every raw line as a separate chunk. The repetition-based boilerplate filter drops a line that appears in more than three documents. Structure-aware preparation merges the continuation lines, drops the known header and footer, and takes the code and class field out of the text and writes it into a field.
• RP12. The measures are the definitions of the previous lesson and the top three chunks are retrieved; the threshold is 0.0750. The cost has two units: the stored chunk count (each chunk holds one vector, thirty-two numbers per vector) and the stored tokens.
• RP13. A concept-document link is the occurrence of a concept’s document surface in a document’s text. This number is independent of the query set and of the run.

The State of the Raw Source

The first block sets up the fictional corpus, applies the raw source format on top of it and prints the raw form of one document. The printed document has three splits, and all three are at different points.

# FICTIONAL. The corpus, concept axis, embedding and raw source format are a simulator;
# no real embedding endpoint is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# ---- FICTIONAL raw source: form header, code field, split paragraphs, page footer
HEADER, FOOTER = "regional directorate field record sheet", "for internal use only do not distribute"
RAW = []
for i, d in enumerate(DOCUMENTS):
    r = rng(SEED + 3011 * i)
    s = [HEADER, f"document code {d['code']} class {d['label']}", d["title"]]
    for p in d["paras"]:
        w = p.split()
        if r() < 0.45:
            k = [1, 5, 7][pick(r(), [0.4, 0.25, 0.35])]
            s += [" ".join(w[:k]), "  " + " ".join(w[k:])]   # continuation line indented by two spaces
        else:
            s.append(p)
    RAW.append(s + [FOOTER])

j = next(i for i, s in enumerate(RAW)
         if sum(x.startswith("  ") for x in s) >= 2 and len(s) <= 10)
print("raw source %s, %d lines:" % (DOCUMENTS[j]["code"], len(RAW[j])))
for x in RAW[j]:
    print("  |" + x)
print("total raw lines %d, ready paragraphs %d"
      % (sum(len(s) for s in RAW), sum(len(d["paras"]) for d in DOCUMENTS)))

raw source SY003, 10 lines:
  |regional directorate field record sheet
  |document code SY003 class meter
  |reading routine procedure
  |reading check for note goes
  |  with reading routine and is filed
  |meter swap for form goes with reading
  |  routine and is filed
  |reading
  |  routine for task goes with reading routine and is filed
  |for internal use only do not distribute
total raw lines 1805, ready paragraphs 693

The eighth line is a single word: reading. Its continuation is one line below. In this document the surface reading routine appears four times, but because the line boundary cuts the surface in the middle, two of those occurrences count as absent on the search side. The sixth line does the same thing. The split is not a loss; it is a loss unless preparation is done.

Four Preparation Levels

The second block sets up the language model simulator, produces the four preparation levels and measures all four on the same forty queries. Because the instruction forces a one-concept answer, the model writes the label directly.

import math
# ---- FICTIONAL language model simulator: the previous lesson's simulator, response space is twenty concepts
WINDOW = 240
PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"}
COUNTER = [0]


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def call(prompt, seed=SEED, temperature=0.6):
    COUNTER[0] += 1
    b = tokenize(prompt)
    if len(b) > WINDOW:
        return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    dm = " " + prompt.lower() + " "
    p = {k: 0.15 for k in KEYS}
    for k in KEYS:
        p[k] += 2.0 * dm.count(CONCEPTS[k][1])           # the model recognizes the document language
        if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm:
            p[k] += 2.0
    w = [math.exp(p[a] / temperature) for a in sorted(p)]
    y = sorted(p)[pick(r(), [x / sum(w) for x in w])]
    return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y}


INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) +
               " . Reply with one concept . ")


def merge_lines(s):
    c = []
    for x in s:
        if x.startswith("  ") and c:
            c[-1] += " " + x.strip()                     # the continuation line is appended to the previous one
        else:
            c.append(x)
    return c


COUNTS = {}
for s in RAW:
    for x in set(merge_lines(s)):
        COUNTS[x] = COUNTS.get(x, 0) + 1


def prepare(mode, threshold=3):
    p, dropped = [], []
    for i, s in enumerate(RAW):
        if mode == "raw block":
            c = [" ".join(s)]
        elif mode == "line-based":
            c = [x.strip() for x in s]
        elif mode == "repeat filter":
            c = [x for x in merge_lines(s) if COUNTS[x] < threshold]
            dropped += [x for x in merge_lines(s) if COUNTS[x] >= threshold]
        else:
            c = [x for x in merge_lines(s) if x not in (HEADER, FOOTER, DOCUMENTS[i]["title"])
                 and not x.startswith("document code")]
        p += [{"code": DOCUMENTS[i]["code"], "label": DOCUMENTS[i]["label"], "text": x,
               "concepts": [k for k in KEYS if CONCEPTS[k][1] in x]} for x in c]
    return p, dropped


def measure(p, k=3):
    COUNTER[0] = 0
    v = [embed(x["text"]) for x in p]
    a = [0] * 6
    for s in QUERIES:
        q = embed(s["text"])
        top = sorted(range(len(p)), key=lambda i: -cosine(q, v[i]))[:k]
        h = [s["concept"] in p[i]["concepts"] for i in top]
        a[0] += any(h)
        a[1] += sum(h) / k
        a[2] += sum(p[i]["label"] == s["label"] for i in top) / k
        y = call(INSTRUCTION + "Source: " + " . ".join(p[i]["text"] for i in top) +
                 " . Request: " + s["text"])
        a[3] += y["status"] == "context_overflow"
        a[4] += y["input"] + y["output"]
        a[5] += y["response"] == s["concept"]
    return [x / len(QUERIES) for x in a[:4]] + [a[4], a[5] / len(QUERIES)]


print("preparation     chunks   stored recall precision  label generation overflow   tokens")
for mode in ("raw block", "line-based", "repeat filter", "structure-aware"):
    p, at = prepare(mode)
    m = measure(p)
    print("%-16s%6d %8d %6.4f %9.4f %6.4f %10.4f %8.2f %8d"
          % (mode, len(p), sum(len(tokenize(x["text"])) for x in p),
             m[0], m[1], m[2], m[5], m[3], m[4]))

preparation     chunks   stored recall precision  label generation overflow   tokens
raw block          200    14925 1.0000    0.9667 0.9667     0.0000     1.00    11839
line-based        1805    14925 0.9500    0.9500 0.9500     0.9250     0.00     4152
repeat filter      695     7904 0.9500    0.9500 0.9500     0.9250     0.00     4850
structure-aware    693     8985 0.9500    0.9500 0.9500     0.9250     0.00     4848

The first row is the most important number of this lesson. The unprepared source is the best of the retrieval column: recall 1.0000, precision and class hit rate 0.9667. The reason is clear: when one chunk is kept per document, that chunk carries every concept in the document, and it becomes harder to miss the target. The generation hit rate on the same row is 0.0000. The context overflow rate is 1.00: all forty requests failed to fit the window, because three documents at once were put into the prompt. Had a single column been printed, this setup would have won.

Line-based splitting is the other extreme. Recall falls to 0.9500, because the split surfaces can no longer be searched; on the other hand, the tokens per request are at their lowest (4152), since the retrieved lines are short. The cost is paid in the stored chunk count: 1805 vectors, that is, 2.6 times that of structure-aware preparation, and with the stored tokens staying the same. Generation is 0.9250, the same as structure-aware preparation’s 0.9250: there is no generation difference to measure between the two. The measured difference between these two setups is not in the hit rate but in the stored vector count.

The last row is where the previous lesson started: 693 chunks, 0.9500 recall, 0.9250 generation, 4848 tokens. The output of structure-aware preparation is the input of the first lesson.

What Preparation Throws Away

The third row looks cheap: it lowers the stored tokens from 8985 to 7904 and moves no column of the hit rate. The repetition-based boilerplate filter rests on a guess: if the same text appears in many documents, it is leftover. That is true for the form header and the page footer. Because record language is formulaic, real paragraphs repeat too.

def links(p):
    return {(x["code"], k) for x in p for k in x["concepts"]}


full = links(prepare("structure-aware")[0])
print("concept-document links built by structure-aware preparation:", len(full))
print("threshold  chunks  dropped lines  dropped tokens  lost links  no-main docs  recall  generation")
for e in (2, 3, 4):
    p, at = prepare("repeat filter", e)
    b = links(p)
    m = measure(p)
    print("%9d %6d %13d %16d %13d %13d  %6.4f  %10.4f"
          % (e, len(p), len(at), sum(len(tokenize(x)) for x in at), len(full - b),
             sum((d["code"], d["main"]) not in b for d in DOCUMENTS), m[0], m[5]))

concept-document links built by structure-aware preparation: 568
threshold  chunks  dropped lines  dropped tokens  lost links  no-main docs  recall  generation
        2    427          1066            10525           283            60  0.9500      0.9250
        3    695           798             7021            96             8  0.9500      0.9250
        4    835           658             5282            17             0  0.9500      0.9250

At all three thresholds recall is 0.9500 and generation 0.9250. The query set sees nothing. Yet at threshold three the filter drops 798 lines and 7021 tokens; 96 of the 568 concept-document links the corpus built vanish, and in eight documents no evidence of that document’s own main concept remains. When the threshold is lowered to two, the loss rises to 283 links and sixty documents, two thirds of the stored tokens are gone, and the two measure columns still do not budge.

The real lesson here is about measurement. Forty queries do not sample the twenty concepts of two hundred documents; the deletion of the evidence for a concept nobody asked about is invisible in this measurement. The number of concept-document links, on the other hand, is independent of the run and of the query set: it says how much of the corpus can still be searched, without running a single query. This is the number to look at when choosing a cleaning rule.

Summary

• Preparation’s counterpart in the retrieval pipeline is two separate questions: what it adds to recall and what it throws away. The second may be invisible in the retrieval and generation columns.
• The unprepared source is the best of the retrieval column (recall 1.0000) and the worst of the generation column (0.0000); all forty requests exceed the context window.
• Crude line-based splitting stores 1805 vectors, structure-aware preparation 693; the generation is the same, 0.9250, in both. The measured gain is not in the hit rate but in the stored vector count.
• Structure-aware preparation gives 693 chunks, 0.9500 recall and 0.9250 generation; this is where the previous lesson started.
• At threshold three the repetition-based boilerplate filter drops 798 lines and 7021 tokens while deleting 96 of 568 concept-document links; at threshold two the loss is 283 links and sixty documents.
• The number of concept-document links is a measure independent of the query set, and it is the real criterion for a cleaning decision.

Next Step

Structure-aware preparation accepted one boundary as given: the source’s own paragraph boundary. The continuation lines were merged and the header and footer dropped, but where a paragraph ends was left as the source’s decision. That boundary is a decision, and it changes everything the retrieval pipeline measures. As the chunk shrinks, precision rises and surfaces on the boundary are lost; as it grows, recall rises and the context window overflows. The next lesson takes that boundary in hand: it sweeps chunk size and overlap, prints each setup’s recall, stored chunk count and stored tokens side by side, and counts the point where what overlap buys stops paying for its cost.