Chunking Strategies

The previous lesson carried preparation up to the boundary and stopped there: continuation lines were merged, the header and footer dropped, the code and class field separated, but where a paragraph ends was left as the source’s decision. In the retrieval pipeline that boundary is not the source’s decision; it is the decision of whoever builds the pipeline. A chunk is both the unit that is indexed and the piece of text handed to the model; its size determines both ends at once.

What this lesson measures is the price of the boundary. As the chunk shrinks, the stored vector count rises and the relations inside the text are cut; as it grows, the context window fills. Overlap takes back the cut relation and pays for what it takes back in tokens. Where these three cross is a number that can be measured.

• RP14. The code is a simulator; the corpus, the embedding and the queries are fictional, seed 20260218. The embedding and the language model are the definitions of the previous lessons.
• RP15. Chunk boundaries are drawn on words, and chunk size is given in tokens: the window grows by adding words until the token budget is full. No split is made in the middle of a token, because a split token matches no surface in search.
• RP16. Overlap is also in tokens: the next window starts by re-taking the last overlap-many tokens of the previous one.
• RP17. A relation occurrence is the span in a paragraph that carries two concept surfaces together, and it is the source’s real unit of information: which record is kept together with which procedure. The corpus has 693 relation occurrences.
• RP18. Relation recall is the share of these occurrences that stay whole inside a single chunk. It is independent of the query set and of the run; it says what the chunking decision did to the source without running a single query.
• RP19. Retrieval recall, generation hit rate and context overflow rate are the definitions of the previous lessons; the top three chunks are retrieved and the threshold is 0.0750.
• RP20. The stored tokens are the token total of all chunks. In a split without overlap this number equals the size of the source itself (8985); how many times overlap multiplies it is written alongside.

Taking the Boundary in Hand

The first block sets up the corpus, puts a token-budgeted chunker on top of it and marks the relation occurrences. The language model simulator follows, and the measurement function is built.

# FICTIONAL. The corpus, concept axis, embedding and language model are a simulator; no real
# endpoint is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# ---- chunking: chunk boundaries fall on words, size is measured in tokens
BODY = [" ".join(d["paras"]).split() for d in DOCUMENTS]
RELATIONS = [(i, 11 * t, 11 * t + 8)              # the span carrying both surfaces of a paragraph together
             for i, d in enumerate(DOCUMENTS) for t in range(len(d["paras"]))]


def windows(w, size, overlap):
    tc = [len(tokenize(x)) for x in w]
    spans, i = [], 0
    while True:
        j, t = i, 0
        while j < len(w) and t + tc[j] <= size:
            t += tc[j]
            j += 1
        j = max(j, i + 1)
        spans.append((i, j))
        if j >= len(w):
            return spans
        new, g = j, 0
        while new > i + 1 and g + tc[new - 1] <= overlap:      # rewinds by the overlap
            new -= 1
            g += tc[new]
        i = new


def split(size, overlap, structure=False):
    p, spans = [], {}
    for i, d in enumerate(DOCUMENTS):
        w = BODY[i]
        spans[i] = ([(11 * t, 11 * t + 11) for t in range(len(d["paras"]))] if structure
                    else windows(w, size, overlap))
        for a, b in spans[i]:
            m = " ".join(w[a:b])
            p.append({"code": d["code"], "label": d["label"], "text": m,
                      "concepts": [k for k in KEYS if CONCEPTS[k][1] in m]})
    kept = sum(any(a <= s and e <= b for a, b in spans[i]) for i, s, e in RELATIONS)
    return p, kept / len(RELATIONS)


import math
# ---- FICTIONAL language model simulator: the previous lessons' simulator
WINDOW = 240
PRIOR_KNOWLEDGE = {"leak", "pressure", "duration", "swap", "valve", "notice", "switch"}
COUNTER = [0]


def trace_id(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


def call(prompt, seed=SEED, temperature=0.6):
    COUNTER[0] += 1
    b = tokenize(prompt)
    if len(b) > WINDOW:
        return {"status": "context_overflow", "input": len(b), "output": 0, "response": ""}
    if COUNTER[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(b), "output": 0, "response": ""}
    r = rng(seed ^ trace_id(prompt))
    dm = " " + prompt.lower() + " "
    p = {k: 0.15 for k in KEYS}
    for k in KEYS:
        p[k] += 2.0 * dm.count(CONCEPTS[k][1])
        if k in PRIOR_KNOWLEDGE and CONCEPTS[k][2] in dm:
            p[k] += 2.0
    w = [math.exp(p[a] / temperature) for a in sorted(p)]
    y = sorted(p)[pick(r(), [x / sum(w) for x in w])]
    return {"status": "ok", "input": len(b), "output": len(tokenize(y)), "response": y}


INSTRUCTION = ("Link the request to one of these concepts: " + " , ".join(KEYS) +
               " . Reply with one concept . ")


def measure(p, k=3):
    COUNTER[0] = 0
    v = [embed(x["text"]) for x in p]
    a = [0] * 4
    for s in QUERIES:
        q = embed(s["text"])
        top = sorted(range(len(p)), key=lambda i: -cosine(q, v[i]))[:k]
        a[0] += any(s["concept"] in p[i]["concepts"] for i in top)
        y = call(INSTRUCTION + "Source: " + " . ".join(p[i]["text"] for i in top) +
                 " . Request: " + s["text"])
        a[1] += y["response"] == s["concept"]
        a[2] += y["input"] + y["output"]
        a[3] += y["status"] == "context_overflow"
    return (sum(len(tokenize(x["text"])) for x in p), a[0] / 40, a[1] / 40, a[2],
            a[3] / 40)


print(" size chunks   stored relation  recall generation request tokens overflow")
for size in (8, 12, 20, 32, 48, 80):
    p, rel = split(size, 0)
    m = measure(p)
    print("%5d %6d %8d %8.4f %7.4f %10.4f %14d %8.2f"
          % (size, len(p), m[0], rel, m[1], m[2], m[3], m[4]))

 size chunks   stored relation  recall generation request tokens overflow
    8   1217     8985   0.0765  0.9500     0.9250           4164     0.00
   12    838     8985   0.5440  0.9500     0.9250           4576     0.00
   20    555     8985   0.6392  0.9500     0.9250           4913     0.00
   32    369     8985   0.8846  1.0000     0.9000           5582     0.00
   48    276     8985   0.9351  1.0000     0.8500           5860     0.00
   80    203     8985   0.9971  1.0000     0.5500           8271     0.15

The size sweep shows two separate degradations at two separate ends. Eight-token chunks keep almost none of the source’s 693 relation occurrences whole (0.0765), because a relation takes about twelve tokens and rarely fits in the window. These chunks do not lower retrieval recall, since they preserve the concept surfaces: 0.9500. What is cut is not the concept but the tie between concepts.

The other end breaks visibly. Eighty-token chunks keep 0.9971 of the relations and retrieval recall rises to 1.0000, but when three chunks are put into the prompt, 0.15 of the forty requests exceed the context window. Generation falls from 0.9250 to 0.5500. The drop is far larger than the threshold of 0.0750: it is a measured degradation. The row where the retrieval column is best is the row where the generation column is worst. At forty-eight tokens generation is 0.8500, and its gap to the 0.9250 of the smaller chunks is no larger than the threshold: whether this row lowers the hit rate cannot be known from this measurement.

The usable band is in the middle: between twenty and thirty-two tokens there is no context overflow, retrieval recall is at its highest level and relation recall runs between 0.6392 and 0.8846. The way to lift this band is not to enlarge the size; it is overlap.

The Price of Overlap

The second block sweeps overlap from zero to twelve in the twenty-token chunk and writes, at each step, how many relations were gained and how many tokens were stored in return. The last row uses the source’s own paragraph boundary instead of overlap.

print("overlap chunks   stored  fold relation   gain extra tokens per relation  recall generation")
SOURCE_TOKENS = sum(len(tokenize(x["text"])) for x in split(0, 0, structure=True)[0])
prev = None
for overlap in (0, 2, 4, 6, 8, 10, 12):
    p, rel = split(20, overlap)
    m = measure(p)
    extra = ""
    if prev:
        k = round((rel - prev[0]) * len(RELATIONS))
        extra = "%7d %12d %12d" % (k, m[0] - prev[1], (m[0] - prev[1]) // k)
    print("%7d %6d %8d %5.2f %8.4f%s %7.4f %10.4f"
          % (overlap, len(p), m[0], m[0] / SOURCE_TOKENS, rel, extra or " " * 33, m[1], m[2]))
    prev = (rel, m[0])
p, rel = split(0, 0, structure=True)
m = measure(p)
print("structure-aware boundary: chunks %d, stored %d, relation %.4f, recall %.4f, generation %.4f"
      % (len(p), m[0], rel, m[1], m[2]))

overlap chunks   stored  fold relation   gain extra tokens per relation  recall generation
      0    555     8985  1.00   0.6392                                   0.9500     0.9250
      2    587     9697  1.08   0.6335     -4          712         -178  0.9500     0.9250
      4    605    10517  1.17   0.7157     57          820           14  0.9500     0.9250
      6    640    11538  1.28   0.8730    109         1021            9  0.9500     0.9250
      8    718    13025  1.45   0.9639     63         1487           23  0.9500     0.9250
     10    790    14825  1.65   0.9827     13         1800          138  0.9500     0.9250
     12    897    17206  1.91   1.0000     12         2381          198  0.9500     0.9250
structure-aware boundary: chunks 693, stored 8985, relation 1.0000, recall 0.9500, generation 0.9250

Relation recall rises with overlap, from 0.6392 to 1.0000, though not at every step: the first step, from zero to two, gains nothing. Relation recall slips to 0.6335 (the gain column reads -4) while 712 tokens are added, because shifting the windows moves boundaries onto relations that were whole. The cost does not rise in a steady way; it multiplies: the stored tokens go from 8985 to 17206 (1.91 times), the stored chunks from 555 to 897. Because every chunk holds one vector, the stored vector count grows by a comparable ratio.

The crossover point is read in the last column. The three steps after the first are cheap: for every relation gained, 14, 9 and 23 tokens are stored respectively. At the fifth step, as overlap goes from eight to ten, the cost per relation jumps to 138 tokens. It does not recover afterward either: 198. Overlap up to eight rescues most of the relations that were cut (relation recall 0.9639); the rest are the few occurrences that happen to fall on a window boundary in the most unlucky way, and rescuing them requires storing the whole corpus again. Overlap is eight tokens; on this workload, what comes after that buys not relations but copies.

Looking at the retrieval recall and generation columns, the whole sweep is flat: recall is 0.9500 in every row, and generation is 0.9250 in all seven rows. What overlap adds to the hit rate has not been measured on this query set; what it adds to the stored tokens has been measured. A mechanism whose cost is not written down counts as unmeasured, and the only thing measured in these rows is the cost.

The last row closes the argument. Splitting along the source’s own paragraph boundary makes relation recall 1.0000: the place overlap twelve reaches. In return it stores 693 chunks and 8985 tokens, that is, roughly half as much as the setup with overlap. Overlap is the insurance paid when the structure of the source is unknown; when the structure is known, its price is wasted.

Summary

• The chunk boundary is the decision of whoever builds the pipeline, not of the source; size determines both the stored vector count and the text that enters the prompt.
• Relation recall measures, without running a query, how many of the source’s units of information stayed whole in a chunk, and it is the real criterion of the chunking decision.
• A small chunk cuts the relation but keeps the concept surface: at eight tokens relation recall is 0.0765 while retrieval recall is 0.9500.
• A large chunk keeps the relation but fills the window: at eighty tokens 0.15 of the requests overflow the context and generation falls from 0.9250 to 0.5500.
• Overlap lifts relation recall from 0.6392 to 1.0000, takes the stored tokens to 1.91 times, and the cost per relation jumps from 23 to 138 after eight: the crossover point is overlap eight.
• Structure-aware splitting gives the same 1.0000 relation recall with roughly half the space of the overlap.

Next Step

In three lessons the corpus was taken as static. The documents were prepared once, chunked once, embedded once, and the index was never touched again. Field records do not behave that way: a procedure is updated, a threshold changes, a record is withdrawn. The index does not learn of it at that moment but at the next refresh, and in the time that passes in between it keeps bringing the model the right chunk, but the information inside the chunk is no longer valid. The next lesson sweeps the refresh interval: it measures how the stale chunk rate, the wrong answer rate and the number of chunks re-embedded in a refresh change as the interval grows, and shows that the retrieval column shows none of this degradation.