Chunking Sources Containing Tables and Images

The previous lesson measured retrieval and fine-tuning side by side on three tasks and showed both rest on the same assumption: that everything in the corpus is plain text. The fictional documents were built from sentences, and each chunk said what it was about by itself.

The procedure documents of the field team are not like that. They contain tables, and in a table meaning does not live in the cell but in its relation to the header. When a table is split like a sentence, row and header are separated and the row says nothing by itself. In sources that carry images the loss goes one step further: no text exists at all. Multimodal input itself was measured in the Natural Language Processing and Computer Vision course, under computer vision, and the token cost of the input format was counted in the Building Applications with Model APIs course: the fine grid gave a hit rate of 0.8167, the cropped grid 0.7000, and the tokens per usable record fell from 224.1 to 60.0. Those procedures are not repeated here. This lesson measures only the chunking decision.

• RP84. The corpus, the concept axis, the embedding, the query set and the table source are fictional and are produced inside the lesson. There is no real institution, product, document, parser or embedding endpoint.
• RP85. This lesson turns the same two hundred documents into tables. The procedure name stays only in the header; the rows carry the field name, the requirement and the source columns. What the table is about is not written in the cells.
• RP86. Five chunking modes are measured: row (each row is a chunk, the header is dropped), block (the whole table is one chunk), row + header (each row is stored together with the header), visual (the table arrives as an image, there is no text) and visual + description (the image has a one-line description).
• RP87. Two measures are counted separately. Retrieval recall: does the name of the target procedure appear in the retrieved chunks. Answerability: do both the procedure name and a row of the table stand together in one of the retrieved chunks. An answer can be produced only if both are present.
• RP88. The cost is written in three units: the number of chunks in the index, the tokens stored in the index, and the tokens entering the prompt per request. The first two are storage, the third is the cost of the call.
• RP89. The whole measurement is deterministic: there is no sampling, the language model simulator is not called and the numbers do not change between runs, so the difference between two runs is zero by definition.
• RP90. Retrieval brings three chunks in every mode; recall and answerability are read over these three.

Five Ways to Chunk a Table

The chunk size and overlap decision was measured on plain text in this course’s chunking lesson. In a table the decision stands on another axis: whether the header is stored together with the row.

# FICTIONAL. The corpus, the concept axis, the embedding, the query set and the table source are
# produced inside the lesson; there is no real institution, product, document or embedding endpoint.
SEED, M32 = 20260218, 0xFFFFFFFF
DIM = 32


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


# ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side
CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"]
CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"}
CONCEPTS = {                     # concept: (label, document-side form, query-side form)
    "tier":      ("tariff", "tier limit", "which band am i in"),
    "unit":      ("tariff", "unit price", "what a ton of water costs"),
    "account":   ("tariff", "account type", "home or business"),
    "switch":    ("tariff", "plan switch", "moving to another package"),
    "faultcode": ("fault", "fault code", "error id"),
    "leak":      ("fault", "line leak", "water escaping"),
    "pressure":  ("fault", "pressure drop", "weak flow"),
    "crew":      ("fault", "field crew", "technician when does it show up"),
    "index":     ("meter", "index value", "digits on the dial"),
    "read":      ("meter", "reading routine", "how it gets written down"),
    "swap":      ("meter", "meter swap", "changing the hardware"),
    "verify":    ("meter", "reading check", "typed in by mistake"),
    "planned":   ("outage", "planned outage", "stoppage announced ahead"),
    "notice":    ("outage", "notice channel", "where do i find out"),
    "duration":  ("outage", "outage duration", "how long does it last"),
    "restore":   ("outage", "service restoration", "when does it show up"),
    "periodic":  ("maintenance", "periodic maintenance", "regular inspection"),
    "valve":     ("maintenance", "valve check", "tap shutoff order"),
    "filter":    ("maintenance", "filter replacement", "cleaning part"),
    "calib":     ("maintenance", "calibration record", "accuracy tweak"),
}
KEYS = list(CONCEPTS)
COMMON = ["entry", "area", "term", "task", "form", "okay", "note"]

# ---- FICTIONAL corpus
DOCUMENTS = []
for i in range(200):
    r = rng(SEED + 7919 * i)
    s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])]
    pool = [k for k in KEYS if CONCEPTS[k][0] == s]
    main = pool[int(r() * len(pool))]
    n = 2 + int(r() * 4)
    paras, concepts = [], [main]
    for _ in range(n):
        y = pool[int(r() * len(pool))]
        concepts.append(y)
        o = COMMON[int(r() * len(COMMON))]
        paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed")
    DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main,
                      "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure",
                      "paras": paras})


# ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions
def _to_concepts(m, domain=True):
    v = [0.0] * len(KEYS)
    dm = " " + m.lower() + " "
    for i, k in enumerate(KEYS):
        doc, qry = CONCEPTS[k][1], CONCEPTS[k][2]
        if doc in dm:
            v[i] += 1.0
        if qry in dm:
            v[i] += 1.0 if domain else 0.0      # the out-of-domain embedding does not recognize the query language
    if not domain:
        for o in COMMON:
            if " " + o + " " in dm:
                for i in range(len(KEYS)):
                    v[i] += 0.05
    return v


PROJECTION = []
for i in range(len(KEYS)):
    r = rng(SEED ^ (i + 1) * 2654435761)
    PROJECTION.append([r() * 2 - 1 for _ in range(64)])


def embed(m, dim=DIM, domain=True):
    k = _to_concepts(m, domain)
    v = [0.0] * dim
    for i, a in enumerate(k):
        if a:
            for j in range(dim):
                v[j] += a * PROJECTION[i][j % 64]
    n = sum(x * x for x in v) ** 0.5
    return [x / n for x in v] if n else v


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


# ---- FICTIONAL query set: the query language is separate from the document language
QUERIES = []
for i in range(40):
    r = rng(SEED + 104729 * i)
    d = DOCUMENTS[int(r() * len(DOCUMENTS))]
    QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}",
                    "label": d["label"], "concept": d["main"]})

# FICTIONAL table source: each procedure document is turned into a table.
# The procedure name is ONLY in the header; the rows carry the field name, the requirement and the source.
TABLES = []
for i, d in enumerate(DOCUMENTS):
    r = rng(SEED + 4093 * (i + 1))
    TABLES.append({"code": d["code"], "main": d["main"],
                   "head": f"{CONCEPTS[d['main']][1]} procedure table",
                   "rows": [f"{p.split()[3]} | {'yes' if r() < 0.5 else 'no'}"
                            f" | {'field' if r() < 0.5 else 'central'}" for p in d["paras"]]})


def build_index(mode):
    chunks = []
    for t in TABLES:
        if mode == "row":
            chunks += [{"code": t["code"], "text": s} for s in t["rows"]]
        elif mode == "block":
            chunks.append({"code": t["code"], "text": t["head"] + " ; " + " ; ".join(t["rows"])})
        elif mode == "row + header":
            chunks += [{"code": t["code"], "text": t["head"] + " ; " + s} for s in t["rows"]]
        elif mode == "visual":
            chunks.append({"code": t["code"], "text": ""})
        else:
            chunks.append({"code": t["code"], "text": t["head"]})
    return chunks


def measure(mode, k=3):
    chunks = build_index(mode)
    vecs = [embed(p["text"]) for p in chunks]
    recall = answerable = tokens = 0
    for s in QUERIES:
        v = embed(s["text"])
        top = sorted(range(len(chunks)), key=lambda i: (-cosine(v, vecs[i]), i))[:k]
        name = CONCEPTS[s["concept"]][1]
        recall += any(name in chunks[i]["text"] for i in top)
        answerable += any(name in chunks[i]["text"] and " | " in chunks[i]["text"] for i in top)
        tokens += sum(len(tokenize(chunks[i]["text"])) for i in top)
    n = len(QUERIES)
    return recall / n, answerable / n, len(chunks), sum(len(tokenize(p["text"])) for p in chunks), tokens / n


print("mode                  recall answerable index chunks stored tokens tokens per request")
for mode in ("row", "block", "row + header", "visual", "visual + description"):
    a, c, np_, sb, ib = measure(mode)
    print(f"{mode:<21} {a:>6.4f} {c:>10.4f} {np_:>12} {sb:>13} {ib:>18.1f}")

mode                  recall answerable index chunks stored tokens tokens per request
row                   0.0000     0.0000          693          3815               17.0
block                 0.9500     0.9500          200          5702               88.7
row + header          0.9500     0.9500          693          8634               38.5
visual                0.0000     0.0000          200             0                0.0
visual + description  0.9500     0.0000          200          1194               18.7

The Source Found, the Value Not Retrieved

Row-level chunking is the cheapest mode: 17.0 tokens per request and 3815 tokens in the index. And its recall is 0.0000. The query asks about a procedure, the rows do not contain its name, and the embedding finds no row close to the query. The loss is not a ranking fault; the surface searched for never entered the index. The same chunking worked on plain text, because there every sentence repeated its own subject.

Block-level chunking brings recall to 0.9500 and answerability with it: the header and the rows are in the same chunk. Its cost is 88.7 tokens per request, over five times the row-level mode. Stored tokens go from 3815 to 5702, since the headers are stored too.

The row + header mode lies between the two. Recall and answerability are identical to the block’s (0.9500), but the tokens per request fall to 38.5: the three chunks retrieved are now three rows, not a whole table. The price is in storage. With the header repeated in every row, stored tokens go from 5702 to 8634, against the block mode. The two costs move in opposite directions: the tokens per request fall while the stored tokens rise. The choice depends on the number of requests; it is a question of numbers, not of style.

The last two rows show the multimodal source’s own loss. When the table arrives as an image with no description, there is no text in the index: recall is 0.0000, stored tokens are 0. The source sits in the index, answering no query: dead weight in the retrieval space.

The real warning is in the fifth row. With a one-line description, recall jumps to 0.9500 while answerability stays at 0.0000. The description says what the table is about, it does not say what is written in it. Read with the four-cell contingency of retrieval and generation from the evaluation lesson, this is the box where the right chunk is retrieved but the answer cannot be produced; whoever reads a single recall number will think this mode works.

Summary

• In a table, meaning is not in the cell but in its relation to the header; row-level chunking drops the header, so recall falls to 0.0000, and the loss is in the index, not the ranking.
• Block-level chunking gives 0.9500 recall at 88.7 tokens per request; row + header gives the same 0.9500 at 38.5.
• Repeating the header costs storage: stored tokens go from 5702 to 8634, against the block mode; the two costs move in opposite directions.
• An image source without a description carries no text in the index, its recall is 0.0000 and it is dead weight in the retrieval space.
• A one-line description brings recall to 0.9500 but leaves answerability at 0.0000: the source is found, and the value inside it cannot be retrieved.

Course Wrap-Up

The course’s rule is written in the table row by row: retrieval and generation are measured separately. No number in the no-retrieval baseline column comes from a bad model; all of them come from the answer being not inside the model but in the corpus. Reducing the two columns to a single ratio hides the most instructive rows: in retrieval-pipeline/02, the unprepared source is the best of the retrieval column (recall 1.0000) and the worst of the generation column (0.0000).

The second claim can also be read from the same table: retrieval spoils too. In embeddings/02, the out-of-domain embedding falls below the keyword baseline on the retrieved set. In embeddings/04, with 1 probe recall falls to 0.7200 and the concept precision of the first five from 0.4700 to 0.3800. In embeddings/05, under a narrow filter the index gives 0.4750 with 18.8 comparisons; exhaustive search of the filtered set gives 1.0000 with 18. In embeddings/06, fusion is worse than both in the document language class and takes 1.0000 to 0.0500 in the code class. In retrieval-pipeline/01, the run with wrong chunks drops generation below the no-retrieval baseline. In retrieval-pipeline/03, raising the chunk size to 80 carries relation occurrence to 0.9971 and drops generation to 0.5500, because 0.15 of the requests exceed the context limit. In retrieval-pipeline/04, a longer refresh interval does not move the retrieval column; all the degradation stays in the generation column: the up-to-date answer share falls from 0.9250 to 0.6125. In retrieval-pipeline/05, rewriting takes the baseline recall from 0.5500 to 0.2750 on the long phrasing. In retrieval-pipeline/06, going from 24 to 48 candidates adds an unmeasurable 0.0083 to precision, takes recall from 1.0000 to 0.9500 and almost doubles the tokens. In retrieval-pipeline/07, ten chunks raise recall to 0.9750 and drop generation to 0.5750: what is retrieved is right, what is used is wrong. In retrieval-pipeline/08, going up to five chunks makes recall 1.0000, does not change the end-to-end number and raises the tokens from 1872 to 2932. In retrieval-pipeline/09, retrieval adds nothing on format compliance, from 0.0000 to 0.0000. And in this lesson, the visual source with a description takes recall to 0.9500 and leaves answerability at 0.0000: a number that improved, a job that did not.

Over sixteen lessons, which information enters the model was measured. But we always made this choice: how many chunks to retrieve, which criterion to rank by, whether to transform the query and how to build the context were written into the pipeline by hand. The pipeline was also single-pass: the query went in, the chunks came, the answer went out. Which tool the model would call by its own decision, how it would judge the result insufficient and retry, and where this loop would stop were never asked even once. The next course builds this loop and turns the stopping condition into a measurement object.