--- title: 'Chunking Sources Containing Tables and Images' source: 'https://academia.sh/en/courses/embeddings-and-retrieval-augmented-generation/chunking-sources-containing-tables-and-images' course: 'Embeddings and Retrieval-Augmented Generation' language: en updated: '2026-10-21T05:02:29+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures row-level, block-level and header repetition chunking of a table on the same query set.' - 'Counts retrieval recall and answerability separately and shows that the two come apart on an image source with a description.' - 'Computes that the tokens stored in the index and the tokens sent per request move in opposite directions.' - "Collects the no-retrieval baseline and retrieval numbers of the course's sixteen lessons in a single table." --- # Chunking Sources Containing Tables and Images Counting what the chunking decision does to retrieval recall when the same fictional corpus is presented in table form: because the procedure name sits only in the header, row-level chunking drops recall to 0.0000; block-level chunking gives 0.9500 but carries 88.7 tokens per request; storing each row together with its header gives the same 0.9500 recall with 38.5 tokens per request and raises the tokens stored in the index from 5702 to 8634. When the table arrives as an image, the source without a description gives 0.0000, and the source with a one-line description gives 0.9500 recall but 0.0000 answerability: the source is found, and the value inside it cannot be retrieved. This lesson closes the Embeddings and Retrieval-Augmented Generation course. The previous lesson measured retrieval and fine-tuning side by side on three tasks and showed both rest on the same assumption: that everything in the corpus is **plain text**. The fictional documents were built from sentences, and each chunk said what it was about by itself. The procedure documents of the field team are not like that. They contain **tables**, and in a table meaning does not live in the cell but in **its relation to the header**. When a table is split like a sentence, row and header are separated and the row says nothing by itself. In sources that carry images the loss goes one step further: no text exists at all. Multimodal input itself was measured in the Natural Language Processing and Computer Vision course, under computer vision, and the token cost of the input format was counted in the Building Applications with Model APIs course: the fine grid gave a hit rate of **0.8167**, the cropped grid **0.7000**, and the tokens per usable record fell from **224.1** to **60.0**. Those procedures are not repeated here. This lesson measures only **the chunking decision**. - **RP84.** The corpus, the concept axis, the embedding, the query set and the table source are **fictional** and are produced inside the lesson. There is no real institution, product, document, parser or embedding endpoint. - **RP85.** This lesson turns the same two hundred documents into **tables**. **The procedure name stays only in the header**; the rows carry the field name, the requirement and the source columns. What the table is about is not written in the cells. - **RP86.** Five chunking modes are measured: **row** (each row is a chunk, the header is dropped), **block** (the whole table is one chunk), **row + header** (each row is stored together with the header), **visual** (the table arrives as an image, there is no text) and **visual + description** (the image has a one-line description). - **RP87.** Two measures are counted separately. **Retrieval recall**: does the name of the target procedure appear in the retrieved chunks. **Answerability**: do both the procedure name and a **row** of the table stand together in one of the retrieved chunks. An answer can be produced only if both are present. - **RP88.** The cost is written in three units: the number of chunks in the index, the tokens **stored** in the index, and the tokens entering the prompt **per request**. The first two are storage, the third is the cost of the call. - **RP89.** The whole measurement is **deterministic**: there is no sampling, the language model simulator is not called and the numbers do not change between runs, so the difference between two runs is zero by definition. - **RP90.** Retrieval brings three chunks in every mode; recall and answerability are read over these three. ## Five Ways to Chunk a Table The chunk size and overlap decision was measured on plain text in this course's chunking lesson. In a table the decision stands on another axis: **whether the header is stored together with the row**. ```python # FICTIONAL. The corpus, the concept axis, the embedding, the query set and the table source are # produced inside the lesson; there is no real institution, product, document or embedding endpoint. SEED, M32 = 20260218, 0xFFFFFFFF DIM = 32 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b # ---- FICTIONAL concept axis: each concept has a separate surface form on the document side and on the query side CLASSES = ["tariff", "fault", "meter", "outage", "maintenance"] CODE = {"tariff": "TR", "fault": "AR", "meter": "SY", "outage": "KS", "maintenance": "BK"} CONCEPTS = { # concept: (label, document-side form, query-side form) "tier": ("tariff", "tier limit", "which band am i in"), "unit": ("tariff", "unit price", "what a ton of water costs"), "account": ("tariff", "account type", "home or business"), "switch": ("tariff", "plan switch", "moving to another package"), "faultcode": ("fault", "fault code", "error id"), "leak": ("fault", "line leak", "water escaping"), "pressure": ("fault", "pressure drop", "weak flow"), "crew": ("fault", "field crew", "technician when does it show up"), "index": ("meter", "index value", "digits on the dial"), "read": ("meter", "reading routine", "how it gets written down"), "swap": ("meter", "meter swap", "changing the hardware"), "verify": ("meter", "reading check", "typed in by mistake"), "planned": ("outage", "planned outage", "stoppage announced ahead"), "notice": ("outage", "notice channel", "where do i find out"), "duration": ("outage", "outage duration", "how long does it last"), "restore": ("outage", "service restoration", "when does it show up"), "periodic": ("maintenance", "periodic maintenance", "regular inspection"), "valve": ("maintenance", "valve check", "tap shutoff order"), "filter": ("maintenance", "filter replacement", "cleaning part"), "calib": ("maintenance", "calibration record", "accuracy tweak"), } KEYS = list(CONCEPTS) COMMON = ["entry", "area", "term", "task", "form", "okay", "note"] # ---- FICTIONAL corpus DOCUMENTS = [] for i in range(200): r = rng(SEED + 7919 * i) s = CLASSES[pick(r(), [0.22, 0.24, 0.20, 0.18, 0.16])] pool = [k for k in KEYS if CONCEPTS[k][0] == s] main = pool[int(r() * len(pool))] n = 2 + int(r() * 4) paras, concepts = [], [main] for _ in range(n): y = pool[int(r() * len(pool))] concepts.append(y) o = COMMON[int(r() * len(COMMON))] paras.append(f"{CONCEPTS[y][1]} for {o} goes with {CONCEPTS[main][1]} and is filed") DOCUMENTS.append({"code": f"{CODE[s]}{i + 1:03d}", "label": s, "main": main, "concepts": concepts, "title": f"{CONCEPTS[main][1]} procedure", "paras": paras}) # ---- FICTIONAL embedding: the surface is first reduced to a concept, the concept then to dimensions def _to_concepts(m, domain=True): v = [0.0] * len(KEYS) dm = " " + m.lower() + " " for i, k in enumerate(KEYS): doc, qry = CONCEPTS[k][1], CONCEPTS[k][2] if doc in dm: v[i] += 1.0 if qry in dm: v[i] += 1.0 if domain else 0.0 # the out-of-domain embedding does not recognize the query language if not domain: for o in COMMON: if " " + o + " " in dm: for i in range(len(KEYS)): v[i] += 0.05 return v PROJECTION = [] for i in range(len(KEYS)): r = rng(SEED ^ (i + 1) * 2654435761) PROJECTION.append([r() * 2 - 1 for _ in range(64)]) def embed(m, dim=DIM, domain=True): k = _to_concepts(m, domain) v = [0.0] * dim for i, a in enumerate(k): if a: for j in range(dim): v[j] += a * PROJECTION[i][j % 64] n = sum(x * x for x in v) ** 0.5 return [x / n for x in v] if n else v def cosine(a, b): return sum(x * y for x, y in zip(a, b)) # ---- FICTIONAL query set: the query language is separate from the document language QUERIES = [] for i in range(40): r = rng(SEED + 104729 * i) d = DOCUMENTS[int(r() * len(DOCUMENTS))] QUERIES.append({"text": f"{CONCEPTS[d['main']][2]} {COMMON[int(r() * len(COMMON))]}", "label": d["label"], "concept": d["main"]}) # FICTIONAL table source: each procedure document is turned into a table. # The procedure name is ONLY in the header; the rows carry the field name, the requirement and the source. TABLES = [] for i, d in enumerate(DOCUMENTS): r = rng(SEED + 4093 * (i + 1)) TABLES.append({"code": d["code"], "main": d["main"], "head": f"{CONCEPTS[d['main']][1]} procedure table", "rows": [f"{p.split()[3]} | {'yes' if r() < 0.5 else 'no'}" f" | {'field' if r() < 0.5 else 'central'}" for p in d["paras"]]}) def build_index(mode): chunks = [] for t in TABLES: if mode == "row": chunks += [{"code": t["code"], "text": s} for s in t["rows"]] elif mode == "block": chunks.append({"code": t["code"], "text": t["head"] + " ; " + " ; ".join(t["rows"])}) elif mode == "row + header": chunks += [{"code": t["code"], "text": t["head"] + " ; " + s} for s in t["rows"]] elif mode == "visual": chunks.append({"code": t["code"], "text": ""}) else: chunks.append({"code": t["code"], "text": t["head"]}) return chunks def measure(mode, k=3): chunks = build_index(mode) vecs = [embed(p["text"]) for p in chunks] recall = answerable = tokens = 0 for s in QUERIES: v = embed(s["text"]) top = sorted(range(len(chunks)), key=lambda i: (-cosine(v, vecs[i]), i))[:k] name = CONCEPTS[s["concept"]][1] recall += any(name in chunks[i]["text"] for i in top) answerable += any(name in chunks[i]["text"] and " | " in chunks[i]["text"] for i in top) tokens += sum(len(tokenize(chunks[i]["text"])) for i in top) n = len(QUERIES) return recall / n, answerable / n, len(chunks), sum(len(tokenize(p["text"])) for p in chunks), tokens / n print("mode recall answerable index chunks stored tokens tokens per request") for mode in ("row", "block", "row + header", "visual", "visual + description"): a, c, np_, sb, ib = measure(mode) print(f"{mode:<21} {a:>6.4f} {c:>10.4f} {np_:>12} {sb:>13} {ib:>18.1f}") ``` ``` mode recall answerable index chunks stored tokens tokens per request row 0.0000 0.0000 693 3815 17.0 block 0.9500 0.9500 200 5702 88.7 row + header 0.9500 0.9500 693 8634 38.5 visual 0.0000 0.0000 200 0 0.0 visual + description 0.9500 0.0000 200 1194 18.7 ``` ## The Source Found, the Value Not Retrieved Row-level chunking is the cheapest mode: **17.0** tokens per request and 3815 tokens in the index. And its recall is **0.0000**. The query asks about a procedure, the rows do not contain its name, and the embedding finds no row close to the query. The loss is not a ranking fault; the surface searched for **never entered the index**. The same chunking worked on plain text, because there every sentence repeated its own subject. Block-level chunking brings recall to **0.9500** and answerability with it: the header and the rows are in the same chunk. Its cost is **88.7** tokens per request, over five times the row-level mode. Stored tokens go from 3815 to **5702**, since the headers are stored too. The row + header mode lies between the two. Recall and answerability are **identical** to the block's (0.9500), but the tokens per request fall to **38.5**: the three chunks retrieved are now three rows, not a whole table. The price is in storage. With the header repeated in every row, stored tokens go from 5702 to **8634**, against the block mode. **The two costs move in opposite directions**: the tokens per request fall while the stored tokens rise. The choice depends on the number of requests; it is a question of numbers, not of style. The last two rows show the multimodal source's own loss. When the table arrives as an image with no description, there is no text in the index: recall is **0.0000**, stored tokens are **0**. The source sits in the index, answering no query: dead weight in the retrieval space. The real warning is in the fifth row. With a one-line description, recall jumps to **0.9500** while answerability stays at **0.0000**. The description says **what the table is about**, it does not say **what is written in it**. Read with the four-cell contingency of retrieval and generation from the evaluation lesson, this is the box where the right chunk is retrieved but the answer cannot be produced; whoever reads a single recall number will think this mode **works**. ## Summary - In a table, meaning is not in the cell but in its relation to the header; row-level chunking drops the header, so recall falls to 0.0000, and the loss is in the index, not the ranking. - Block-level chunking gives 0.9500 recall at 88.7 tokens per request; row + header gives the same 0.9500 at 38.5. - Repeating the header costs storage: stored tokens go from 5702 to 8634, against the block mode; the two costs move in opposite directions. - An image source without a description carries no text in the index, its recall is 0.0000 and it is dead weight in the retrieval space. - A one-line description brings recall to 0.9500 but leaves answerability at 0.0000: the source is found, and the value inside it cannot be retrieved. ## Course Wrap-Up | Lesson | No-retrieval baseline | With retrieval | Cost | |---|---|---|---| | `embeddings/01` — first result's concept hit rate | keyword search 0.0250 | embedding similarity 0.5000 (**+0.4750**) | 8000 comparisons, 240 embedding calls, 6400 stored values | | `embeddings/02` — concept hit rate, 32 dimensions | out-of-domain embedding 0.0500, barely above the 0.0250 baseline | in-domain embedding 0.5000 (**+0.4500**) | stored values 6400 in both setups; band between two models 0.0250 | | `embeddings/03` — first result's concept hit rate | dot product 0.4250 | cosine similarity 0.5000 (**+0.0750**) | first three diverge in thirty of forty queries; average of the retrieved set 66.2 → 57.7 tokens | | `embeddings/04` — recall against exhaustive search | exhaustive search 1.0000, 200 comparisons | bucket-based index with 2 probes 0.9500 (**−0.0500**) | 47.2 comparisons; with 1 probe recall 0.7200 and concept precision 0.4700 → 0.3800 | | `embeddings/05` — filtered recall | unfiltered index 0.9500 | in-index filtering, width 1: 0.4750 (**−0.4750**) | candidates 31.2 → 2.8; exhaustive search of the filtered set 1.0000 with 18 comparisons | | `embeddings/06` — first rank in the document language class | embedding alone 0.5000 | rank fusion 0.4000 (**−0.1000**) | in the code class 1.0000 → 0.0500; recall up to the twentieth rank 0.7122 → 0.7214 | | `retrieval-pipeline/01` — generation hit rate | no-retrieval baseline 0.2000 | pipeline with retrieval 0.9250 (**+0.7250**) | 3033 → 4848 tokens, 120 chunks; run with wrong chunks 0.0500 | | `retrieval-pipeline/02` — generation hit rate | single block without preparation 0.0000, its recall 1.0000 | structure-aware preparation 0.9250 (**+0.9250**) | stored chunks 1805 → 693; the filter deletes 96 of 568 concept links | | `retrieval-pipeline/03` — share of chunks carrying a relation occurrence whole | size 8 chunking 0.0765 | size 32 chunking 0.8846 (**+0.8081**) | stored chunks 1217 → 369; at size 80 generation 0.5500, 0.15 of requests exceed the context limit | | `retrieval-pipeline/04` — up-to-date answer share | refresh once in twelve periods 0.6125 | refresh every period 0.9250 (**+0.3125**) | embedded chunks 519 → 881; stale chunk rate 0.2562 → 0.0000, wrong answer rate 0.3875 → 0.0750 | | `retrieval-pipeline/05` — retrieval recall on the short phrasing | without transformation 0.3250 | expansion 0.7500 (**+0.4250**) | one call per query, 4609 → 5804 tokens; on the long phrasing recall 0.5500 → 0.5000 | | `retrieval-pipeline/06` — precision of the top three | single-stage retrieval 0.4167 (generation 0.7000) | reranking 24 candidates 0.9000 (**+0.4833**, generation 1.0000) | calls 40 → 1000, tokens 3654 → 38003; at 48 candidates recall 1.0000 → 0.9500 | | `retrieval-pipeline/07` — generation hit rate / recall | one chunk 0.7750 / 0.7750 | ten chunks 0.5750 (**−0.2000**) / 0.9750 | 13.4 chunks fit the context window; at k=13 fifteen of forty queries break the limit | | `retrieval-pipeline/08` — end-to-end accuracy rate | no-retrieval baseline 0.3750, all of it chance | three chunks 0.9500 (**+0.5750**) | 120 chunks, 1872 tokens; same 0.9500 with five chunks, the fault carries into generation | | `retrieval-pipeline/09` — knowledge freshness | no retrieval 0.0000 | with retrieval 0.9500 (**+0.9500**) | 2232 tokens, 40 retrieval steps; on format compliance 0.0000 → 0.0000 | | `retrieval-pipeline/10` — answerability | row-level chunking 0.0000 | row + header 0.9500 (**+0.9500**) | 17.0 → 38.5 tokens per request; stored in the index 3815 → 8634 | The course's rule is written in the table row by row: **retrieval and generation are measured separately.** No number in the no-retrieval baseline column comes from a bad model; all of them come from the answer being not inside the model but **in the corpus**. Reducing the two columns to a single ratio hides the most instructive rows: in `retrieval-pipeline/02`, the unprepared source is **the best of the retrieval column** (recall 1.0000) and **the worst of the generation column** (0.0000). **The second claim can also be read from the same table: retrieval spoils too.** In `embeddings/02`, the out-of-domain embedding falls **below** the keyword baseline on the retrieved set. In `embeddings/04`, with 1 probe recall falls to 0.7200 and the concept precision of the first five from 0.4700 to 0.3800. In `embeddings/05`, under a narrow filter the index gives 0.4750 with 18.8 comparisons; exhaustive search of the filtered set gives 1.0000 with 18. In `embeddings/06`, fusion is **worse than both** in the document language class and takes 1.0000 to 0.0500 in the code class. In `retrieval-pipeline/01`, the run with wrong chunks drops generation below the no-retrieval baseline. In `retrieval-pipeline/03`, raising the chunk size to 80 carries relation occurrence to 0.9971 and drops generation to **0.5500**, because 0.15 of the requests exceed the context limit. In `retrieval-pipeline/04`, a longer refresh interval **does not move** the retrieval column; all the degradation stays in the generation column: the up-to-date answer share falls from 0.9250 to 0.6125. In `retrieval-pipeline/05`, rewriting takes the baseline recall from 0.5500 to 0.2750 on the long phrasing. In `retrieval-pipeline/06`, going from 24 to 48 candidates adds an unmeasurable 0.0083 to precision, takes recall from 1.0000 to 0.9500 and almost doubles the tokens. In `retrieval-pipeline/07`, ten chunks raise recall to 0.9750 and drop generation to **0.5750**: what is retrieved is right, what is used is wrong. In `retrieval-pipeline/08`, going up to five chunks makes recall 1.0000, **does not change** the end-to-end number and raises the tokens from 1872 to 2932. In `retrieval-pipeline/09`, retrieval adds nothing on format compliance, from 0.0000 to 0.0000. And in this lesson, the visual source with a description takes recall to 0.9500 and leaves answerability at 0.0000: a number that improved, a job that did not. Over sixteen lessons, **which information** enters the model was measured. But **we** always made this choice: how many chunks to retrieve, which criterion to rank by, whether to transform the query and how to build the context were written into the pipeline by hand. The pipeline was also **single-pass**: the query went in, the chunks came, the answer went out. Which tool the model would call by its own decision, how it would judge the result insufficient and retry, and **where** this loop would stop were never asked even once. The next course builds this loop and turns the stopping condition into a measurement object.