--- title: 'Multimodal Input' source: 'https://academia.sh/en/courses/model-apis/multimodal-input' course: 'Building Applications with Model APIs' language: en updated: '2026-10-21T05:02:32+00:00' license: 'CC BY-SA 4.0' objectives: - 'Shows in a simulator written with the standard library how an image, a sound and a sequence are turned into tokens as they enter the interface.' - 'Measures side by side the token cost and the reading accuracy of the same information in six input formats.' - 'Computes the ratio of tokens per usable record and shows how a resolution increase distorts it.' - 'Shows with numbers a mode staying blind to a field and rich input collapsing in the context window.' --- # Multimodal Input Measuring the same field inspection record in six input formats: the text summary 21, the cropped fine grid 42, the coarse grid 59, the fine grid 183 tokens, and the sequence between 195 and 330 per record. The usable record rate is 0.0833 on the coarse grid, 0.8167 on the fine grid and 0.7000 on the cropped grid; the cropped format delivers about eighty-six percent of the fine grid's usable records with a quarter of the tokens, and the cost per record falls from 224.1 to 60.0. Although the sequence format has the highest visibility, it stays at 0.3667 because it exceeds the context window in thirty-four of sixty records, and it becomes the most expensive format. The audio mode reads the index at 0.8833 but, being blind to the body condition, produces no usable record. The previous four lessons measured what the interface offers: a message sequence made of roles and contents, a response that arrives piece by piece, a call tied to code execution and an output mode that constrains generation. In all four, what entered the model was **text**. Token counting was done on text, the context window filled up with text, truncation decisions were pruned from text. The inspection record in the field team's hands is not text. The face of the meter has an image, sometimes there is a voice note, sometimes a short capture of a few frames. A multimodal interface accepts these; accepting them is not free. This lesson sends the same inspection record in six different formats and prints two numbers side by side: **the token cost of the input** and **the accuracy of the reading**. What is measured is not the model's ability to see, but the bill for the choice of format. - **AU41.** The code is a **simulator**; no real multimodal endpoint is called. The inspection records, the meter face and all the mode visibilities are **fictional**, seed **20260218**. - **AU42.** The unit a multimodal endpoint bills is still the **token**. An image is split into grid cells, a sound into pieces, a sequence into frames, and **each cell counts as a token**. The whole cost column of this lesson is written in this unit. - **AU43.** The six formats are: a hand-written **text** summary, the **coarse grid** (4x10), the **fine grid** (8x20), the **cropped** fine grid (3x8), the **sequence** (four to seven frames per record, each frame a coarse grid) and **audio** (eighteen to thirty pieces). - **AU44.** Each mode carries two **visibility** shares: the share with which the index and the body condition are read correctly. These shares are fictional constants of the simulator; they rise as the resolution rises. - **AU45.** The audio mode's visibility of the body condition is **zero**. A mode can be **blind** to a field, and this is not an error but the limit of the information that mode carries. - **AU46.** A **usable record** is a record whose index and body condition are both read correctly; this is the product criterion of this lesson. A record with only one field read correctly is of no use to the field team. - **AU47.** The context window is **240 tokens** and the rate limit is **once in thirty-seven calls**. The surface built in the Introduction to AI Engineering course continues unchanged. - **AU48.** The Natural Language Processing and Computer Vision course of the Machine Learning and Data Science curriculum measured the **shared representation** of a multimodal model; that procedure is not repeated here. The only thing measured is the **cost of the input format**. ## Six Formats of One Record When an image passes through the interface it is turned into a sequence of numbers, and that sequence into tokens. To make the conversion visible, the simulator of this lesson writes the grid directly into the prompt text: each cell a mark, each mark a token. At a real endpoint the intermediate steps are different, but **the bill comes from the same place**: as the resolution rises, the token count grows together with the cell count. The block below produces sixty fictional inspection records, builds the six formats and defines the multimodal call. The visibility shares changing by mode is the simulator's only assumption. ```python # SIMULATOR -- FICTIONAL. No real multimodal endpoint is called. The inspection # records, the meter face and all the mode visibilities are fictional. SEED, M32 = 20260218, 0xFFFFFFFF WINDOW = 240 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def tokenize(m): # FICTIONAL tokenizer, at most five characters per piece b = [] for s in m.lower().replace(",", " , ").split(): while len(s) > 5: b.append(s[:5]) s = s[5:] b.append(s) return b def hash_text(m): h = 2166136261 for c in m: h = ((h ^ ord(c)) * 16777619) & M32 return h CONDITIONS = ["saglam", "catlak", "muhursuz"] RECORDS = [] for i in range(60): # 60 FICTIONAL field inspection records r = rng(SEED + 900 + i) RECORDS.append({"number": 7001 + i, "index": "".join(str(int(r() * 10)) for _ in range(5)), "condition": CONDITIONS[pick(r(), [0.58, 0.27, 0.15])], "frames": 4 + int(r() * 4), "length": 18 + int(r() * 13)}) CELLS = ".:-=+*#" def grid(record, rows, cols): """FICTIONAL grid of the meter face: each cell is written as a separate token.""" r = rng(hash_text(record["index"] + record["condition"]) ^ (rows * 31 + cols)) out = [] for y in range(rows): out.append(f"s{y}") out += [CELLS[int(r() * 7)] for _ in range(cols)] return " ".join(out) def render(record, mode): """Six input formats of the same record. The information carried is the same, the length is not.""" if mode == "text": return f"endeks {record['index']} govde {record['condition']}" if mode == "coarse grid": return grid(record, 4, 10) if mode == "fine grid": return grid(record, 8, 20) if mode == "cropped grid": return grid(record, 3, 8) if mode == "audio": r = rng(hash_text(record["index"]) ^ 7717) return " ".join(f"d{int(r() * 9)}" for _ in range(record["length"])) return " ".join(f"k{j} " + grid(record, 4, 10) for j in range(record["frames"])) # sequence: frame by frame VISIBILITY = {"text": (1.00, 1.00), "coarse grid": (0.62, 0.16), "fine grid": (0.93, 0.85), "cropped grid": (0.91, 0.80), "sequence": (0.88, 0.95), "audio": (0.90, 0.00)} PREFIX = "Sayac yuzunu oku . Endeks ve govde durumunu yaz . Kayit: " CALL_COUNT = [0] def multimodal_call(record, mode, seed=SEED): """Multimodal simulator: each mode carries each field with its own visibility share.""" CALL_COUNT[0] += 1 toks = tokenize(PREFIX + render(record, mode)) if len(toks) > WINDOW: return {"status": "context_overflow", "input": len(toks), "output": 0} if CALL_COUNT[0] % 37 == 0: return {"status": "rate_limit", "input": len(toks), "output": 0} r = rng(seed ^ hash_text(" ".join(toks))) vis_index, vis_condition = VISIBILITY[mode] e, d = record["index"], record["condition"] if r() >= vis_index: # an unreadable digit slips to its neighbor j = int(r() * 5) e = e[:j] + str((int(e[j]) + 1 + int(r() * 8)) % 10) + e[j + 1:] if r() >= vis_condition: d = CONDITIONS[(CONDITIONS.index(d) + 1 + int(r() * 2)) % 3] return {"status": "done", "input": len(toks), "index": e, "body": d, "output": len(tokenize(f"endeks {e} govde {d}"))} k = RECORDS[0] print(f"record {k['number']} : index {k['index']} , condition {k['condition']} ," f" {k['frames']} frames , {k['length']} audio pieces") for mode in VISIBILITY: print(f" {mode:<14} input tokens {len(tokenize(PREFIX + render(k, mode))):>4}") print("first two rows of the coarse grid:") print(" ", " ".join(grid(k, 4, 10).split()[:22])) long_record = next(x for x in RECORDS if x["frames"] == 7) for record, mode in ((k, "text"), (k, "coarse grid"), (k, "fine grid"), (long_record, "sequence")): s = multimodal_call(record, mode) info = (f"index {s['index']} condition {s['body']}" if s["status"] == "done" else f"{s['input']} tokens , window {WINDOW}") print(f"{record['number']} {mode:<14} {s['status']:<16} {info}") ``` ``` record 7001 : index 17328 , condition saglam , 5 frames , 21 audio pieces text input tokens 21 coarse grid input tokens 59 fine grid input tokens 183 cropped grid input tokens 42 sequence input tokens 240 audio input tokens 36 first two rows of the coarse grid: s0 . * : - = * - : : + s1 = . : + = + : * + . 7001 text done index 17328 condition saglam 7001 coarse grid done index 17328 condition catlak 7001 fine grid done index 17328 condition saglam 7005 sequence context_overflow 330 tokens , window 240 ``` The difference among the six formats of a single record ranges between **21 tokens and 240 tokens**. The information carried is the same in every row: a five-digit index and a three-valued body condition. The elevenfold gap between them comes not from the information but from **the container the information is wrapped in**. The last four rows show two separate failures. The coarse grid reads the index correctly but gives the body condition as `cracked` instead of `intact`: the resolution is not enough to tell the seal from the crack. For record 7005, which has seven frames, no reading takes place at all; the 330-token sequence does not fit the 240-token window and the call returns with `context_overflow`. **The richest input is the input that is no input.** ## Cost and Accuracy Table of the Six Formats All sixty records are run separately in the six formats. Each row prints the total input tokens, the total output tokens, the number of failed calls, the separate accuracies of the two fields, the usable record rate and the **tokens per usable record**. The last column is the second run: the usable record rate obtained with seed 20260219, which says whether a difference can be told from noise. ```python def measure(mode, seed=SEED): CALL_COUNT[0] = 0 tok_in = tok_out = failed = index_ok = cond_ok = usable = 0 for x in RECORDS: s = multimodal_call(x, mode, seed) tok_in, tok_out = tok_in + s["input"], tok_out + s["output"] if s["status"] != "done": failed += 1 continue e, d = s["index"] == x["index"], s["body"] == x["condition"] index_ok, cond_ok, usable = index_ok + e, cond_ok + d, usable + (e and d) return tok_in, tok_out, failed, index_ok / 60, cond_ok / 60, usable / 60 print("mode input output failed index condition usable record" " tokens/record second run") for mode in VISIBILITY: a, b = measure(mode), measure(mode, SEED + 1) tpr = f"{a[0] / (a[5] * 60):.1f}" if a[5] else "-" print(f"{mode:<14} {a[0]:>6} {a[1]:>6} {a[2]:>10} {a[3]:>7.4f} {a[4]:>9.4f}" f" {a[5]:>15.4f} {tpr:>13} {b[5]:>13.4f}") ``` ``` mode input output failed index condition usable record tokens/record second run text 1260 354 1 0.9833 0.9833 0.9833 21.4 0.9833 coarse grid 3540 354 1 0.6500 0.1333 0.0833 708.0 0.0833 fine grid 10980 354 1 0.9500 0.8500 0.8167 224.1 0.7833 cropped grid 2520 354 1 0.8833 0.7833 0.7000 60.0 0.6833 sequence 16155 150 35 0.3667 0.4167 0.3667 734.3 0.2833 audio 2300 354 1 0.8833 0.0000 0.0000 - 0.0000 ``` The output token column is the same in five rows: 354; in the sequence it is 150, because thirty-five calls produce no response at all. The response is a single record of six tokens in every format. What changes is only the input, and this is the plainest row showing **which side of the bill of a multimodal interface swells**. The total cost of the text summary is 1260 tokens; the cost of the fine grid is 10980, that is, eight and a half times as much. The same information, the same response. The first row is a boundary case and should be read honestly. The text summary is both the cheapest and the most accurate, but that summary must **already have been read by someone**. The reason the usable record rate stays at 0.9833 is not the reading but the rate limit rejection that arrives on the thirty-seventh call. This row is not a mode option but the form the job takes once it has been solved on the human side; its function in the table is to show **how much of it the other five rows are trying to win back**. ## Cropping, the Blind Mode and the Sequence That Does Not Fit the Window The difference between the coarse grid and the fine grid tells which field the resolution rescues. The index accuracy rises from 0.6500 to 0.9500, but the real leap is in the body condition: from 0.1333 to 0.8500. A crack is **a finer detail** than a five-digit number and is invisible on the coarse grid. The usable record behaves like the product of these two fields and rises from 0.0833 to 0.8167. The cost triples: from 3540 tokens to 10980 tokens. The cropped grid row is the real finding of this lesson. Cropping shrinks not the resolution but the **area**: the same fine level of detail is applied only to the index and seal region of the meter. The result is 2520 tokens and 0.7000 usable record. It delivers about eighty-six percent of the fine grid's usable records with **a quarter** of its tokens. The cost per usable record falls from 224.1 to 60.0. The two run columns support this reading: the fine grid gives 0.8167 and 0.7833, the cropped grid gives 0.7000 and 0.6833, so the 0.1167 difference between them is larger than the difference between runs and is real. What is real is the superiority of the fine grid; what is debatable is **whether that superiority is worth 164 tokens per record**. The sequence row is the format with the highest visibility and one of the worst rows of the table. Looking frame by frame makes the body condition visible with a 0.95 share, because motion gives the crack away. But thirty-four of the sixty records carry six or seven frames and do not fit the 240-token window. The number of failed calls is 35 (thirty-four context overflows and one rate limit), the usable record is 0.3667, and the total cost is the highest of the table at 16155 tokens. The mode with the highest visibility is, in the records it delivers, half of the cropped grid. This is the counterpart in multimodal input of the second claim of this curriculum: **added richness can lower the measure.** The audio row shows another limit. The index accuracy is 0.8833, close to the fine grid, and the body condition accuracy is **0.0000**. The audio recording carries a person reading out the index; it does not carry the crack. The usable record is therefore zero and the cost per record cannot be computed. A mode not carrying a field does not show up in an average accuracy number; it appears only when the fields are counted **separately**. Choosing a mode is not a resolution decision but a decision about **which fields will be carried**. ## Summary - The bill of a multimodal interface is still written in tokens: a grid cell, an audio piece and a sequence frame turn into input tokens, and the output tokens stay fixed at 354 in five formats. - The cost of the same record is 21 tokens in the text summary, 42 on the cropped grid, 59 on the coarse grid and 183 on the fine grid; the information carried does not change, the container does. - Raising the resolution lifts the body condition accuracy from 0.1333 to 0.8500; the fine detail is invisible on the coarse grid and the usable record stays at 0.0833. - Cropping shrinks the area, not the resolution: it delivers about eighty-six percent of the fine grid's usable records with a quarter of the tokens, and the cost per record falls from 224.1 to 60.0. - The sequence format has the highest visibility but stays at 0.3667 because thirty-four of the sixty records do not fit the window, and at 16155 tokens it is the most expensive row. - The audio mode reads the index at 0.8833, is blind to the body condition and produces no usable record; a field a mode does not carry becomes visible only when the fields are counted separately. ## Next Step In this lesson the input was always sent **about one record** and the model read it. The question left unasked is this: if we had to find which of the sixty records resembles the current request, since putting all of them into the prompt is not possible, what would similarity be computed over? The interface offers a separate surface for this: an endpoint that turns text into a sequence of numbers and returns not text but a **vector** as the response. The next lesson measures this surface and focuses on a single decision: how many dimensions the vector will have. As the dimensions fall, the stored numbers and the comparison cost fall, and so does the neighborhood accuracy; the point at which the two part ways is counted.