Multimodal Input

The previous four lessons measured what the interface offers: a message sequence made of roles and contents, a response that arrives piece by piece, a call tied to code execution and an output mode that constrains generation. In all four, what entered the model was text. Token counting was done on text, the context window filled up with text, truncation decisions were pruned from text.

The inspection record in the field team’s hands is not text. The face of the meter has an image, sometimes there is a voice note, sometimes a short capture of a few frames. A multimodal interface accepts these; accepting them is not free. This lesson sends the same inspection record in six different formats and prints two numbers side by side: the token cost of the input and the accuracy of the reading. What is measured is not the model’s ability to see, but the bill for the choice of format.

• AU41. The code is a simulator; no real multimodal endpoint is called. The inspection records, the meter face and all the mode visibilities are fictional, seed 20260218.
• AU42. The unit a multimodal endpoint bills is still the token. An image is split into grid cells, a sound into pieces, a sequence into frames, and each cell counts as a token. The whole cost column of this lesson is written in this unit.
• AU43. The six formats are: a hand-written text summary, the coarse grid (4x10), the fine grid (8x20), the cropped fine grid (3x8), the sequence (four to seven frames per record, each frame a coarse grid) and audio (eighteen to thirty pieces).
• AU44. Each mode carries two visibility shares: the share with which the index and the body condition are read correctly. These shares are fictional constants of the simulator; they rise as the resolution rises.
• AU45. The audio mode’s visibility of the body condition is zero. A mode can be blind to a field, and this is not an error but the limit of the information that mode carries.
• AU46. A usable record is a record whose index and body condition are both read correctly; this is the product criterion of this lesson. A record with only one field read correctly is of no use to the field team.
• AU47. The context window is 240 tokens and the rate limit is once in thirty-seven calls. The surface built in the Introduction to AI Engineering course continues unchanged.
• AU48. The Natural Language Processing and Computer Vision course of the Machine Learning and Data Science curriculum measured the shared representation of a multimodal model; that procedure is not repeated here. The only thing measured is the cost of the input format.

Six Formats of One Record

When an image passes through the interface it is turned into a sequence of numbers, and that sequence into tokens. To make the conversion visible, the simulator of this lesson writes the grid directly into the prompt text: each cell a mark, each mark a token. At a real endpoint the intermediate steps are different, but the bill comes from the same place: as the resolution rises, the token count grows together with the cell count.

The block below produces sixty fictional inspection records, builds the six formats and defines the multimodal call. The visibility shares changing by mode is the simulator’s only assumption.

# SIMULATOR -- FICTIONAL. No real multimodal endpoint is called. The inspection
# records, the meter face and all the mode visibilities are fictional.
SEED, M32 = 20260218, 0xFFFFFFFF
WINDOW = 240


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def tokenize(m):                        # FICTIONAL tokenizer, at most five characters per piece
    b = []
    for s in m.lower().replace(",", " , ").split():
        while len(s) > 5:
            b.append(s[:5])
            s = s[5:]
        b.append(s)
    return b


def hash_text(m):
    h = 2166136261
    for c in m:
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CONDITIONS = ["saglam", "catlak", "muhursuz"]
RECORDS = []
for i in range(60):                     # 60 FICTIONAL field inspection records
    r = rng(SEED + 900 + i)
    RECORDS.append({"number": 7001 + i,
                    "index": "".join(str(int(r() * 10)) for _ in range(5)),
                    "condition": CONDITIONS[pick(r(), [0.58, 0.27, 0.15])],
                    "frames": 4 + int(r() * 4), "length": 18 + int(r() * 13)})
CELLS = ".:-=+*#"


def grid(record, rows, cols):
    """FICTIONAL grid of the meter face: each cell is written as a separate token."""
    r = rng(hash_text(record["index"] + record["condition"]) ^ (rows * 31 + cols))
    out = []
    for y in range(rows):
        out.append(f"s{y}")
        out += [CELLS[int(r() * 7)] for _ in range(cols)]
    return " ".join(out)


def render(record, mode):
    """Six input formats of the same record. The information carried is the same, the length is not."""
    if mode == "text":
        return f"endeks {record['index']} govde {record['condition']}"
    if mode == "coarse grid":
        return grid(record, 4, 10)
    if mode == "fine grid":
        return grid(record, 8, 20)
    if mode == "cropped grid":
        return grid(record, 3, 8)
    if mode == "audio":
        r = rng(hash_text(record["index"]) ^ 7717)
        return " ".join(f"d{int(r() * 9)}" for _ in range(record["length"]))
    return " ".join(f"k{j} " + grid(record, 4, 10)
                    for j in range(record["frames"]))       # sequence: frame by frame


VISIBILITY = {"text": (1.00, 1.00), "coarse grid": (0.62, 0.16),
              "fine grid": (0.93, 0.85), "cropped grid": (0.91, 0.80),
              "sequence": (0.88, 0.95), "audio": (0.90, 0.00)}
PREFIX = "Sayac yuzunu oku . Endeks ve govde durumunu yaz . Kayit: "
CALL_COUNT = [0]


def multimodal_call(record, mode, seed=SEED):
    """Multimodal simulator: each mode carries each field with its own visibility share."""
    CALL_COUNT[0] += 1
    toks = tokenize(PREFIX + render(record, mode))
    if len(toks) > WINDOW:
        return {"status": "context_overflow", "input": len(toks), "output": 0}
    if CALL_COUNT[0] % 37 == 0:
        return {"status": "rate_limit", "input": len(toks), "output": 0}
    r = rng(seed ^ hash_text(" ".join(toks)))
    vis_index, vis_condition = VISIBILITY[mode]
    e, d = record["index"], record["condition"]
    if r() >= vis_index:                # an unreadable digit slips to its neighbor
        j = int(r() * 5)
        e = e[:j] + str((int(e[j]) + 1 + int(r() * 8)) % 10) + e[j + 1:]
    if r() >= vis_condition:
        d = CONDITIONS[(CONDITIONS.index(d) + 1 + int(r() * 2)) % 3]
    return {"status": "done", "input": len(toks), "index": e, "body": d,
            "output": len(tokenize(f"endeks {e} govde {d}"))}


k = RECORDS[0]
print(f"record {k['number']} : index {k['index']} , condition {k['condition']} ,"
      f" {k['frames']} frames , {k['length']} audio pieces")
for mode in VISIBILITY:
    print(f"  {mode:<14} input tokens {len(tokenize(PREFIX + render(k, mode))):>4}")
print("first two rows of the coarse grid:")
print(" ", " ".join(grid(k, 4, 10).split()[:22]))
long_record = next(x for x in RECORDS if x["frames"] == 7)
for record, mode in ((k, "text"), (k, "coarse grid"), (k, "fine grid"),
                     (long_record, "sequence")):
    s = multimodal_call(record, mode)
    info = (f"index {s['index']} condition {s['body']}" if s["status"] == "done"
            else f"{s['input']} tokens , window {WINDOW}")
    print(f"{record['number']} {mode:<14} {s['status']:<16} {info}")

record 7001 : index 17328 , condition saglam , 5 frames , 21 audio pieces
  text           input tokens   21
  coarse grid    input tokens   59
  fine grid      input tokens  183
  cropped grid   input tokens   42
  sequence       input tokens  240
  audio          input tokens   36
first two rows of the coarse grid:
  s0 . * : - = * - : : + s1 = . : + = + : * + .
7001 text           done             index 17328 condition saglam
7001 coarse grid    done             index 17328 condition catlak
7001 fine grid      done             index 17328 condition saglam
7005 sequence       context_overflow 330 tokens , window 240

The difference among the six formats of a single record ranges between 21 tokens and 240 tokens. The information carried is the same in every row: a five-digit index and a three-valued body condition. The elevenfold gap between them comes not from the information but from the container the information is wrapped in.

The last four rows show two separate failures. The coarse grid reads the index correctly but gives the body condition as cracked instead of intact: the resolution is not enough to tell the seal from the crack. For record 7005, which has seven frames, no reading takes place at all; the 330-token sequence does not fit the 240-token window and the call returns with context_overflow. The richest input is the input that is no input.

Cost and Accuracy Table of the Six Formats

All sixty records are run separately in the six formats. Each row prints the total input tokens, the total output tokens, the number of failed calls, the separate accuracies of the two fields, the usable record rate and the tokens per usable record. The last column is the second run: the usable record rate obtained with seed 20260219, which says whether a difference can be told from noise.

def measure(mode, seed=SEED):
    CALL_COUNT[0] = 0
    tok_in = tok_out = failed = index_ok = cond_ok = usable = 0
    for x in RECORDS:
        s = multimodal_call(x, mode, seed)
        tok_in, tok_out = tok_in + s["input"], tok_out + s["output"]
        if s["status"] != "done":
            failed += 1
            continue
        e, d = s["index"] == x["index"], s["body"] == x["condition"]
        index_ok, cond_ok, usable = index_ok + e, cond_ok + d, usable + (e and d)
    return tok_in, tok_out, failed, index_ok / 60, cond_ok / 60, usable / 60


print("mode            input output     failed   index condition   usable record"
      " tokens/record    second run")
for mode in VISIBILITY:
    a, b = measure(mode), measure(mode, SEED + 1)
    tpr = f"{a[0] / (a[5] * 60):.1f}" if a[5] else "-"
    print(f"{mode:<14} {a[0]:>6} {a[1]:>6} {a[2]:>10} {a[3]:>7.4f} {a[4]:>9.4f}"
          f" {a[5]:>15.4f} {tpr:>13} {b[5]:>13.4f}")

mode            input output     failed   index condition   usable record tokens/record    second run
text             1260    354          1  0.9833    0.9833          0.9833          21.4        0.9833
coarse grid      3540    354          1  0.6500    0.1333          0.0833         708.0        0.0833
fine grid       10980    354          1  0.9500    0.8500          0.8167         224.1        0.7833
cropped grid     2520    354          1  0.8833    0.7833          0.7000          60.0        0.6833
sequence        16155    150         35  0.3667    0.4167          0.3667         734.3        0.2833
audio            2300    354          1  0.8833    0.0000          0.0000             -        0.0000

The output token column is the same in five rows: 354; in the sequence it is 150, because thirty-five calls produce no response at all. The response is a single record of six tokens in every format. What changes is only the input, and this is the plainest row showing which side of the bill of a multimodal interface swells. The total cost of the text summary is 1260 tokens; the cost of the fine grid is 10980, that is, eight and a half times as much. The same information, the same response.

The first row is a boundary case and should be read honestly. The text summary is both the cheapest and the most accurate, but that summary must already have been read by someone. The reason the usable record rate stays at 0.9833 is not the reading but the rate limit rejection that arrives on the thirty-seventh call. This row is not a mode option but the form the job takes once it has been solved on the human side; its function in the table is to show how much of it the other five rows are trying to win back.

Cropping, the Blind Mode and the Sequence That Does Not Fit the Window

The difference between the coarse grid and the fine grid tells which field the resolution rescues. The index accuracy rises from 0.6500 to 0.9500, but the real leap is in the body condition: from 0.1333 to 0.8500. A crack is a finer detail than a five-digit number and is invisible on the coarse grid. The usable record behaves like the product of these two fields and rises from 0.0833 to 0.8167. The cost triples: from 3540 tokens to 10980 tokens.

The cropped grid row is the real finding of this lesson. Cropping shrinks not the resolution but the area: the same fine level of detail is applied only to the index and seal region of the meter. The result is 2520 tokens and 0.7000 usable record. It delivers about eighty-six percent of the fine grid’s usable records with a quarter of its tokens. The cost per usable record falls from 224.1 to 60.0. The two run columns support this reading: the fine grid gives 0.8167 and 0.7833, the cropped grid gives 0.7000 and 0.6833, so the 0.1167 difference between them is larger than the difference between runs and is real. What is real is the superiority of the fine grid; what is debatable is whether that superiority is worth 164 tokens per record.

The sequence row is the format with the highest visibility and one of the worst rows of the table. Looking frame by frame makes the body condition visible with a 0.95 share, because motion gives the crack away. But thirty-four of the sixty records carry six or seven frames and do not fit the 240-token window. The number of failed calls is 35 (thirty-four context overflows and one rate limit), the usable record is 0.3667, and the total cost is the highest of the table at 16155 tokens. The mode with the highest visibility is, in the records it delivers, half of the cropped grid. This is the counterpart in multimodal input of the second claim of this curriculum: added richness can lower the measure.

The audio row shows another limit. The index accuracy is 0.8833, close to the fine grid, and the body condition accuracy is 0.0000. The audio recording carries a person reading out the index; it does not carry the crack. The usable record is therefore zero and the cost per record cannot be computed. A mode not carrying a field does not show up in an average accuracy number; it appears only when the fields are counted separately. Choosing a mode is not a resolution decision but a decision about which fields will be carried.

Summary

• The bill of a multimodal interface is still written in tokens: a grid cell, an audio piece and a sequence frame turn into input tokens, and the output tokens stay fixed at 354 in five formats.
• The cost of the same record is 21 tokens in the text summary, 42 on the cropped grid, 59 on the coarse grid and 183 on the fine grid; the information carried does not change, the container does.
• Raising the resolution lifts the body condition accuracy from 0.1333 to 0.8500; the fine detail is invisible on the coarse grid and the usable record stays at 0.0833.
• Cropping shrinks the area, not the resolution: it delivers about eighty-six percent of the fine grid’s usable records with a quarter of the tokens, and the cost per record falls from 224.1 to 60.0.
• The sequence format has the highest visibility but stays at 0.3667 because thirty-four of the sixty records do not fit the window, and at 16155 tokens it is the most expensive row.
• The audio mode reads the index at 0.8833, is blind to the body condition and produces no usable record; a field a mode does not carry becomes visible only when the fields are counted separately.

Next Step

In this lesson the input was always sent about one record and the model read it. The question left unasked is this: if we had to find which of the sixty records resembles the current request, since putting all of them into the prompt is not possible, what would similarity be computed over? The interface offers a separate surface for this: an endpoint that turns text into a sequence of numbers and returns not text but a vector as the response. The next lesson measures this surface and focuses on a single decision: how many dimensions the vector will have. As the dimensions fall, the stored numbers and the comparison cost fall, and so does the neighborhood accuracy; the point at which the two part ways is counted.