--- title: 'Multimodal Models' source: 'https://academia.sh/en/courses/nlp-and-vision/multimodal-models' course: 'Natural Language Processing and Computer Vision' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Learns two linear projections that carry the note and the coverage frame of the same event into a joint space with a contrastive objective' - 'Reads the matching success against the random matching baseline and the ceiling of the shared information carried by the fiction' - 'Compares the numbers of the note only, the frame only, and the joint representation on the event kind decision with the same procedure' --- # Multimodal Models Measuring a joint representation, which brings the note and the coverage frame of the same event together in a single space, against its own baseline: in rooms of twenty-five candidates, random matching gives 0.0400 while the joint representation learned with a contrastive objective gives 0.1660, and the ceiling of the shared information carried by the fiction is 0.2659. On the event kind decision, the most frequent class baseline is 0.4420, note only 0.8940, frame only 0.6140, joint representation 0.8660: the representation that wins at matching stays below what the note alone gives in classification. The course wrap-up collects the domain baseline of fourteen lessons, the method's number, and the difference in a single table. Five lessons measured images and the first eight lessons measured text on its own; yet the **note** the field crew files for an event and the **coverage frame** of the same event belong to the same event. The measure of this lesson is not an accuracy but **matching**: given a note, can we pick the right one among the candidate frames. Its baseline is **random matching**; its ceiling is not 1, it is whatever information the two modalities **share**, and that is computed inside the lesson. - **CV48. The event record is fiction and has two modalities.** The note generator is the same as in the `nlp` topic, the frame is the same as the 12x12 gray-level fiction of lesson 04. Seed 20260218, split 1,100/500. - **CV49. The two modalities do not carry the same information equally.** The frame carries the event kind with a pattern (no block, an unreadable band, one block, two blocks) and the line with a faint band; the note carries the kind with words but writes the line only some of the time. There is noise on both sides: a borrowed side record in the note, and in the frame a missing pattern and an unexpected bright spot. - **CV50. The matching metric is 25 candidates per room.** The ceiling is computed from indistinguishable candidates: two events that carry the same kind and, if the line is written in the note, the same line cannot be told apart. - **CV51. The joint representation is two linear projections**: one from the vocabulary, the other from the frame, into the same space. It is learned with a **contrastive objective**: the similarity of the correct pair is raised, and those of the other twenty-four pairs in the batch are lowered. - **CV52. The event kind decision is read on three representations with the same procedure:** cosine similarity to the class mean, with the most frequent class as the baseline. The vocabulary, the scale, and the weights are learned from training only; no library is called. ## The Two Modalities of the Same Event ```python # event.py — MODEL: Event record fiction: the NOTE of the same event (from the nlp topic) and its # COVERAGE FRAME (the 12x12 gray-level fiction of lesson 04). Seed 20260218. import math SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12 BACKGROUND, BLOCK, NOISE = 0.30, 0.74, 0.30 KINDS = ("ordinary", "unreachable", "meter fault", "illicit use") SHARE, PATTERN = (0.46, 0.22, 0.19, 0.13), (0, 0, 1, 2) PHRASES = [s.split("|") for s in ( "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|" "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|" "gosterge okunakli|gostergede rakam net", "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|" "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|" "bodrum kilitli|adres bulunamadi|adres eksik", "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|" "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|" "mekanizmada takilma|sayac durmus|sayac calismiyor", "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|" "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|" "sayac ters takilmis|sayaci ters takmislar")] EXTRA = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|" "not kisa tutuldu|ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|") LINES = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|") def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def event(number): """Two modalities of the same event: a short Turkish field note and a 12x12 coverage frame.""" r, k, t = rng(SEED + 41000 + number), 0.0, 0 u = r() for i, p in enumerate(SHARE): k += p if u < k: t = i break parts = [PHRASES[t][int(r() * 12)]] if r() < 0.55: parts.append(PHRASES[t][int(r() * 12)]) if r() < 0.34: # side record borrowed from another kind parts.append(PHRASES[int(r() * 4)][int(r() * 12)]) if r() < 0.62: parts.append(EXTRA[int(r() * 8)]) line, written = int(r() * 5), r() < 0.30 # the line is written in the note only some of the time if written: parts.append(LINES[line]) q = rng(SEED + 600000 + number) g = [[BACKGROUND + (q() + q() + q() - 1.5) * NOISE for _ in range(SIZE)] for _ in range(SIZE)] for i in range(SIZE): # the line that is read appears as a faint band g[i][line * 2] += 0.16 g[i][line * 2 + 1] += 0.16 def block(sy, sx): for i in range(sy, min(SIZE, sy + 2 + int(q() * 2))): for j in range(sx, min(SIZE, sx + 2 + int(q() * 2))): g[i][j] = BLOCK + (q() + q() + q() - 1.5) * NOISE for b in range(PATTERN[t]): if q() >= 0.12: # in some frames the pattern is missing block(int(q() * (SIZE - 3)), line * 2 + b * 2) if t == 1: # not reached: unreadable band for i in range(SIZE): g[i][line * 2] = g[i][line * 2 + 1] = 0.02 + q() * 0.05 if q() < 0.15: # unexpected bright spot elsewhere block(int(q() * (SIZE - 3)), int(q() * (SIZE - 3))) return {"text": ", ".join(parts), "kind": KINDS[t], "line": line, "written": written, "frame": g} DATA = [event(i) for i in range(1600)] TEST = DATA[1100:] count, VOCAB, seen = {}, {}, {} for x in DATA[:1100]: count[x["kind"]] = count.get(x["kind"], 0) + 1 for s in set(x["text"].split()): seen[s] = seen.get(s, 0) + 1 for s in sorted(seen): if seen[s] >= 2: VOCAB[s] = len(VOCAB) MOST_COMMON = max(count, key=count.get) BASELINE = sum(x["kind"] == MOST_COMMON for x in TEST) / len(TEST) print(f"fiction: 1600 events, train 1100, held-out {len(TEST)}; vocabulary {len(VOCAB)} words") print(f"event kind baseline (most frequent class '{MOST_COMMON}') {BASELINE:.4f}; share of events " f"with the line written in the note {sum(x['written'] for x in TEST) / len(TEST):.4f}") x = DATA[1124] print(f"note: {x['text']} -> {x['kind']}, line {x['line']}") for row in x["frame"]: print(" " + "".join(str(min(9, max(0, int(v * 10)))) for v in row)) ``` ``` fiction: 1600 events, train 1100, held-out 500; vocabulary 154 words event kind baseline (most frequent class 'ordinary') 0.4420; share of events with the line written in the note 0.2860 note: boruda yeni ek, dogu hatti -> illicit use, line 2 143245002250 114254245050 423337867534 531264793421 351236684120 231245223341 212444253203 435135111311 154587830332 622058911323 222433403040 241331231422 ``` The event kind baseline is **0.4420** and is the same as the number measured in the `nlp` topic. The second number sets up the problem of the lesson: the line is written in the note for only **0.2860** of the test events, while in the frame it is always visible. In the printed example the note says "dogu hatti" (east line); in the frame that band stands at the fifth and sixth columns, and the two blocks of the suspected illicit use ("illicit use") sit on top of it. ## The Joint Representation and Matching ```python FEATURES, DIM, SCALE, BATCH, CANDIDATES = 15, 12, 10.0, 25, 25 def frame_features(g): """Twelve column means, the mean, the brightest 3x3, the share above the threshold.""" return [sum(g[i][j] for i in range(SIZE)) / SIZE for j in range(SIZE)] + [ sum(sum(s) for s in g) / (SIZE * SIZE), max(sum(g[i + a][j + b] for a in range(3) for b in range(3)) / 9.0 for i in range(SIZE - 2) for j in range(SIZE - 2)), sum(1 for s in g for v in s if v > 0.55) / (SIZE * SIZE)] WORDS = [[VOCAB[s] for s in x["text"].split() if s in VOCAB] for x in DATA] FRAME_FEATURES = [frame_features(x["frame"]) for x in DATA] for k in range(FEATURES): # the scale is learned from the training set only m = sum(o[k] for o in FRAME_FEATURES[:1100]) / 1100 sd = math.sqrt(sum((o[k] - m) ** 2 for o in FRAME_FEATURES[:1100]) / 1100) for o in FRAME_FEATURES: o[k] = (o[k] - m) / sd r = rng(SEED + 77) WN = [[(r() * 2 - 1) * 0.5 for _ in range(DIM)] for _ in range(len(VOCAB))] WF = [[(r() * 2 - 1) * 0.3 for _ in range(DIM)] for _ in range(FEATURES)] def embed(i): """Projections that carry the note and the frame into the same space. MODEL.""" c = [] for v in ([sum(WN[w][d] for w in WORDS[i]) for d in range(DIM)], [sum(FRAME_FEATURES[i][k] * WF[k][d] for k in range(FEATURES)) for d in range(DIM)]): n = math.sqrt(sum(a * a for a in v)) + 1e-9 c.append(([a / n for a in v], n)) return c for step in range(800): # CONTRASTIVE OBJECTIVE start = (step * BATCH) % (1100 - BATCH) E = [embed(i) for i in range(start, start + BATCH)] U, V = [e[0][0] for e in E], [e[1][0] for e in E] dU = [[0.0] * DIM for _ in E] dV = [[0.0] * DIM for _ in E] for i in range(BATCH): # correct pair close, the other 24 pairs far s = [SCALE * sum(a * b for a, b in zip(U[i], V[j])) for j in range(BATCH)] e = [math.exp(v - max(s)) for v in s] t = sum(e) for j in range(BATCH): k = SCALE * (e[j] / t - (1.0 if i == j else 0.0)) for d in range(DIM): dU[i][d] += k * V[j][d] dV[j][d] += k * U[i][d] for p, i in enumerate(range(start, start + BATCH)): for side, (dv, W) in enumerate(((dU[p], WN), (dV[p], WF))): u, n = E[p][side] dot = sum(a * b for a, b in zip(u, dv)) grad = [(dv[d] - u[d] * dot) / n for d in range(DIM)] for w in (WORDS[i] if side == 0 else range(FEATURES)): for d in range(DIM): W[w][d] -= 0.15 * grad[d] * (1.0 if side == 0 else FRAME_FEATURES[i][w]) JOINT = [[embed(i)[y][0] for y in (0, 1)] for i in range(1600)] correct = ceiling = 0 for start in range(1100, 1600, CANDIDATES): room = list(range(start, start + CANDIDATES)) for i in room: correct += i == max(room, key=lambda j: sum(a * b for a, b in zip(JOINT[i][0], JOINT[j][1]))) same = [j for j in room if DATA[j]["kind"] == DATA[i]["kind"]] ceiling += 1.0 / len([j for j in same if DATA[j]["line"] == DATA[i]["line"]] if DATA[i]["written"] else same) print(f"matching, {CANDIDATES} candidates per room, {len(TEST)} held-out events; " f"random matching {1.0 / CANDIDATES:.4f}") print(f"joint representation {correct / len(TEST):.4f}, ceiling of the fiction " f"{ceiling / len(TEST):.4f}") ``` ``` matching, 25 candidates per room, 500 held-out events; random matching 0.0400 joint representation 0.1660, ceiling of the fiction 0.2659 ``` The baseline is **0.0400**: picking one of twenty-five candidates at random. The joint representation gives **0.1660**, more than four times the baseline. The third number says why this metric cannot be read alone: the information the fiction carries in both modalities consists of the kind and, if it is written in the note, the line, and the ceiling computed from those groups is **0.2659**. Compared with 1.0000, 0.1660 looks like a failure; compared with 0.2659, **62% of what can be learned** has been learned. The ceiling of a metric is a number that must be measured as much as its baseline. The source of the difference is the objective: both projections are linear and do nothing on their own. When the contrastive objective pulls the correct pair closer and pushes the wrong ones apart, the projections are forced to turn whatever the two modalities share in the same direction. ## One Modality on Its Own ```python def class_mean(X): """Cosine similarity to the mean vector of each class. MODEL.""" def unit(v): n = math.sqrt(sum(a * a for a in v)) + 1e-9 return [a / n for a in v] M = {} for t in KINDS: d = [i for i in range(1100) if DATA[i]["kind"] == t] M[t] = unit([sum(X[i][k] for i in d) / len(d) for k in range(len(X[0]))]) return sum(max(KINDS, key=lambda t: sum(a * b for a, b in zip(unit(X[i]), M[t]))) == DATA[i]["kind"] for i in range(1100, 1600)) / len(TEST) BAG = [[1.0 if k in WORDS[i] else 0.0 for k in range(len(VOCAB))] for i in range(1600)] print(f"event kind decision, with class mean; baseline {BASELINE:.4f}") for name, X in (("note only", BAG), ("frame only", FRAME_FEATURES), ("joint representation", [JOINT[i][0] + JOINT[i][1] for i in range(1600)])): print(f" {name:<21} {class_mean(X):.4f}") ``` ``` event kind decision, with class mean; baseline 0.4420 note only 0.8940 frame only 0.6140 joint representation 0.8660 ``` The three rows were read with the same procedure and against the same baseline; only the representation changes. Note only **0.8940**, frame only **0.6140**: both are above the baseline, but the frame carries only a part of the event kind, because two of the four kinds carry no block in the frame. The third row says what the joint representation is and what it is not: **0.8660**, only 0.0280 below the note only. This representation was learned **for matching** and beat the baseline fourfold there; on the event kind it cannot reach what the note gives on its own. Bringing two modalities together does not pay off automatically, and the quality of a representation depends on the task in which it is measured. ## Summary - The two modalities do not carry the same information equally: the frame always shows the line, while only 0.2860 of the notes write it. - The joint representation gives 0.1660 at matching; the baseline is 0.0400 and the ceiling of the fiction is 0.2659, that is, 62% of what can be learned. - On the event kind, the baseline is 0.4420, note only 0.8940, frame only 0.6140, joint representation 0.8660: a representation that beats the baseline fourfold on the task it was trained for can fall below a single modality on another task. ## Course Wrap-Up The accounting of fourteen lessons is in a single table, and every number was read from its own lesson. **The rows do not share a common unit**; the last column writes the metric and the direction of each row. **The difference is given with a single sign convention: the method's number minus the domain baseline.** When the direction is small is good, a plus difference says the baseline won. | Lesson | Domain baseline | Method's number | Difference | Metric | |---|---|---|---|---| | Text 01 | 748 | 641 | −107 | vocabulary size, small is good | | Text 02 | 0.7383 | 0.8217 | +0.0833 | accuracy, big is good | | Text 03 | 0.4938 | 0.6049 | +0.1111 | neighborhood hit, big is good | | Text 04 | 0.5061 | 0.9824 | +0.4763 | token accuracy, big is good | | Text 05 | 0.4420 | 0.9620 | +0.5200 | document accuracy, big is good | | Text 06 | 0.4420 | 0.7480 | +0.3060 | document accuracy, big is good | | Text 07 | 0.7777 | 0.9445 | +0.1668 | balanced overlap, big is good | | Text 08 | 0.3970 | 0.8378 | +0.4408 | token accuracy, big is good | | Vision 01 | 0.4875 | 0.9250 | +0.4375 | accuracy, big is good | | Vision 02 | 0.5500 | 0.6687 | +0.1187 | accuracy, big is good | | Vision 03 | 0.3011 | 0.9076 | +0.6065 | mean overlap, big is good | | Vision 04 | 0.8731 | 0.9851 | +0.1120 | pixel accuracy, big is good | | Vision 05 | 0 | 99 | +99 | instability, small is good | | Vision 06 | 0.0400 | 0.1660 | +0.1260 | matching, big is good | **The rule of the course is this: a domain metric cannot be read without writing the number of the dumbest procedure of that domain on the same metric; a number whose domain baseline is not written is an unmeasured number.** The second claim of the course is read from the same table. In seven lessons the domain baseline turned out surprisingly high: 0.8731 in segmentation, 0.7777 in text 07, 0.7383 in text 02, 0.5061 in text 04 — while the same procedure gets 0.0000 at the entity level — and 0.5500 in vision 01 and 02, and in vision 05 an instability of 0, which is an unbeatable number on this metric. In two lessons a procedure that had learned nothing came in **first**: in text 07 the procedure that copies the input, in text 08 the procedure that marks every token, both 1.0000 in the recall direction of the overlap. The metrics that hide this come in four forms: **accuracy read as a share on an imbalanced class**, **accuracy read in a unit other than the one counted**, **overlap read in one direction only**, and **stability that a constant output wins by itself**. In thirty-eight lessons — fourteen of them specific to the domain — the model looked at an input and produced an output, and the correct answer was either at hand or a metric described it. What is learned in a setting where the model learns the **result of the decision only after it is made** and the decision itself **changes the next input** was never asked. The next course, Reinforcement Learning and Recommender Systems, opens with this question.