Multimodal Models

Five lessons measured images and the first eight lessons measured text on its own; yet the note the field crew files for an event and the coverage frame of the same event belong to the same event.

The measure of this lesson is not an accuracy but matching: given a note, can we pick the right one among the candidate frames. Its baseline is random matching; its ceiling is not 1, it is whatever information the two modalities share, and that is computed inside the lesson.

• CV48. The event record is fiction and has two modalities. The note generator is the same as in the nlp topic, the frame is the same as the 12x12 gray-level fiction of lesson 04. Seed 20260218, split 1,100/500.
• CV49. The two modalities do not carry the same information equally. The frame carries the event kind with a pattern (no block, an unreadable band, one block, two blocks) and the line with a faint band; the note carries the kind with words but writes the line only some of the time. There is noise on both sides: a borrowed side record in the note, and in the frame a missing pattern and an unexpected bright spot.
• CV50. The matching metric is 25 candidates per room. The ceiling is computed from indistinguishable candidates: two events that carry the same kind and, if the line is written in the note, the same line cannot be told apart.
• CV51. The joint representation is two linear projections: one from the vocabulary, the other from the frame, into the same space. It is learned with a contrastive objective: the similarity of the correct pair is raised, and those of the other twenty-four pairs in the batch are lowered.
• CV52. The event kind decision is read on three representations with the same procedure: cosine similarity to the class mean, with the most frequent class as the baseline. The vocabulary, the scale, and the weights are learned from training only; no library is called.

The Two Modalities of the Same Event

# event.py — MODEL: Event record fiction: the NOTE of the same event (from the nlp topic) and its
# COVERAGE FRAME (the 12x12 gray-level fiction of lesson 04). Seed 20260218.
import math

SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12
BACKGROUND, BLOCK, NOISE = 0.30, 0.74, 0.30
KINDS = ("ordinary", "unreachable", "meter fault", "illicit use")
SHARE, PATTERN = (0.46, 0.22, 0.19, 0.13), (0, 0, 1, 2)
PHRASES = [s.split("|") for s in (
    "okuma normal|okuma normaldi|sayac erisimi saglandi|sayaca erisildi|olcum tekrarlandi|"
    "olcumu yineledik|vana kontrol edildi|vanada sorun yok|kapak temiz|kapakta iz yok|"
    "gosterge okunakli|gostergede rakam net",
    "abone evde degildi|abone bulunamadi|erisim saglanamadi|erisim yok|kapi zili calismiyor|"
    "zil calismadi|bahce kapisi kilitli|bahce kapisi kapali|bodrum katina inilemedi|"
    "bodrum kilitli|adres bulunamadi|adres eksik",
    "gosterge donmuyor|gostergede hareket yok|sayac cami kirik|sayacin cami catlak|"
    "rakam okunmuyor|rakamlar silik|ibre sabit kalmis|ibre kilitli|mekanizma takilmis|"
    "mekanizmada takilma|sayac durmus|sayac calismiyor",
    "muhur izi var|muhurde iz goruldu|muhur kopmus|muhurler kopuk|baglanti degistirilmis|"
    "baglantida degisiklik|kelepce gevsek|kelepcede gevseme|boru ek yeri yeni|boruda yeni ek|"
    "sayac ters takilmis|sayaci ters takmislar")]
EXTRA = ("arac park halinde|kapak kirli|hava yagisli|adres teyit edildi|"
         "not kisa tutuldu|ikinci ziyaret|bahce icinde|kaldirim uzerinde").split("|")
LINES = "kuzey hatti|guney hatti|dogu hatti|bati hatti|merkez hatti".split("|")


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def event(number):
    """Two modalities of the same event: a short Turkish field note and a 12x12 coverage frame."""
    r, k, t = rng(SEED + 41000 + number), 0.0, 0
    u = r()
    for i, p in enumerate(SHARE):
        k += p
        if u < k:
            t = i
            break
    parts = [PHRASES[t][int(r() * 12)]]
    if r() < 0.55:
        parts.append(PHRASES[t][int(r() * 12)])
    if r() < 0.34:                            # side record borrowed from another kind
        parts.append(PHRASES[int(r() * 4)][int(r() * 12)])
    if r() < 0.62:
        parts.append(EXTRA[int(r() * 8)])
    line, written = int(r() * 5), r() < 0.30  # the line is written in the note only some of the time
    if written:
        parts.append(LINES[line])
    q = rng(SEED + 600000 + number)
    g = [[BACKGROUND + (q() + q() + q() - 1.5) * NOISE for _ in range(SIZE)] for _ in range(SIZE)]
    for i in range(SIZE):                     # the line that is read appears as a faint band
        g[i][line * 2] += 0.16
        g[i][line * 2 + 1] += 0.16

    def block(sy, sx):
        for i in range(sy, min(SIZE, sy + 2 + int(q() * 2))):
            for j in range(sx, min(SIZE, sx + 2 + int(q() * 2))):
                g[i][j] = BLOCK + (q() + q() + q() - 1.5) * NOISE
    for b in range(PATTERN[t]):
        if q() >= 0.12:                       # in some frames the pattern is missing
            block(int(q() * (SIZE - 3)), line * 2 + b * 2)
    if t == 1:                                # not reached: unreadable band
        for i in range(SIZE):
            g[i][line * 2] = g[i][line * 2 + 1] = 0.02 + q() * 0.05
    if q() < 0.15:                            # unexpected bright spot elsewhere
        block(int(q() * (SIZE - 3)), int(q() * (SIZE - 3)))
    return {"text": ", ".join(parts), "kind": KINDS[t], "line": line, "written": written,
            "frame": g}


DATA = [event(i) for i in range(1600)]
TEST = DATA[1100:]
count, VOCAB, seen = {}, {}, {}
for x in DATA[:1100]:
    count[x["kind"]] = count.get(x["kind"], 0) + 1
    for s in set(x["text"].split()):
        seen[s] = seen.get(s, 0) + 1
for s in sorted(seen):
    if seen[s] >= 2:
        VOCAB[s] = len(VOCAB)
MOST_COMMON = max(count, key=count.get)
BASELINE = sum(x["kind"] == MOST_COMMON for x in TEST) / len(TEST)
print(f"fiction: 1600 events, train 1100, held-out {len(TEST)}; vocabulary {len(VOCAB)} words")
print(f"event kind baseline (most frequent class '{MOST_COMMON}') {BASELINE:.4f}; share of events "
      f"with the line written in the note {sum(x['written'] for x in TEST) / len(TEST):.4f}")
x = DATA[1124]
print(f"note: {x['text']}  ->  {x['kind']}, line {x['line']}")
for row in x["frame"]:
    print("   " + "".join(str(min(9, max(0, int(v * 10)))) for v in row))

fiction: 1600 events, train 1100, held-out 500; vocabulary 154 words
event kind baseline (most frequent class 'ordinary') 0.4420; share of events with the line written in the note 0.2860
note: boruda yeni ek, dogu hatti  ->  illicit use, line 2
   143245002250
   114254245050
   423337867534
   531264793421
   351236684120
   231245223341
   212444253203
   435135111311
   154587830332
   622058911323
   222433403040
   241331231422

The event kind baseline is 0.4420 and is the same as the number measured in the nlp topic. The second number sets up the problem of the lesson: the line is written in the note for only 0.2860 of the test events, while in the frame it is always visible. In the printed example the note says “dogu hatti” (east line); in the frame that band stands at the fifth and sixth columns, and the two blocks of the suspected illicit use (“illicit use”) sit on top of it.

The Joint Representation and Matching

FEATURES, DIM, SCALE, BATCH, CANDIDATES = 15, 12, 10.0, 25, 25


def frame_features(g):
    """Twelve column means, the mean, the brightest 3x3, the share above the threshold."""
    return [sum(g[i][j] for i in range(SIZE)) / SIZE for j in range(SIZE)] + [
        sum(sum(s) for s in g) / (SIZE * SIZE),
        max(sum(g[i + a][j + b] for a in range(3) for b in range(3)) / 9.0
            for i in range(SIZE - 2) for j in range(SIZE - 2)),
        sum(1 for s in g for v in s if v > 0.55) / (SIZE * SIZE)]


WORDS = [[VOCAB[s] for s in x["text"].split() if s in VOCAB] for x in DATA]
FRAME_FEATURES = [frame_features(x["frame"]) for x in DATA]
for k in range(FEATURES):                     # the scale is learned from the training set only
    m = sum(o[k] for o in FRAME_FEATURES[:1100]) / 1100
    sd = math.sqrt(sum((o[k] - m) ** 2 for o in FRAME_FEATURES[:1100]) / 1100)
    for o in FRAME_FEATURES:
        o[k] = (o[k] - m) / sd
r = rng(SEED + 77)
WN = [[(r() * 2 - 1) * 0.5 for _ in range(DIM)] for _ in range(len(VOCAB))]
WF = [[(r() * 2 - 1) * 0.3 for _ in range(DIM)] for _ in range(FEATURES)]


def embed(i):
    """Projections that carry the note and the frame into the same space. MODEL."""
    c = []
    for v in ([sum(WN[w][d] for w in WORDS[i]) for d in range(DIM)],
              [sum(FRAME_FEATURES[i][k] * WF[k][d] for k in range(FEATURES))
               for d in range(DIM)]):
        n = math.sqrt(sum(a * a for a in v)) + 1e-9
        c.append(([a / n for a in v], n))
    return c


for step in range(800):                       # CONTRASTIVE OBJECTIVE
    start = (step * BATCH) % (1100 - BATCH)
    E = [embed(i) for i in range(start, start + BATCH)]
    U, V = [e[0][0] for e in E], [e[1][0] for e in E]
    dU = [[0.0] * DIM for _ in E]
    dV = [[0.0] * DIM for _ in E]
    for i in range(BATCH):                    # correct pair close, the other 24 pairs far
        s = [SCALE * sum(a * b for a, b in zip(U[i], V[j])) for j in range(BATCH)]
        e = [math.exp(v - max(s)) for v in s]
        t = sum(e)
        for j in range(BATCH):
            k = SCALE * (e[j] / t - (1.0 if i == j else 0.0))
            for d in range(DIM):
                dU[i][d] += k * V[j][d]
                dV[j][d] += k * U[i][d]
    for p, i in enumerate(range(start, start + BATCH)):
        for side, (dv, W) in enumerate(((dU[p], WN), (dV[p], WF))):
            u, n = E[p][side]
            dot = sum(a * b for a, b in zip(u, dv))
            grad = [(dv[d] - u[d] * dot) / n for d in range(DIM)]
            for w in (WORDS[i] if side == 0 else range(FEATURES)):
                for d in range(DIM):
                    W[w][d] -= 0.15 * grad[d] * (1.0 if side == 0 else FRAME_FEATURES[i][w])

JOINT = [[embed(i)[y][0] for y in (0, 1)] for i in range(1600)]
correct = ceiling = 0
for start in range(1100, 1600, CANDIDATES):
    room = list(range(start, start + CANDIDATES))
    for i in room:
        correct += i == max(room, key=lambda j: sum(a * b for a, b in zip(JOINT[i][0],
                                                                          JOINT[j][1])))
        same = [j for j in room if DATA[j]["kind"] == DATA[i]["kind"]]
        ceiling += 1.0 / len([j for j in same if DATA[j]["line"] == DATA[i]["line"]]
                             if DATA[i]["written"] else same)
print(f"matching, {CANDIDATES} candidates per room, {len(TEST)} held-out events; "
      f"random matching {1.0 / CANDIDATES:.4f}")
print(f"joint representation {correct / len(TEST):.4f}, ceiling of the fiction "
      f"{ceiling / len(TEST):.4f}")

matching, 25 candidates per room, 500 held-out events; random matching 0.0400
joint representation 0.1660, ceiling of the fiction 0.2659

The baseline is 0.0400: picking one of twenty-five candidates at random. The joint representation gives 0.1660, more than four times the baseline. The third number says why this metric cannot be read alone: the information the fiction carries in both modalities consists of the kind and, if it is written in the note, the line, and the ceiling computed from those groups is 0.2659. Compared with 1.0000, 0.1660 looks like a failure; compared with 0.2659, 62% of what can be learned has been learned. The ceiling of a metric is a number that must be measured as much as its baseline.

The source of the difference is the objective: both projections are linear and do nothing on their own. When the contrastive objective pulls the correct pair closer and pushes the wrong ones apart, the projections are forced to turn whatever the two modalities share in the same direction.

One Modality on Its Own

def class_mean(X):
    """Cosine similarity to the mean vector of each class. MODEL."""
    def unit(v):
        n = math.sqrt(sum(a * a for a in v)) + 1e-9
        return [a / n for a in v]
    M = {}
    for t in KINDS:
        d = [i for i in range(1100) if DATA[i]["kind"] == t]
        M[t] = unit([sum(X[i][k] for i in d) / len(d) for k in range(len(X[0]))])
    return sum(max(KINDS, key=lambda t: sum(a * b for a, b in zip(unit(X[i]), M[t])))
               == DATA[i]["kind"] for i in range(1100, 1600)) / len(TEST)


BAG = [[1.0 if k in WORDS[i] else 0.0 for k in range(len(VOCAB))] for i in range(1600)]
print(f"event kind decision, with class mean; baseline {BASELINE:.4f}")
for name, X in (("note only", BAG), ("frame only", FRAME_FEATURES),
                ("joint representation", [JOINT[i][0] + JOINT[i][1] for i in range(1600)])):
    print(f"  {name:<21} {class_mean(X):.4f}")

event kind decision, with class mean; baseline 0.4420
  note only             0.8940
  frame only            0.6140
  joint representation  0.8660

The three rows were read with the same procedure and against the same baseline; only the representation changes. Note only 0.8940, frame only 0.6140: both are above the baseline, but the frame carries only a part of the event kind, because two of the four kinds carry no block in the frame.

The third row says what the joint representation is and what it is not: 0.8660, only 0.0280 below the note only. This representation was learned for matching and beat the baseline fourfold there; on the event kind it cannot reach what the note gives on its own. Bringing two modalities together does not pay off automatically, and the quality of a representation depends on the task in which it is measured.

Summary

• The two modalities do not carry the same information equally: the frame always shows the line, while only 0.2860 of the notes write it.
• The joint representation gives 0.1660 at matching; the baseline is 0.0400 and the ceiling of the fiction is 0.2659, that is, 62% of what can be learned.
• On the event kind, the baseline is 0.4420, note only 0.8940, frame only 0.6140, joint representation 0.8660: a representation that beats the baseline fourfold on the task it was trained for can fall below a single modality on another task.

Course Wrap-Up

The accounting of fourteen lessons is in a single table, and every number was read from its own lesson. The rows do not share a common unit; the last column writes the metric and the direction of each row. The difference is given with a single sign convention: the method’s number minus the domain baseline. When the direction is small is good, a plus difference says the baseline won.

The rule of the course is this: a domain metric cannot be read without writing the number of the dumbest procedure of that domain on the same metric; a number whose domain baseline is not written is an unmeasured number.

The second claim of the course is read from the same table. In seven lessons the domain baseline turned out surprisingly high: 0.8731 in segmentation, 0.7777 in text 07, 0.7383 in text 02, 0.5061 in text 04 — while the same procedure gets 0.0000 at the entity level — and 0.5500 in vision 01 and 02, and in vision 05 an instability of 0, which is an unbeatable number on this metric. In two lessons a procedure that had learned nothing came in first: in text 07 the procedure that copies the input, in text 08 the procedure that marks every token, both 1.0000 in the recall direction of the overlap. The metrics that hide this come in four forms: accuracy read as a share on an imbalanced class, accuracy read in a unit other than the one counted, overlap read in one direction only, and stability that a constant output wins by itself.

In thirty-eight lessons — fourteen of them specific to the domain — the model looked at an input and produced an output, and the correct answer was either at hand or a metric described it. What is learned in a setting where the model learns the result of the decision only after it is made and the decision itself changes the next input was never asked. The next course, Reinforcement Learning and Recommender Systems, opens with this question.