Video and Temporal Models

The previous lesson made the pixel decision by looking at neighbors, and the neighborhood window moved the intersection over union by 0.2431. That neighborhood was in space: the eight neighbors of a pixel were inside the same frame. A coverage network produces frames not one by one but one after another; the same region is read again at certain intervals, and the readings form a sequence. The neighborhood of this sequence is in time, and the question of the lesson is this: what does a procedure that decides on every frame independently lose when it never uses time.

The loss is not seen by looking at frame accuracy. An event really starts, lasts for a while, and ends; the label sequence therefore changes two or three times. A procedure that decides independently on each frame changes its mind every time the noise changes direction. The second metric of this lesson is instability: how many times the decision changes between two consecutive frames. The metric also brings a new question, because the baseline that learns nothing gets zero on this metric.

• CV40. The temporal fiction is a short sequence of the frame of the fourth lesson. The same 12x12 gray-level coverage frame is produced twelve times for one region; the background, the block level, and the noise are the same, and the seed is 20260218.
• CV41. The event is an interval. The fiction block is present in two to five consecutive frames of the sequence; the starting frame is between the first and the seventh frame. The frame label is whether the block is present in that frame.
• CV42. There are two sources of noise. In about a third of the active frames the reading is weak and the block looks dim; in about one in seven of the inactive frames a transient bright blob appears. The first produces a miss, the second a false alarm.
• CV43. The split is 24/36 sequences, that is, there are 432 frames in the test set. The split is at the sequence level; the frames of the same sequence are not distributed to two sets.
• CV44. The domain baseline is writing the dominant class into every frame. The dominant class is “not active.”
• CV45. Two metrics are read side by side. Frame accuracy (bigger is better) and instability: how many times the decision changes across consecutive frames. In instability, small is not good; the number to read is the instability of the true label sequences, and that number is printed too.
• CV46. The frame model is linear and is fitted from three numbers per frame: the mean gray level, the mean of the brightest 3x3 window, and the ratio of above-threshold pixels. Least squares, threshold 0.5. No library is called.
• CV47. The decision that uses time uses the same scores. The model, the weights, and the frame scores do not change; only a fixed penalty per transition is added to the whole sequence, and the lowest-cost decision sequence is chosen.

The Temporal Fiction and Two Baseline Numbers

# sequence.py — MODEL: The temporal fiction is A SHORT SEQUENCE of the gray-level 12x12
# coverage frame of the fourth lesson: the same region is read twelve times and the
# event is active in some of the frames. Seed 20260218.
SEED, M32, SIZE, FRAMES = 20260218, 0xFFFFFFFF, 12, 12
BACKGROUND, BLOCK, NOISE = 0.30, 0.74, 0.30


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def sequence(no):
    """Twelve-frame reading of one region: frames and the event label per frame."""
    r = rng(SEED + 500000 + no)
    start, length = 1 + int(r() * 7), 2 + int(r() * 4)
    h, w = 2 + int(r() * 3), 2 + int(r() * 3)
    sy, sx = int(r() * (SIZE - h + 1)), int(r() * (SIZE - w + 1))
    frames, labels = [], []
    for t in range(FRAMES):
        active = start <= t < start + length
        g = [[BACKGROUND + (r() + r() + r() - 1.5) * NOISE for _ in range(SIZE)]
             for _ in range(SIZE)]
        if active:
            strength = 0.35 if r() < 0.28 else 1.0        # the reading is weak in some frames
            for i in range(sy, sy + h):
                for j in range(sx, sx + w):
                    g[i][j] += (BLOCK - BACKGROUND) * strength
        elif r() < 0.14:                                  # transient bright blob
            a, b = int(r() * (SIZE - 1)), int(r() * (SIZE - 1))
            for i in (a, a + 1):
                for j in (b, b + 1):
                    g[i][j] += BLOCK - BACKGROUND
        frames.append(g)
        labels.append(int(active))
    return frames, labels


DATA = [sequence(i) for i in range(60)]
TRAIN, TEST = DATA[:24], DATA[24:]
TOTAL = len(TEST) * FRAMES
ACTIVE = sum(sum(e) for _, e in TEST)
BASELINE = (TOTAL - ACTIVE) / TOTAL
TRUE_INSTABILITY = sum(sum(e[t] != e[t - 1] for t in range(1, FRAMES)) for _, e in TEST)
print(f"fiction: {len(DATA)} sequences x {FRAMES} frames, train {len(TRAIN)}, test {len(TEST)}; "
      f"{ACTIVE}/{TOTAL} frames active in the test set")
print(f"baseline - writing the dominant class (not active) to every frame: "
      f"frame accuracy {BASELINE:.4f}, instability 0")
print(f"instability of the true label sequences: {TRUE_INSTABILITY} transitions "
      f"({TRUE_INSTABILITY / len(TEST):.2f} per sequence)")

fiction: 60 sequences x 12 frames, train 24, test 36; 129/432 frames active in the test set
baseline - writing the dominant class (not active) to every frame: frame accuracy 0.7014, instability 0
instability of the true label sequences: 71 transitions (1.97 per sequence)

The domain baseline gives two numbers, and the two are read in opposite directions. In frame accuracy it gets 0.7014, because only 129 of the 432 frames are active. In instability it gets 0, because it never changes its mind; a procedure that repeats a constant cannot be beaten on this metric.

The third line says how the metric is to be read. The true label sequences carry 71 transitions in total over 36 sequences, 1.97 per sequence. If an event starts and ends, that makes two transitions; for those that last until the last frame of the sequence, one transition remains. Small is not good in instability: the number sought is not zero but 71. The baseline’s zero is below the truth, and this is the clearest form of the fact that the metric cannot be read on its own.

The Independent Frame Decision

def features(g):
    """Three numbers per frame: mean gray level, brightest 3x3 window, above-threshold ratio."""
    total = sum(sum(s) for s in g)
    brightest = 0.0
    for i in range(SIZE - 2):
        for j in range(SIZE - 2):
            p = sum(g[i + a][j + b] for a in range(3) for b in range(3)) / 9.0
            if p > brightest:
                brightest = p
    above = sum(1 for s in g for v in s if v > 0.55) / (SIZE * SIZE)
    return [1.0, total / (SIZE * SIZE), brightest, above]


def solve(A, b):
    n, x = len(b), [0.0] * len(b)
    M = [A[i][:] + [b[i]] for i in range(n)]
    for k in range(n):
        p = max(range(k, n), key=lambda r: abs(M[r][k]))
        M[k], M[p] = M[p], M[k]
        for r in range(k + 1, n):
            f = M[r][k] / M[k][k]
            for c in range(k, n + 1):
                M[r][c] -= f * M[k][c]
    for k in range(n - 1, -1, -1):
        x[k] = (M[k][n] - sum(M[k][c] * x[c] for c in range(k + 1, n))) / M[k][k]
    return x


def train_model(dataset):
    """Per-frame linear decision function; fitted by least squares. MODEL."""
    A, b = [[0.0] * 4 for _ in range(4)], [0.0] * 4
    for frames, labels in dataset:
        for g, y in zip(frames, labels):
            x = features(g)
            for p in range(4):
                b[p] += x[p] * y
                for q in range(4):
                    A[p][q] += x[p] * x[q]
    return solve(A, b)


W = train_model(TRAIN)
SCORES = [[sum(a * c for a, c in zip(W, features(g))) for g in frames]
          for frames, _ in TEST]
INDEPENDENT = [[1 if s >= 0.5 else 0 for s in d] for d in SCORES]


def measure(decisions):
    correct = sum(k == y for kd, (_, e) in zip(decisions, TEST) for k, y in zip(kd, e))
    flips = sum(sum(kd[t] != kd[t - 1] for t in range(1, FRAMES)) for kd in decisions)
    return correct / TOTAL, flips


d, fl = measure(INDEPENDENT)
print(f"independent frame decision: frame accuracy {d:.4f}, instability {fl} transitions "
      f"({fl / len(TEST):.2f} per sequence)")
print(f"difference from the baseline: in accuracy {d - BASELINE:+.4f}, in instability {fl - 0:+d}")

independent frame decision: frame accuracy 0.8634, instability 99 transitions (2.75 per sequence)
difference from the baseline: in accuracy +0.1620, in instability +99

On the first metric the model passes the baseline by 0.1620, and this says that the per-frame decision really learned something. On the second metric the same model is 99 transitions above the baseline. The place to read the number is the truth’s 71: the model changes its mind 28 more times than reality changes, 39% more in ratio. Looking per sequence, where there are 1.97 transitions in reality, there are 2.75 transitions in the decision.

This excess hardly shows up in frame accuracy. In a sequence, changing its mind on a single frame in the middle of the event takes only 1/432 away from accuracy, but it adds two transitions to instability. In a coverage network the counterpart of this difference is concrete: it means that two separate alerts are produced for the same region in two consecutive readings, and for the party that reads the alert this is more expensive than a loss of accuracy.

The Decision That Uses Time with a Transition Penalty

def penalized_path(score, penalty):
    """The decision that uses time: frame cost + a fixed penalty per transition."""
    cost = [[(score[0] - s) ** 2 for s in (0, 1)]]
    back = [[0, 0]]
    for t in range(1, len(score)):
        new, trace = [0.0, 0.0], [0, 0]
        for s in (0, 1):
            best, arg = None, 0
            for p in (0, 1):
                m = cost[t - 1][p] + (0.0 if p == s else penalty)
                if best is None or m < best:
                    best, arg = m, p
            new[s], trace[s] = best + (score[t] - s) ** 2, arg
        cost.append(new)
        back.append(trace)
    s = 0 if cost[-1][0] <= cost[-1][1] else 1
    path = [s]
    for t in range(len(score) - 1, 0, -1):
        s = back[t][s]
        path.append(s)
    return path[::-1]


print(f"{'penalty':>8}{'frame accuracy':>16}{'instability':>13}{'per sequence':>13}")
for penalty in (0.00, 0.05, 0.15, 0.40, 1.00, 4.00):
    d, fl = measure([penalized_path(score, penalty) for score in SCORES])
    print(f"{penalty:>8.2f}{d:>16.4f}{fl:>13}{fl / len(TEST):>13.2f}")
print(f"{'truth':>8}{1.0:>16.4f}{TRUE_INSTABILITY:>13}"
      f"{TRUE_INSTABILITY / len(TEST):>13.2f}")
print(f"{'baseline':>8}{BASELINE:>16.4f}{0:>13}{0.0:>13.2f}")
TEMPORAL = [penalized_path(score, 0.15) for score in SCORES]
worst = max(range(len(TEST)),
            key=lambda i: sum(INDEPENDENT[i][t] != INDEPENDENT[i][t - 1] for t in range(1, FRAMES)))
print(f"test sequence that changed its mind the most (at position {worst + 1}):")
for name, d in (("truth", TEST[worst][1]), ("independent", INDEPENDENT[worst]),
                ("temporal", TEMPORAL[worst])):
    print(f"   {name:<11} " + "".join(str(v) for v in d))

 penalty  frame accuracy  instability per sequence
    0.00          0.8634           99         2.75
    0.05          0.8657           86         2.39
    0.15          0.8611           71         1.97
    0.40          0.8472           45         1.25
    1.00          0.7801           21         0.58
    4.00          0.7014            1         0.03
   truth          1.0000           71         1.97
baseline          0.7014            0         0.00
test sequence that changed its mind the most (at position 18):
   truth       000111110000
   independent 000101100010
   temporal    000101100000

The first row of the table is the numbers of the second section: zero penalty is the independent frame decision itself. At zero penalty every frame is rounded to the value its own score is closest to, and the sequence plays no role. As the penalty grows, the same scores stay the same — the model, the weights, and the frame scores never change — but the decisions change.

So where the difference comes from can be said precisely. The representation did not change: the same three numbers per frame. The model did not change: the same linear function, the same weights. The only thing that changed is the assumption that the decision rarely changes over time, and that assumption was entered with a single number, the transition penalty. Six rows are six separate outputs of the same model.

The penalty itself is an assumption turned into a number: the cost of a transition is written in the same unit as the cost of being wrong on a frame. A penalty of 0.15 means accepting a transition only if the frame evidence that justifies that transition is strong enough. This is an indirect assumption about how long the event lasts, and in the fiction the events last between two and five frames.

The trade-off is read in three places. Accuracy rises to its highest value, 0.8657, at a penalty of 0.05, and instability falls from 99 to 86; a small penalty gains both accuracy and stability. At a penalty of 0.15 instability settles on 71, the truth’s number, and the price of this is giving up 0.0046 from the highest accuracy. When the penalty keeps growing, accuracy falls: 0.8472 at 0.40, 0.7801 at 1.00.

The last row pays off the second claim of the course for this lesson. At a penalty of 4.00 the decision sequence falls almost to a constant — 1 transition over 36 sequences — and the frame accuracy falls to 0.7014, the baseline’s own number. Here the domain baseline is not a separate procedure but the extreme value of the penalty; the temporal correction applied with an infinite penalty gives the same output as writing the dominant class into every frame. A setting that tries to reduce instability alone ends up at the procedure that has learned nothing.

The last three lines show this in a single sequence. The true sequence carries an event of five frames; the independent decision misses one frame in the middle of the event and gives a false alarm on one frame after the event ends, six transitions. The decision that uses time erases the false alarm and falls to four transitions, but it cannot fill the gap in the middle: the score in that frame is low enough, and a penalty of 0.15 is not enough to flip it. A transition penalty smooths the decision; it does not bring back the missed frame.

Summary

• The per-frame decision has two metrics: frame accuracy, and instability, which counts how many times the decision changes its mind across consecutive frames. The baseline gets 0.7014 on the first and 0 on the second.
• In instability, small is not good; the number to read is the instability of the true label sequences, and that number is 71 transitions over 36 sequences.
• The independent frame decision produces 99 transitions with 0.8634 accuracy: 28 more than the truth, 2.75 per sequence instead of 1.97. This excess is almost invisible in frame accuracy.
• The transition penalty produces six separate outputs without changing the representation or the model; at 0.05 accuracy rises to its highest, 0.8657, and at 0.15 instability settles on the truth’s 71.
• At a penalty of 4.00 the decision falls to the baseline itself (0.7014 and 1 transition): on this metric the domain baseline is not a separate procedure but the extreme value of the penalty.

Next Step

In five lessons the input was always of the same kind: a gray-level grid, as a single frame or as a short sequence. In the first eight lessons of this course the input was a text, and the same fiction came from the notes of the measurement network. The two modalities have so far been measured separately; yet the note that the field team wrote for an event and the coverage frame of the same event belong to the same event. The next lesson brings the two together in a single space and asks this: how far above random matching does a representation that says whether a note and an image belong to the same event get, and how much of this does a single modality give on its own.