Image Classification

The highest number of the previous lesson did not come from a model. On the per-image normalized representation, a rule of three lines that extracts the bounding box of the above-threshold pixels and compares its height with its width read 0.9250 on the test set. Writing that rule required knowing the geometry of the task: that the label is the block’s orientation, that the block is single, that the noise stays below the threshold.

This lesson takes that knowledge back. The rule is deleted, a model that learns from labeled examples is put in its place, and two questions are asked. The first is how far the model exceeds the domain baseline. The second is what happens when the number of labels drops to twenty — and how many labeled examples the representation carried by a model trained on another task is worth.

• CV9. The image fiction is the same as in the previous lesson: 12x12 single-channel gray-level image, a single rectangular block, the label is the block’s orientation, seed 20260218, split 160/80/160.
• CV10. The domain baseline is two procedures: the most frequent class and the mean image. The second is measured on raw pixels and reads 0.5500 on this set.
• CV11. The model is a small network with one hidden layer and is written inside the lesson. Eight hidden units, sigmoid output, update per example, one hundred epochs.
• CV12. Every setup is run twice. In the second run only the initial values and the shuffling order change; the data, the split, the epoch count, and the learning step are the same.
• CV13. Pooling and convolution are not built here. 2x2 mean pooling is an operation measured in the Deep Learning course, and in this lesson it is used only as an input representation.
• CV14. The source task is fiction and separate from the target task: another 240 images, the question “is the block wide”. The pretrained model is produced from this task by our own hands; it sees no label and no image of the target task.
• CV15. Freezing is taking the hidden layer from the source and leaving it unchanged; fine-tuning is continuing to train the same layer on the target task. In both, the output layer is rebuilt.

The Domain Baseline and the Representation

# data.py — MODEL: The image fiction of the previous lesson continues unchanged: a 12x12
# single-channel gray-level image, a single rectangular block, the label is the block's orientation.
SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12
MIN_SIDE, MAX_SIDE, HID = 2, 6, 8


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def image(no):
    r = rng(SEED + 700000 + no)
    background, contrast = 30 + int(r() * 70), 20 + int(r() * 36)
    height = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1))
    width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1))
    while width == height:
        width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1))
    i = int((r() + r() + r()) / 3 * (SIZE - height + 1))
    j = int((r() + r() + r()) / 3 * (SIZE - width + 1))
    g = [[0] * SIZE for _ in range(SIZE)]
    for a in range(SIZE):
        for b in range(SIZE):
            v = background + (contrast if i <= a < i + height and j <= b < j + width else 0)
            v += int(r() * 61) - 30
            g[a][b] = 0 if v < 0 else (255 if v > 255 else v)
    return g, (i, j, height, width), (1 if width > height else 0)


def shuffle(n, seed):
    r, s = rng(seed), list(range(n))
    for i in range(n - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return s


RAW = [image(i) for i in range(400)]
DATA = [RAW[i] for i in shuffle(400, SEED + 40000)]
TRAIN, VAL, TEST = DATA[:160], DATA[160:240], DATA[240:]


def flat(g):
    return [v for s in g for v in s]


def norm_t(g):                                   # per-image normalization
    x = flat(g)
    m = sum(x) / len(x)
    sd = (sum((v - m) ** 2 for v in x) / len(x)) ** 0.5
    return [(v - m) / sd for v in x]


def features(g):                                 # normalize, 2x2 mean pool
    x = norm_t(g)
    return [sum(x[(2 * a + p) * SIZE + 2 * b + q] for p in (0, 1) for q in (0, 1)) / 4
            for a in range(6) for b in range(6)]


def mean_image(train, test):
    """Domain baseline: class means, the decision is the nearest mean."""
    M = {}
    for e in (0, 1):
        group = [x for x, y in train if y == e]
        M[e] = [sum(a[i] for a in group) / len(group) for i in range(len(group[0]))]

    def dist(x, e):
        return sum((x[i] - M[e][i]) ** 2 for i in range(len(x)))
    return sum(1 for x, y in test if (1 if dist(x, 1) < dist(x, 0) else 0) == y) / len(test)


MAJORITY = max(sum(e for _, _, e in TEST), len(TEST) - sum(e for _, _, e in TEST)) / len(TEST)
BASELINE = mean_image([(flat(g), e) for g, _, e in TRAIN],
                      [(flat(g), e) for g, _, e in TEST])
print(f"split {len(TRAIN)}/{len(VAL)}/{len(TEST)}, input {SIZE * SIZE} pixels; "
      f"after 2x2 pooling {len(features(TRAIN[0][0]))} numbers")
print(f"domain baseline, most frequent class: {MAJORITY:.4f}")
print(f"domain baseline, mean image: {BASELINE:.4f}")

split 160/80/160, input 144 pixels; after 2x2 pooling 36 numbers
domain baseline, most frequent class: 0.5312
domain baseline, mean image: 0.5500

The mean image procedure keeps the mean image of the two classes and writes a new image to whichever is closer. It stores 288 numbers in total, makes no iterations, and asks for no tuning variable. It is the dumbest procedure of the domain and reads 0.5500; 0.0188 higher than the 0.5312 of the most frequent class. Every number from here on is read together with these two.

In this lesson the validation set is not looked at. The epoch count, the learning step, and the hidden unit count are held the same in all setups; no sweep is done. The reason is what the measure is: the question asked is not “which network is best” but “how much does changing the input representation or the initial weights move the number on the same budget”. If the tuning budget changed between setups, it could not be read whether the difference came from the tuning or from the representation.

End-to-End Training

# network.py — MODEL: A small network with one hidden layer, written inside the lesson.
# Sigmoid output, update per example, cross-entropy loss.
import math


def build(D, seed):
    r, a = rng(seed), (1 / D) ** 0.5
    return [[[(r() * 2 - 1) * a for _ in range(D)] for _ in range(HID)], [0.0] * HID,
            [(r() * 2 - 1) * (1 / HID) ** 0.5 for _ in range(HID)], [0.0]]


def forward(P, x):
    W, b, v, c = P
    h = [max(0.0, b[k] + sum(W[k][i] * x[i] for i in range(len(x)))) for k in range(HID)]
    z = c[0] + sum(v[k] * h[k] for k in range(HID))
    return h, (1 / (1 + math.exp(-z)) if z > -30 else 0.0)


def train_model(P, data, epochs, step, seed, hidden=True):
    W, b, v, c = P
    r, n, D = rng(seed), len(data), len(data[0][0])
    for _ in range(epochs):
        s = list(range(n))
        for i in range(n - 1, 0, -1):
            j = int(r() * (i + 1))
            s[i], s[j] = s[j], s[i]
        for idx in s:
            x, y = data[idx]
            h, p = forward(P, x)
            for k in range(HID):
                if hidden and h[k] > 0.0:
                    dh = (p - y) * v[k]
                    b[k] -= step * dh
                    for i in range(D):
                        W[k][i] -= step * dh * x[i]
                v[k] -= step * (p - y) * h[k]
            c[0] -= step * (p - y)
    return P


def accuracy(P, data):
    return sum(1 for x, y in data if (forward(P, x)[1] >= 0.5) == (y == 1)) / len(data)


def dataset(transform, data):
    return [(transform(g), e) for g, _, e in data]


print(f"{'input':<22}{'parameters':>11}{'run':>7}{'train':>8}{'test':>8}")
print(f"{'most frequent class':<22}{0:>11}{'-':>7}{'-':>8}{MAJORITY:>8.4f}")
print(f"{'mean image':<22}{2 * SIZE * SIZE:>11}{'-':>7}{'-':>8}{BASELINE:>8.4f}")
for name, transform, D in (("144 pixels", norm_t, SIZE * SIZE), ("36 numbers (pooled)", features, 36)):
    E, S = dataset(transform, TRAIN), dataset(transform, TEST)
    for t in (0, 1):
        P = train_model(build(D, SEED + 11 + t), E, 100, 0.05, SEED + 511 + t)
        print(f"{name if t == 0 else '':<22}{D * HID + 2 * HID + 1 if t == 0 else '':>11}"
              f"{t + 1:>7}{accuracy(P, E):>8.4f}{accuracy(P, S):>8.4f}")

input                  parameters    run   train    test
most frequent class             0      -       -  0.5312
mean image                    288      -       -  0.5500
144 pixels                   1169      1  1.0000  0.6000
                                       2  1.0000  0.6000
36 numbers (pooled)           305      1  1.0000  0.6687
                                       2  1.0000  0.7063

In all four rows the training accuracy is 1.0000. One hundred sixty images do not resemble one another in a 144-dimensional space; the network is wide enough to memorize them one by one, and memorizing is easier than learning what the orientation is. The test numbers are the counterpart of this: 0.6000 and 0.6000.

Reducing the input from 144 numbers to 36 numbers — taking the mean of 2x2 windows — raises the test number to 0.6687 and 0.7063. The parameter count drops from 1,169 to 305. The difference is not an optimization gain but an assumption: pooling says “what matters is not a pixel’s value on its own but its mean together with its neighbors”, and this is true in this fiction, because the block is defined not by a single pixel but by an adjacent region.

The gap between the two runs is 0.0376 on the pooled input. This range, opened by only the initial values changing on the same network, the same data, and the same budget, is as large as half of the 0.0875 difference pooling adds. The 0.9250 of the hand-written threshold rule is still far away, and this is not the fault of this network but of the measure itself: the rule knows the geometry of the task, while the network has to extract it from one hundred sixty examples.

Transfer Learning

# transfer.py — MODEL: The pretrained model is produced by ourselves from a FICTIONAL
# source task: the source task is the question "is the block wide", it sees no label of
# the target task and uses none of the target images.
SOURCE = [image(1000 + i) for i in range(240)]
SOURCE_DATA = [(features(g), 1 if k[3] >= 4 else 0) for g, k, _ in SOURCE]
PRETRAINED = [train_model(build(36, SEED + 21 + t), SOURCE_DATA, 100, 0.05, SEED + 521 + t)
              for t in (0, 1)]
print(f"source task: {len(SOURCE_DATA)} images, share of label 1 "
      f"{sum(y for _, y in SOURCE_DATA) / len(SOURCE_DATA):.4f}, accuracy on the source task "
      f"{accuracy(PRETRAINED[0], SOURCE_DATA):.4f} and {accuracy(PRETRAINED[1], SOURCE_DATA):.4f}")


def transfer(P, seed):
    """The hidden layer is taken from the source; the output layer is rebuilt."""
    r, (W, b, _, _) = rng(seed), P
    return [[s[:] for s in W], b[:],
            [(r() * 2 - 1) * (1 / HID) ** 0.5 for _ in range(HID)], [0.0]]


TRAIN36, TEST36 = dataset(features, TRAIN), dataset(features, TEST)
print(f"\n{'labels':>7}{'from scratch':>19}{'frozen':>19}{'fine-tuning':>19}")
print(f"{'':>7}{'run 1    run 2':>19}{'run 1    run 2':>19}{'run 1    run 2':>19}")
for n in (20, 60, 160):
    line = f"{n:>7}"
    for setup in ("from scratch", "frozen", "fine-tuning"):
        for t in (0, 1):
            if setup == "from scratch":
                P = build(36, SEED + 31 + t)
            else:
                P = transfer(PRETRAINED[t], SEED + 41 + t)
            P = train_model(P, TRAIN36[:n], 100, 0.05, SEED + 531 + t,
                            hidden=(setup != "frozen"))
            line += f"{accuracy(P, TEST36):>9.4f}" if t else f"{accuracy(P, TEST36):>10.4f}"
    print(line)
print(f"domain baseline: most frequent class {MAJORITY:.4f}, mean image {BASELINE:.4f}")

source task: 240 images, share of label 1 0.6292, accuracy on the source task 1.0000 and 0.9958

 labels       from scratch             frozen        fine-tuning
            run 1    run 2     run 1    run 2     run 1    run 2
     20    0.5188   0.5250    0.6312   0.5500    0.5938   0.5625
     60    0.6188   0.6625    0.5938   0.6250    0.6188   0.6250
    160    0.6125   0.6875    0.6000   0.5625    0.6188   0.6562
domain baseline: most frequent class 0.5312, mean image 0.5500

The pretrained model here is not downloaded from a repository; it is produced from a fictional source task. The source task is the question “is the block wide” on 240 images different from the target’s, and this task’s labels are plentiful. What the hidden layer learns there is to read the block’s horizontal extent.

With twenty labels, training from scratch reads 0.5188 and 0.5250: below the 0.5312 of the most frequent class and the 0.5500 of the mean image. With the same twenty labels, transfer gives 0.6312 and 0.5500 in freezing, and 0.5938 and 0.5625 in fine-tuning. Over the means, the difference transfer adds is +0.0687 in freezing and +0.0562 in fine-tuning.

At sixty labels the advantage closes (from scratch 0.6406, freezing 0.6094, fine-tuning 0.6219), and at one hundred sixty training from scratch pulls ahead with 0.6500 while freezing stays at 0.5812. The second half of the question — with how many examples — is read from these three rows: the advantage of transfer runs out between twenty and sixty labels. A span of forty labels is the entire value of a pretrained model in this fiction.

The reason lies in what the source task teaches. The question “is the block wide” teaches the hidden layer to read the block’s horizontal extent; the target task asks whether the horizontal extent is greater than the vertical extent. The source carries one of the two required measurements and does not carry the comparison. A frozen layer stands still as data grows, because what it can learn is frozen in the source task. In the transfer learning lesson of the Deep Learning course the same pattern had been measured in a table: with a baseline of 0.6444, training from scratch with twenty labels gave 0.7194, freezing 0.8417, and fine-tuning 0.8083; and freezing had stood flat in all four measurements. The pattern measured on images is the same; the differences are smaller.

One number limits the reading of this table. The two runs of freezing at twenty labels are 0.6312 and 0.5500, a gap of 0.0812. This is larger than the 0.0687 difference transfer adds. Writing the number of a single run would have shown transfer stronger or weaker than it is; two runs say that the sign of the difference can be trusted but its magnitude cannot.

Summary

• The domain baseline on this task is the most frequent class at 0.5312 and the mean image at 0.5500; the second stores 288 numbers and makes no iterations.
• The network with 1,169 parameters reads 0.6000 in both runs on 144 pixels; the network with 305 parameters reads 0.6687 and 0.7063 on 36 pooled numbers. All four memorize the training set at 1.0000.
• The 0.0875 difference pooling adds comes from the representation; the gap between the two runs of the same network is 0.0376, which is as large as half of that difference.
• With twenty labels, training from scratch stays below the domain baseline at 0.5188 and 0.5250; freezing a model produced from a fictional source task adds +0.0687, and fine-tuning adds +0.0562.
• At sixty labels the advantage closes, at one hundred sixty training from scratch pulls ahead at 0.6500, and freezing stays at 0.5812.
• The gap of 0.0812 between the two runs of freezing at twenty labels is larger than the difference transfer adds.

Next Step

In these two lessons the question was always the same: what is in the image. The answer was a single number, and the model did not have to find the block, only state its orientation. Yet a box stands inside the fiction: for every image the block’s row, column, height, and width are written. The next lesson changes the question — where is the object. This requires a new measure, because accuracy cannot say how well two boxes match. And the domain baseline changes too: what a procedure that states the same box for every image without looking scores on that measure is the most surprising number of this course.