--- title: 'Image Classification' source: 'https://academia.sh/en/courses/nlp-and-vision/image-classification' course: 'Natural Language Processing and Computer Vision' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Trains the same network on two inputs, 144 pixels and 36 pooled numbers, and reads the parameter count and the held-out set figure side by side' - 'Produces a pretrained model by hand from a fictional source task and shows that it never sees a single label of the target task' - 'Compares freezing and fine-tuning with training from scratch at three different label counts and measures the difference transfer adds together with the run gap' --- # Image Classification Measuring end-to-end training and transfer learning against the domain baseline: while the most frequent class reads 0.5312 and the mean image 0.5500, the network with 1,169 parameters reads 0.6000 in both runs on 144 pixels, and the network with 305 parameters reads 0.6687 and 0.7063 on 36 pooled numbers; both memorize the training set completely. With twenty labels, training from scratch stays below the domain baseline at 0.5188 and 0.5250, while freezing a pretrained model produced from a fictional source task gives 0.6312 and 0.5500, and fine-tuning gives 0.5938 and 0.5625. At sixty labels the advantage closes, and at one hundred sixty training from scratch pulls ahead with 0.6125 and 0.6875; the gap of 0.0812 between the two runs of freezing is larger than the difference transfer adds. The highest number of the previous lesson did not come from a model. On the per-image normalized representation, a rule of three lines that extracts the bounding box of the above-threshold pixels and compares its height with its width read 0.9250 on the test set. Writing that rule required knowing the geometry of the task: that the label is the block's orientation, that the block is single, that the noise stays below the threshold. This lesson takes that knowledge back. The rule is deleted, a model that learns from labeled examples is put in its place, and two questions are asked. The first is how far the model exceeds the domain baseline. The second is what happens when the number of labels drops to twenty — and how many labeled examples the representation carried by a model trained **on another task** is worth. - **CV9. The image fiction is the same as in the previous lesson:** 12x12 single-channel gray-level image, a single rectangular block, the label is the block's orientation, seed 20260218, split 160/80/160. - **CV10. The domain baseline is two procedures:** the most frequent class and the **mean image**. The second is measured on raw pixels and reads 0.5500 on this set. - **CV11. The model is a small network with one hidden layer** and is written inside the lesson. Eight hidden units, sigmoid output, update per example, one hundred epochs. - **CV12. Every setup is run twice.** In the second run only the initial values and the shuffling order change; the data, the split, the epoch count, and the learning step are the same. - **CV13. Pooling and convolution are not built here.** 2x2 mean pooling is an operation measured in the Deep Learning course, and in this lesson it is used only as an **input representation**. - **CV14. The source task is fiction and separate from the target task:** another 240 images, the question "is the block wide". The pretrained model is produced from this task **by our own hands**; it sees no label and no image of the target task. - **CV15. Freezing** is taking the hidden layer from the source and leaving it unchanged; **fine-tuning** is continuing to train the same layer on the target task. In both, the output layer is rebuilt. ## The Domain Baseline and the Representation ```python # data.py — MODEL: The image fiction of the previous lesson continues unchanged: a 12x12 # single-channel gray-level image, a single rectangular block, the label is the block's orientation. SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12 MIN_SIDE, MAX_SIDE, HID = 2, 6, 8 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def image(no): r = rng(SEED + 700000 + no) background, contrast = 30 + int(r() * 70), 20 + int(r() * 36) height = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) while width == height: width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1)) i = int((r() + r() + r()) / 3 * (SIZE - height + 1)) j = int((r() + r() + r()) / 3 * (SIZE - width + 1)) g = [[0] * SIZE for _ in range(SIZE)] for a in range(SIZE): for b in range(SIZE): v = background + (contrast if i <= a < i + height and j <= b < j + width else 0) v += int(r() * 61) - 30 g[a][b] = 0 if v < 0 else (255 if v > 255 else v) return g, (i, j, height, width), (1 if width > height else 0) def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s RAW = [image(i) for i in range(400)] DATA = [RAW[i] for i in shuffle(400, SEED + 40000)] TRAIN, VAL, TEST = DATA[:160], DATA[160:240], DATA[240:] def flat(g): return [v for s in g for v in s] def norm_t(g): # per-image normalization x = flat(g) m = sum(x) / len(x) sd = (sum((v - m) ** 2 for v in x) / len(x)) ** 0.5 return [(v - m) / sd for v in x] def features(g): # normalize, 2x2 mean pool x = norm_t(g) return [sum(x[(2 * a + p) * SIZE + 2 * b + q] for p in (0, 1) for q in (0, 1)) / 4 for a in range(6) for b in range(6)] def mean_image(train, test): """Domain baseline: class means, the decision is the nearest mean.""" M = {} for e in (0, 1): group = [x for x, y in train if y == e] M[e] = [sum(a[i] for a in group) / len(group) for i in range(len(group[0]))] def dist(x, e): return sum((x[i] - M[e][i]) ** 2 for i in range(len(x))) return sum(1 for x, y in test if (1 if dist(x, 1) < dist(x, 0) else 0) == y) / len(test) MAJORITY = max(sum(e for _, _, e in TEST), len(TEST) - sum(e for _, _, e in TEST)) / len(TEST) BASELINE = mean_image([(flat(g), e) for g, _, e in TRAIN], [(flat(g), e) for g, _, e in TEST]) print(f"split {len(TRAIN)}/{len(VAL)}/{len(TEST)}, input {SIZE * SIZE} pixels; " f"after 2x2 pooling {len(features(TRAIN[0][0]))} numbers") print(f"domain baseline, most frequent class: {MAJORITY:.4f}") print(f"domain baseline, mean image: {BASELINE:.4f}") ``` ``` split 160/80/160, input 144 pixels; after 2x2 pooling 36 numbers domain baseline, most frequent class: 0.5312 domain baseline, mean image: 0.5500 ``` The mean image procedure keeps the mean image of the two classes and writes a new image to whichever is closer. It stores 288 numbers in total, makes no iterations, and asks for no tuning variable. It is the dumbest procedure of the domain and reads **0.5500**; 0.0188 higher than the 0.5312 of the most frequent class. Every number from here on is read together with these two. In this lesson the validation set is not looked at. The epoch count, the learning step, and the hidden unit count are held the same in all setups; no sweep is done. The reason is what the measure is: the question asked is not "which network is best" but "how much does changing the input representation or the initial weights move the number on the same budget". If the tuning budget changed between setups, it could not be read whether the difference came from the tuning or from the representation. ## End-to-End Training ```python # network.py — MODEL: A small network with one hidden layer, written inside the lesson. # Sigmoid output, update per example, cross-entropy loss. import math def build(D, seed): r, a = rng(seed), (1 / D) ** 0.5 return [[[(r() * 2 - 1) * a for _ in range(D)] for _ in range(HID)], [0.0] * HID, [(r() * 2 - 1) * (1 / HID) ** 0.5 for _ in range(HID)], [0.0]] def forward(P, x): W, b, v, c = P h = [max(0.0, b[k] + sum(W[k][i] * x[i] for i in range(len(x)))) for k in range(HID)] z = c[0] + sum(v[k] * h[k] for k in range(HID)) return h, (1 / (1 + math.exp(-z)) if z > -30 else 0.0) def train_model(P, data, epochs, step, seed, hidden=True): W, b, v, c = P r, n, D = rng(seed), len(data), len(data[0][0]) for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: x, y = data[idx] h, p = forward(P, x) for k in range(HID): if hidden and h[k] > 0.0: dh = (p - y) * v[k] b[k] -= step * dh for i in range(D): W[k][i] -= step * dh * x[i] v[k] -= step * (p - y) * h[k] c[0] -= step * (p - y) return P def accuracy(P, data): return sum(1 for x, y in data if (forward(P, x)[1] >= 0.5) == (y == 1)) / len(data) def dataset(transform, data): return [(transform(g), e) for g, _, e in data] print(f"{'input':<22}{'parameters':>11}{'run':>7}{'train':>8}{'test':>8}") print(f"{'most frequent class':<22}{0:>11}{'-':>7}{'-':>8}{MAJORITY:>8.4f}") print(f"{'mean image':<22}{2 * SIZE * SIZE:>11}{'-':>7}{'-':>8}{BASELINE:>8.4f}") for name, transform, D in (("144 pixels", norm_t, SIZE * SIZE), ("36 numbers (pooled)", features, 36)): E, S = dataset(transform, TRAIN), dataset(transform, TEST) for t in (0, 1): P = train_model(build(D, SEED + 11 + t), E, 100, 0.05, SEED + 511 + t) print(f"{name if t == 0 else '':<22}{D * HID + 2 * HID + 1 if t == 0 else '':>11}" f"{t + 1:>7}{accuracy(P, E):>8.4f}{accuracy(P, S):>8.4f}") ``` ``` input parameters run train test most frequent class 0 - - 0.5312 mean image 288 - - 0.5500 144 pixels 1169 1 1.0000 0.6000 2 1.0000 0.6000 36 numbers (pooled) 305 1 1.0000 0.6687 2 1.0000 0.7063 ``` In all four rows the training accuracy is **1.0000**. One hundred sixty images do not resemble one another in a 144-dimensional space; the network is wide enough to memorize them one by one, and memorizing is easier than learning what the orientation is. The test numbers are the counterpart of this: **0.6000** and **0.6000**. Reducing the input from 144 numbers to 36 numbers — taking the mean of 2x2 windows — raises the test number to **0.6687** and **0.7063**. The parameter count drops from 1,169 to **305**. The difference is not an optimization gain but an assumption: pooling says "what matters is not a pixel's value on its own but its mean together with its neighbors", and this is true in this fiction, because the block is defined not by a single pixel but by an adjacent region. The gap between the two runs is **0.0376** on the pooled input. This range, opened by only the initial values changing on the same network, the same data, and the same budget, is as large as half of the 0.0875 difference pooling adds. The 0.9250 of the hand-written threshold rule is still far away, and this is not the fault of this network but of the measure itself: the rule knows the geometry of the task, while the network has to extract it from one hundred sixty examples. ## Transfer Learning ```python # transfer.py — MODEL: The pretrained model is produced by ourselves from a FICTIONAL # source task: the source task is the question "is the block wide", it sees no label of # the target task and uses none of the target images. SOURCE = [image(1000 + i) for i in range(240)] SOURCE_DATA = [(features(g), 1 if k[3] >= 4 else 0) for g, k, _ in SOURCE] PRETRAINED = [train_model(build(36, SEED + 21 + t), SOURCE_DATA, 100, 0.05, SEED + 521 + t) for t in (0, 1)] print(f"source task: {len(SOURCE_DATA)} images, share of label 1 " f"{sum(y for _, y in SOURCE_DATA) / len(SOURCE_DATA):.4f}, accuracy on the source task " f"{accuracy(PRETRAINED[0], SOURCE_DATA):.4f} and {accuracy(PRETRAINED[1], SOURCE_DATA):.4f}") def transfer(P, seed): """The hidden layer is taken from the source; the output layer is rebuilt.""" r, (W, b, _, _) = rng(seed), P return [[s[:] for s in W], b[:], [(r() * 2 - 1) * (1 / HID) ** 0.5 for _ in range(HID)], [0.0]] TRAIN36, TEST36 = dataset(features, TRAIN), dataset(features, TEST) print(f"\n{'labels':>7}{'from scratch':>19}{'frozen':>19}{'fine-tuning':>19}") print(f"{'':>7}{'run 1 run 2':>19}{'run 1 run 2':>19}{'run 1 run 2':>19}") for n in (20, 60, 160): line = f"{n:>7}" for setup in ("from scratch", "frozen", "fine-tuning"): for t in (0, 1): if setup == "from scratch": P = build(36, SEED + 31 + t) else: P = transfer(PRETRAINED[t], SEED + 41 + t) P = train_model(P, TRAIN36[:n], 100, 0.05, SEED + 531 + t, hidden=(setup != "frozen")) line += f"{accuracy(P, TEST36):>9.4f}" if t else f"{accuracy(P, TEST36):>10.4f}" print(line) print(f"domain baseline: most frequent class {MAJORITY:.4f}, mean image {BASELINE:.4f}") ``` ``` source task: 240 images, share of label 1 0.6292, accuracy on the source task 1.0000 and 0.9958 labels from scratch frozen fine-tuning run 1 run 2 run 1 run 2 run 1 run 2 20 0.5188 0.5250 0.6312 0.5500 0.5938 0.5625 60 0.6188 0.6625 0.5938 0.6250 0.6188 0.6250 160 0.6125 0.6875 0.6000 0.5625 0.6188 0.6562 domain baseline: most frequent class 0.5312, mean image 0.5500 ``` The pretrained model here is not downloaded from a repository; it is produced from a fictional source task. The source task is the question "is the block wide" on 240 images different from the target's, and this task's labels are plentiful. What the hidden layer learns there is to read the block's **horizontal extent**. With twenty labels, training from scratch reads **0.5188** and **0.5250**: **below** the 0.5312 of the most frequent class and the 0.5500 of the mean image. With the same twenty labels, transfer gives 0.6312 and 0.5500 in freezing, and 0.5938 and 0.5625 in fine-tuning. Over the means, the difference transfer adds is **+0.0687** in freezing and **+0.0562** in fine-tuning. At sixty labels the advantage closes (from scratch 0.6406, freezing 0.6094, fine-tuning 0.6219), and at one hundred sixty training from scratch pulls ahead with **0.6500** while freezing stays at 0.5812. The second half of the question — with how many examples — is read from these three rows: the advantage of transfer runs out between twenty and sixty labels. A span of forty labels is the entire value of a pretrained model in this fiction. The reason lies in what the source task teaches. The question "is the block wide" teaches the hidden layer to read the block's **horizontal** extent; the target task asks whether the horizontal extent is greater than the **vertical** extent. The source carries one of the two required measurements and does not carry the comparison. A frozen layer stands still as data grows, because what it can learn is frozen in the source task. In the transfer learning lesson of the Deep Learning course the same pattern had been measured in a table: with a baseline of 0.6444, training from scratch with twenty labels gave 0.7194, freezing 0.8417, and fine-tuning 0.8083; and freezing had stood flat in all four measurements. The pattern measured on images is the same; the differences are smaller. One number limits the reading of this table. The two runs of freezing at twenty labels are 0.6312 and 0.5500, a gap of **0.0812**. This is **larger** than the 0.0687 difference transfer adds. Writing the number of a single run would have shown transfer stronger or weaker than it is; two runs say that the sign of the difference can be trusted but its magnitude cannot. ## Summary - The domain baseline on this task is the most frequent class at 0.5312 and the mean image at 0.5500; the second stores 288 numbers and makes no iterations. - The network with 1,169 parameters reads 0.6000 in both runs on 144 pixels; the network with 305 parameters reads 0.6687 and 0.7063 on 36 pooled numbers. All four memorize the training set at 1.0000. - The 0.0875 difference pooling adds comes from the representation; the gap between the two runs of the same network is 0.0376, which is as large as half of that difference. - With twenty labels, training from scratch stays below the domain baseline at 0.5188 and 0.5250; freezing a model produced from a fictional source task adds +0.0687, and fine-tuning adds +0.0562. - At sixty labels the advantage closes, at one hundred sixty training from scratch pulls ahead at 0.6500, and freezing stays at 0.5812. - The gap of 0.0812 between the two runs of freezing at twenty labels is larger than the difference transfer adds. ## Next Step In these two lessons the question was always the same: **what is** in the image. The answer was a single number, and the model did not have to find the block, only state its orientation. Yet a box stands inside the fiction: for every image the block's row, column, height, and width are written. The next lesson changes the question — **where** is the object. This requires a new measure, because accuracy cannot say how well two boxes match. And the domain baseline changes too: what a procedure that states the same box for every image without looking scores on that measure is the most surprising number of this course.