Image Data and Preprocessing

In eight lessons the input was a sequence. Tokens ran from left to right, a token’s neighbors were the ones before and after it, and the order itself carried information. An image has no single order: a pixel’s neighbors are to its right, to its left, above, below, and at its four corners, and none of them comes before the others. What is sought is not in a single cell either but in a pattern formed by adjacent cells, and that pattern can be found anywhere in the grid.

The second difference is quieter. In text a token is a symbol; the same token in two separate documents is exactly the same thing. A pixel, on the other hand, is a measurement: the same object is written with different numbers when the measuring conditions change. So in images normalization means transforming pixel values, and it has nothing to do with the text normalization of the Data Collection and Preparation course, which brings a string into canonical form. This is the measure of the lesson — how far the decision on how to turn a pixel into a number moves the same procedure’s number, with no model changed.

• CV1. The image is fiction. The 12x12 coverage grid of the Visualization, Reporting and Analysis at Scale course stands in here for a small single-channel, gray-level image. Each cell is a pixel, and its value is a gray level between 0 and 255. Seed 20260218.
• CV2. Background and contrast vary from image to image; this is a difficulty the fiction puts in on purpose.
• CV3. The object is fiction. A single rectangular block is planted in the grid; its sides are between 2 and 6 and differ from each other.
• CV4. The label is the block’s orientation. 1 if the width is greater than the height, 0 otherwise. The label depends not on where the block is but on its shape.
• CV5. The block gathers toward the middle of the grid. The position is chosen by the mean of three independent draws; this is the counterpart of the center bias in real image sets, and it will have a cost in the third lesson.
• CV6. The split is 160/80/160 over 400 images and is fixed throughout the topic.
• CV7. The domain baseline is two procedures: stating the most frequent class, and the mean image — class means are computed, and an image is written to whichever one it is closer to.
• CV8. Threshold and tuning decisions are read from the validation set; the sweep does not exceed nine candidates. No library is called; the procedures are written inside the lesson.

Image, Pixel, and Channel

# image.py — MODEL: Image fiction: the 12x12 coverage grid of the Visualization,
# Reporting and Analysis at Scale course stands in here for a small single-channel,
# gray-level image. Each cell is a pixel and its value is a gray level between 0 and 255.
import statistics

SEED, M32, SIZE = 20260218, 0xFFFFFFFF, 12
MIN_SIDE, MAX_SIDE = 2, 6
RAMP = " .:-=+*#%@"


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def image(no):
    """One coverage image: background, a single rectangular block, and noise."""
    r = rng(SEED + 700000 + no)
    background = 30 + int(r() * 70)
    contrast = 20 + int(r() * 36)
    height = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1))
    width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1))
    while width == height:
        width = MIN_SIDE + int(r() * (MAX_SIDE - MIN_SIDE + 1))
    i = int((r() + r() + r()) / 3 * (SIZE - height + 1))    # position gathered toward the center
    j = int((r() + r() + r()) / 3 * (SIZE - width + 1))
    g = [[0] * SIZE for _ in range(SIZE)]
    for a in range(SIZE):
        for b in range(SIZE):
            v = background + (contrast if i <= a < i + height and j <= b < j + width else 0)
            v += int(r() * 61) - 30
            g[a][b] = 0 if v < 0 else (255 if v > 255 else v)
    return g, (i, j, height, width), (1 if width > height else 0)


def shuffle(n, seed):
    r, s = rng(seed), list(range(n))
    for i in range(n - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return s


def draw(g):
    for row in g:
        print("   " + "".join(RAMP[v * len(RAMP) // 256] for v in row))


RAW = [image(i) for i in range(400)]
DATA = [RAW[i] for i in shuffle(400, SEED + 40000)]
TRAIN, VAL, TEST = DATA[:160], DATA[160:240], DATA[240:]
means = [statistics.mean(v for s in g for v in s) for g, _, _ in DATA]

g0, k0, e0 = TRAIN[0]
print(f"sample image: block row {k0[0]}, column {k0[1]}, {k0[2]}x{k0[3]}, label {e0}")
draw(g0)
print(f"{SIZE}x{SIZE} pixels = {SIZE * SIZE}, single channel; the same image with three channels "
      f"would hold {SIZE * SIZE * 3} numbers")
print(f"image means between {min(means):.1f} and {max(means):.1f}")
print(f"split {len(TRAIN)}/{len(VAL)}/{len(TEST)}")

MAJORITY = max(sum(e for _, _, e in TEST), len(TEST) - sum(e for _, _, e in TEST)) / len(TEST)
print(f"domain baseline, most frequent class: {MAJORITY:.4f}")

sample image: block row 4, column 2, 4x6, label 1
   :-:..::::::.
   ....:.::.:::
   :.::-.::.-:-
   ...-.:.::.:.
   ..=+==++:.-.
   --=---=--..:
   :.====---.-:
   :.=-==+=:..:
   .-::.:.-.:-:
   ::.:-:....::
   ::..:-::-:::
   .:-.:--:.-.:
12x12 pixels = 144, single channel; the same image with three channels would hold 432 numbers
image means between 32.9 and 108.0
split 160/80/160
domain baseline, most frequent class: 0.5312

In the sample image the block starts at the fourth row, is four rows tall and six columns wide. It is hard to make out in the drawing: because the ramp is spread over the 0-255 range, both the background and the block fall on the same two or three glyphs. The number says so too — the image means wander between 32.9 and 108.0, so in one image the inside of the block is as bright as the background itself in another.

An image of 12x12 pixels and one channel is 144 numbers; with three channels it would hold 432. A channel is a quantity measured separately for each pixel, and it multiplies the count by its own number. Stating the most frequent class on the test set gives 0.5312: this is the domain’s cheapest baseline.

Normalization

# preprocessing.py — three representations and two procedures. The procedures stay
# the same; only the way a pixel is turned into a number changes.
def raw_t(g):
    return g


def scale_t(g):                                  # dataset-wide scaling
    return [[v / 255 for v in s] for s in g]


def norm_t(g):                                   # per-image normalization
    x = [v for s in g for v in s]
    m = sum(x) / len(x)
    sd = (sum((v - m) ** 2 for v in x) / len(x)) ** 0.5
    return [[(v - m) / sd for v in s] for s in g]


def bright_box(x, threshold):
    """Bounding box of the pixels above the threshold."""
    lit = [(a, b) for a in range(SIZE) for b in range(SIZE) if x[a][b] >= threshold]
    if not lit:
        return None
    return (min(a for a, _ in lit), min(b for _, b in lit),
            max(a for a, _ in lit) - min(a for a, _ in lit) + 1,
            max(b for _, b in lit) - min(b for _, b in lit) + 1)


def threshold_rule(x, threshold):
    k = bright_box(x, threshold)
    return 0 if k is None else (1 if k[3] > k[2] else 0)


def mean_image(train, transform):
    """Domain baseline: class means, the decision is the nearest mean."""
    total = {0: [0.0] * (SIZE * SIZE), 1: [0.0] * (SIZE * SIZE)}
    count = {0: 0, 1: 0}
    for g, _, e in train:
        x = [v for s in transform(g) for v in s]
        for i in range(SIZE * SIZE):
            total[e][i] += x[i]
        count[e] += 1
    M = {e: [v / count[e] for v in total[e]] for e in (0, 1)}

    def decide(g):
        x = [v for s in transform(g) for v in s]
        d0 = sum((x[i] - M[0][i]) ** 2 for i in range(SIZE * SIZE))
        d1 = sum((x[i] - M[1][i]) ** 2 for i in range(SIZE * SIZE))
        return 1 if d1 < d0 else 0
    return decide


def measure(data, decide):
    return sum(1 for g, _, e in data if decide(g) == e) / len(data)


print("same image, normalized per image; same ramp:")
draw([[max(0, min(255, int((v + 2) * 51))) for v in s] for s in norm_t(g0)])

REPRESENTATIONS = (("raw pixel", raw_t, [100 + 10 * k for k in range(9)]),
                   ("dataset-wide scaling", scale_t, [(100 + 10 * k) / 255 for k in range(9)]),
                   ("per-image normal.", norm_t, [0.6 + 0.15 * k for k in range(9)]))
print(f"\n{'representation':<24}{'threshold':>10}{'mean image':>12}{'threshold rule':>16}")
for name, transform, candidates in REPRESENTATIONS:
    best = max(candidates, key=lambda t: measure(VAL, lambda g: threshold_rule(transform(g), t)))
    print(f"{name:<24}{best:>10.2f}{measure(TEST, mean_image(TRAIN, transform)):>12.4f}"
          f"{measure(TEST, lambda g: threshold_rule(transform(g), best)):>16.4f}")

same image, normalized per image; same ramp:
   ===..--:-:-.
   ..:.-::=:-==
   -.===:==:+-+
   ...=::.-=.=.
   :.#%##@%:.+:
   =+%+++%*+:.=
   -:%*%%+++.+:
   =.**%*%%=..:
   .=-:.-:+.=+=
   -=:-==:..::-
   ==..=+-=+-==
   :-+.==+=.+.=

representation           threshold  mean image  threshold rule
raw pixel                   130.00      0.5500          0.6937
dataset-wide scaling          0.51      0.5500          0.6937
per-image normal.             1.65      0.4875          0.9250

The drawing says enough on its own: when each pixel has its own image’s mean subtracted and is divided by its own standard deviation, the block becomes visible. The grid, the block, and the noise are the same; only the scale of the numbers changes.

The second row of the table gives a zero result. Dividing all pixels by 255 leaves both numbers of both procedures as they are: 0.5500 and 0.6937. Applying the same factor to every image changes neither the nearest mean nor the threshold ranking; this step is a change of units.

Per-image normalization carries the threshold rule from 0.6937 to 0.9250: a difference of 0.2313. A fixed threshold finds the block in an image whose background is 33, and in an image whose background is 108 it folds half of the background into the block too; after normalization, “1.65 standard deviations above the mean” means the same thing in every image. The model did not change, the data did not change, only the representation changed.

The same representation lowers the mean image baseline from 0.5500 to 0.4875. This is not a contradiction: the mean image procedure stays near the baseline in both representations, because the block is in a different place in every image and the orientation is erased when averaging. The 0.5500 on the raw representation is only 0.0188 above the 0.5312 of the most frequent class. The domain baseline draws a lower bound here, not a target.

Augmentation and the Transform That Breaks the Label

# augmentation.py — Augmentation can be measured only on a procedure that learns; the
# threshold rule does not learn. The measuring tool is nearest neighbor and here it is
# only a tool.
def shift(g, di, dj):
    fill = statistics.median(v for s in g for v in s)
    return [[g[a - di][b - dj] if 0 <= a - di < SIZE and 0 <= b - dj < SIZE else fill
             for b in range(SIZE)] for a in range(SIZE)]


TRANSFORMS = (("shift (1,1)", lambda g: shift(g, 1, 1)),
              ("horizontal flip", lambda g: [s[::-1] for s in g]),
              ("vertical flip", lambda g: g[::-1]),
              ("quarter turn", lambda g: [[g[SIZE - 1 - b][a] for b in range(SIZE)]
                                          for a in range(SIZE)]),
              ("brightness +25", lambda g: [[min(255, v + 25) for v in s] for s in g]))


def box_transform(k, name):
    i, j, h, w = k
    if name.startswith("shift"):
        i, j = i + 1, j + 1
        h, w = min(h, SIZE - i), min(w, SIZE - j)
    elif name == "horizontal flip":
        j = SIZE - j - w
    elif name == "vertical flip":
        i = SIZE - i - h
    elif name == "quarter turn":
        i, j, h, w = j, SIZE - i - h, w, h
    return (i, j, h, w)


print(f"{'transform':<17}{'label broken':>16}")
for name, _ in TRANSFORMS:
    broken = sum(1 for _, k, e in TRAIN
                 if (1 if box_transform(k, name)[3] > box_transform(k, name)[2] else 0) != e)
    print(f"{name:<17}{f'{broken}/{len(TRAIN)}':>16}")


def features(g):                             # normalize, 2x2 mean pool
    x = [v for s in norm_t(g) for v in s]
    return [sum(x[(2 * a + p) * SIZE + 2 * b + q] for p in (0, 1) for q in (0, 1)) / 4
            for a in range(6) for b in range(6)]


def nearest(store, data):
    d = 0
    for x, y in data:
        t = min(store, key=lambda u: sum((u[0][i] - x[i]) ** 2 for i in range(len(x))))
        d += 1 if t[1] == y else 0
    return d / len(data)


TEST_F = [(features(g), e) for g, _, e in TEST]
print(f"\nwith forty labels, two separate slices of forty (most frequent class {MAJORITY:.4f})")
print(f"{'setup':<36}{'examples':>9}{'slice 1':>9}{'slice 2':>9}")
for name, choice in (("no augmentation", ()),
                     ("+ shift", ("shift (1,1)",)),
                     ("+ two flips", ("horizontal flip", "vertical flip")),
                     ("+ quarter turn, label kept", ("quarter turn",)),
                     ("+ quarter turn, label corrected", ("quarter turn*",))):
    count, results = 0, []
    for start in (0, 40):
        FEW = TRAIN[start:start + 40]
        store = [(features(g), e) for g, _, e in FEW]
        for s in choice:
            f = dict(TRANSFORMS)[s.rstrip("*")]
            store += [(features(f(g)), (1 - e) if s.endswith("*") else e) for g, _, e in FEW]
        count = len(store)
        results.append(nearest(store, TEST_F))
    print(f"{name:<36}{count:>9}{results[0]:>9.4f}{results[1]:>9.4f}")

transform            label broken
shift (1,1)                 1/160
horizontal flip             0/160
vertical flip               0/160
quarter turn              160/160
brightness +25              0/160

with forty labels, two separate slices of forty (most frequent class 0.5312)
setup                                examples  slice 1  slice 2
no augmentation                            40   0.5938   0.6250
+ shift                                    80   0.6625   0.6687
+ two flips                               120   0.6875   0.6250
+ quarter turn, label kept                 80   0.5188   0.5000
+ quarter turn, label corrected            80   0.6312   0.7063

The upper table is this lesson’s image-specific number. Four of the five transforms leave the label as it is: flips mirror the block and do not change its dimensions; brightness does not touch the geometry; the shift breaks it in only 1 example, the one where the block overflows the edge and is clipped. The quarter turn, on the other hand, flips the label in 160 of 160 examples, because it swaps height and width, and the label is exactly the comparison of those two.

Augmentation has no task-independent list: the same transform is harmless in a task whose label is the block’s position and destructive in a task whose label is the block’s orientation. The lower table counts the cost. With forty labels, nearest neighbor reads 0.5938 and 0.6250 without augmentation. When two flips are added it reads 0.6875 in the first slice and 0.6250 in the second; when the shift is added, 0.6625 and 0.6687. The contribution rises to 0.0937 in the first slice and stays at 0.0437 in the second: the difference between the two slices is the same size as the difference augmentation adds. In the training practice lesson of the Deep Learning course the same contribution had been measured as +0.0026, +0.0013, and −0.0079; it shrinks as the scale grows.

When the transform that breaks the label is added, the number becomes 0.5188 and 0.5000: below the case without augmentation, and below even the 0.5312 of the most frequent class. When the same images are added with their labels corrected, 0.6312 and 0.7063 are read; the only difference between the two rows is what was written as the label. Where the transforms are applied is also critical, and here they are applied only to the training slice. The Deep Learning course had measured that doing augmentation before the split shifted the held-out set figure by +0.0203; that leak is not repeated here.

Summary

• An image is 12x12 pixels and one channel, that is 144 numbers; with three channels it would hold 432. The image means wander between 32.9 and 108.0.
• The domain baseline is two procedures: the most frequent class is 0.5312, the mean image is 0.5500. The mean image cannot read orientation, because the orientation is erased when the position is averaged.
• Dataset-wide scaling moves no number (0.5500 and 0.6937 stay as they are); this step is not a preprocessing but a change of units.
• Per-image normalization raises the threshold rule from 0.6937 to 0.9250; the difference is 0.2313 and comes entirely from the representation. The same representation lowers the mean image from 0.5500 to 0.4875.
• The quarter turn flips the label in 160 of 160 examples, the shift breaks it in 1, and the flips and brightness break it in none.
• With forty labels, 0.5938 and 0.6250 are read without augmentation; two flips give 0.6875 and 0.6250, and a quarter turn with its label kept gives 0.5188 and 0.5000 — below the baseline.

Next Step

In this lesson the deciding procedures either did not learn at all (the threshold rule) or stored the training set as it was (nearest neighbor). The 0.9250 that the threshold rule reads on the normalized representation is the number of three lines written by a human who knows the geometry of the task. The next lesson removes those three lines, puts a learning model in their place, and measures two things: how far the model exceeds the domain baseline, and, when the number of labels drops to twenty, how many examples a representation carried over from another task is worth.