Data Loading Pipeline

The previous lesson computed the gradient by two routes and showed that the two gave the same number to machine precision. One line of that lesson’s training loop passed without question: the training set was shuffled every epoch and split into batches of thirty. This line determines not what the network sees but in which order it sees it.

The data loading pipeline consists of three decisions: which examples, in which order, how many together. The third is the batch size and is left to the next lesson. This lesson measures the first two and adds a fourth: data augmentation. The measure is two numbers: the difference augmentation adds, and the inflation in the case where augmentation produces leakage.

• TP7. The dataset, the network, the baseline and the hand-written backpropagation are inherited from the previous lesson: the fictional subscriber table, the 300-row training set, the 8-8-1 network, 81 parameters, a baseline of 0.7579 on the test set.
• TP8. So that the comparisons can run, the number of updates is held constant at 400. Since the augmented set is twice as large, the number of epochs drops from forty to twenty; the network takes the same number of gradient steps in both setups.
• TP9. Augmentation here is adding noise to tabular data and is a model: copy rows are produced by adding noise, measured in units of standard deviation, to the four continuous columns; the label is copied and not changed. Domain-specific augmentation procedures are left to the Natural Language Processing and Computer Vision course.
• TP10. In the leakage setup the test set is the same 252 original rows in both cases; what changes is only which rows’ twins enter the training set. Leakage was measured in the Data Preparation and Feature Engineering course, and its procedure is not repeated.
• TP11. Run-to-run variability is not noise but a measured quantity: every comparison is printed with at least two runs and the difference it adds is read together with this range.

The Dataset, the Network and the Baseline

# COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through
# Unsupervised Learning courses is generated with the same seed (20260218);
# the training set is reduced from 756 to 300 rows. The network and the hand-written
# backpropagation are inherited from the previous lesson: 8-8-1, 81 parameters, 400 updates, step 0.30.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]
FIELDS = ["area_south", "area_east", "area_west", "area_central",
          "household", "satisfaction", "period", "mean"]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def mix(items, seed):
    r, s = rng(seed), list(items)
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return s


DATA = []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() < 0.046:
        continue
    subscriber_id = 10001 + i
    r, v = rng(SEED + 7000 + subscriber_id), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            b[2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id)
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3]
         + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": household, "satisfaction": rating, "period": len(v),
         "mean": round(mean, 2), "suspicious": int(z > 1.35)}
    for q in REGION[1:]:
        x["area_" + q[0]] = int(b[0] == q[0])
    DATA.append(x)

K = mix(DATA, SEED + 90000)
TRAIN, TEST = K[:300], K[1008:]


def table(rows, stats=None):
    X = [[float(s[a]) for a in FIELDS] for s in rows]
    if stats is None:                     # scaling is learned from the training set only
        stats = []
        for j in range(4, 8):
            m = sum(v[j] for v in X) / len(X)
            stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X))))
    for v in X:
        for j, m, sd in stats:
            v[j] = (v[j] - m) / sd
    return X, [s["suspicious"] for s in rows], stats


X_train, y_train, STATS = table(TRAIN)
X_test, y_test, _ = table(TEST, STATS)
BASELINE = sum(v == 0 for v in y_test) / len(y_test)

G, H = len(FIELDS), 8


def init_weights(seed):                  # gain-scaled initialization
    r = rng(seed)
    return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(H * G)] + [0.0] * H \
        + [(r() * 2 - 1) * math.sqrt(2.0 / H) for _ in range(H)] + [0.0]


def gradient(p, X, y, idx):              # the hand-written backpropagation of the previous lesson
    g = [0.0] * len(p)
    for k in idx:
        x, hh = X[k], [0.0] * H
        for j in range(H):
            s = p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G))
            hh[j] = s if s > 0 else 0.0
        s = p[-1] + sum(p[H * G + H + j] * hh[j] for j in range(H))
        d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k]
        g[-1] += d
        for j in range(H):
            g[H * G + H + j] += d * hh[j]
            if hh[j] > 0:
                g[H * G + j] += d * p[H * G + H + j]
                for i in range(G):
                    g[j * G + i] += d * p[H * G + H + j] * x[i]
    return [v / len(idx) for v in g]


STEP, BATCH_SIZE = 0.30, 30


def train_model(X, y, seed, order, updates=400):
    p, t = init_weights(seed), 0
    while t * (len(X) // BATCH_SIZE) < updates:
        perm = order(t, len(X))
        for b in range(0, len(perm) - BATCH_SIZE + 1, BATCH_SIZE):
            p = [a - STEP * v for a, v in zip(p, gradient(p, X, y, perm[b:b + BATCH_SIZE]))]
        t += 1
    return p


def accuracy(p, X, y):
    n = 0
    for k in range(len(X)):
        x = X[k]
        o = p[-1] + sum(p[H * G + H + j] * max(0.0, p[H * G + j]
                        + sum(p[j * G + i] * x[i] for i in range(G)))
                        for j in range(H))
        n += (o >= 0.0) == (y[k] == 1)
    return n / len(X)


print(f"fictional: train {len(X_train)} rows (reduced from 756), test {len(X_test)}, "
      f"features {G}, parameters {len(init_weights(SEED))}")
print(f"baseline (predicting the most frequent class, on the test set) {BASELINE:.4f}")

fictional: train 300 rows (reduced from 756), test 252, features 8, parameters 81
baseline (predicting the most frequent class, on the test set) 0.7579

Batch Generation Order

There are four order decisions and all four are defensible. Unordered: rows are given in the order of the file, the same batches are built every epoch. Once: the set is shuffled once at the start and that order is kept for forty epochs. Every epoch: the set is shuffled again every epoch and the batches are built from different rows every epoch; this last one is tried with two separate seeds.

# The batch generation ORDER is a decision: same data, same network, same 400 updates.
# Four orders, each with two initializations.
ORDERS = {"unordered": lambda t, n: list(range(n)),
          "once": lambda t, n: mix(range(n), SEED + 700),
          "every epoch A": lambda t, n: mix(range(n), SEED + 700 + t),
          "every epoch B": lambda t, n: mix(range(n), SEED + 800 + t)}
S = {}
print(f"{'batch order':<13} {'run 1':>8} {'run 2':>8} {'diff':>8}")
for name, o in ORDERS.items():
    S[name] = [accuracy(train_model(X_train, y_train, SEED + k, o), X_test, y_test) for k in (1, 2)]
    print(f"{name:<13} {S[name][0]:>8.4f} {S[name][1]:>8.4f} {S[name][1] - S[name][0]:>+8.4f}")
first = [v[0] for v in S.values()]
widest = max(abs(v[1] - v[0]) for v in S.values())
print(f"moved by the order (same initialization, four orders) {max(first) - min(first):.4f} "
      f"({round((max(first) - min(first)) * len(X_test))} subscribers)")
print(f"moved by the initialization (same order, two runs) at most {widest:.4f} "
      f"({round(widest * len(X_test))} subscribers)")
ORDER = ORDERS["every epoch A"]

batch order      run 1    run 2     diff
unordered       0.8056   0.7897  -0.0159
once            0.8016   0.8016  +0.0000
every epoch A   0.8095   0.7937  -0.0159
every epoch B   0.8016   0.8056  +0.0040
moved by the order (same initialization, four orders) 0.0079 (2 subscribers)
moved by the initialization (same order, two runs) at most 0.0159 (4 subscribers)

Four orders, the same network, the same 400 updates: test numbers of 0.8056, 0.8016, 0.8095 and 0.8016. What the order moves is 0.0079, two subscribers. This is not a finding but it is a decision: whichever order is chosen, its justification must be written, because it moves the result.

Read through the second column, the same table falls into scale. When the order is held constant and only the initial values change, the difference rises to 0.0159, four subscribers — twice what the order moves. In the “unordered” and “every epoch A” rows the second run falls four subscribers below the first, while in the “once” row the two runs give exactly the same number. Two runs coming out equal in the same row does not show that the order has no effect; it shows that two runs are not all runs.

Data Augmentation

# Data augmentation: COPY rows with noise added to the four continuous columns; the noise
# scale is in units of standard deviation. The NUMBER OF UPDATES is HELD CONSTANT at 400 —
# since the augmented set is twice as large, the number of epochs is halved.
def augment(X, y, seed, noise):
    r, X_aug, y_aug = rng(seed), list(X), list(y)
    for k in range(len(X)):
        v = list(X[k])
        for j in range(4, 8):
            v[j] += (r() + r() + r() - 1.5) * 2 * noise
        X_aug.append(v)
        y_aug.append(y[k])
    return X_aug, y_aug


MODELS, MEANS = {}, {}
print(f"{'augmentation':<13} {'rows':>6} {'epochs':>6} {'run1':>7} {'run2':>7} "
      f"{'run3':>7} {'mean':>7}")
for noise in (0.0, 0.05, 0.15, 0.40):
    X_aug, y_aug = (X_train, y_train) if noise == 0.0 else augment(X_train, y_train, SEED + 11000, noise)
    MODELS[noise] = [train_model(X_aug, y_aug, SEED + k, ORDER) for k in (1, 2, 3)]
    r = [accuracy(p, X_test, y_test) for p in MODELS[noise]]
    MEANS[noise] = sum(r) / len(r)
    name = "none" if noise == 0.0 else f"noise {noise:.2f}"
    print(f"{name:<13} {len(X_aug):>6} {400 // (len(X_aug) // 30):>6} "
          + " ".join(f"{v:>7.4f}" for v in r) + f" {MEANS[noise]:>7.4f}")
for noise in (0.05, 0.15, 0.40):
    print(f"added by augmentation (noise {noise:.2f}) {MEANS[noise] - MEANS[0.0]:+.4f}")

augmentation    rows epochs    run1    run2    run3    mean
none             300     40  0.8095  0.7937  0.8016  0.8016
noise 0.05       600     20  0.8016  0.8095  0.8016  0.8042
noise 0.15       600     20  0.8016  0.8095  0.7976  0.8029
noise 0.40       600     20  0.7937  0.7937  0.7937  0.7937
added by augmentation (noise 0.05) +0.0026
added by augmentation (noise 0.15) +0.0013
added by augmentation (noise 0.40) -0.0079

The difference augmentation adds is read from the mean of three runs: +0.0026 at noise 0.05, +0.0013 at 0.15 and −0.0079 at 0.40. The first two are positive and the third negative; but all three are smaller than the setup without augmentation’s own run range (0.7937–0.8095, that is, 0.0158). The honest sentence on this table is: augmentation does not add a measurable difference on this data. The contribution cannot be told apart from zero, because the difference between two runs of the same setup is larger than the contribution.

Something else shows in the last row. When the noise rises to 0.40, all three runs give exactly the same number: 0.7937. Strong noise behaves like regularization and erases the difference the initial values opened, but together with the difference it erases it also erases the gain. The number of rows the network sees doubled; since the number of epochs halved, the gradient steps it took stayed the same, and the diversity augmentation gained did not turn out larger than the signal it lost.

Where Augmentation Leaks

Augmentation is an order decision. The right order is split, then augment: copies are derived only from training rows. The wrong order is augment, then split: the noisy twin of a test row falls into the training set and the network has nearly seen that row.

# LEAKAGE: if augmentation is done BEFORE the split, the noisy twin of a test row enters
# the training set (the Data Preparation and Feature Engineering course, data leakage). The test set is the SAME
# 252 original rows in both setups; the shift is read from the subset comparison.
def leaky(noise, seed):
    r, X_aug, y_aug = rng(seed), list(X_train), list(y_train)
    pool = mix(range(len(X_train) + len(X_test)), seed + 5)[:len(X_train)]
    for k in pool:
        x, e = (X_train[k], y_train[k]) if k < len(X_train) else (X_test[k - len(X_train)], y_test[k - len(X_train)])
        v = list(x)
        for j in range(4, 8):
            v[j] += (r() + r() + r() - 1.5) * 2 * noise
        X_aug.append(v)
        y_aug.append(e)
    return X_aug, y_aug, sorted(k - len(X_train) for k in pool if k >= len(X_train))


def subset(p, idx):
    return accuracy(p, [X_test[k] for k in idx], [y_test[k] for k in idx])


X_leak, y_leak, TWINS = leaky(0.05, SEED + 11000)
CLEAN = [k for k in range(len(X_test)) if k not in set(TWINS)]
print(f"{len(TWINS)} rows of the test set have a twin in the training set, "
      f"{len(CLEAN)} do not")
print(f"{'training':<11} {'run':>5} {'all':>7} {'twinned':>7} {'clean':>7} {'diff':>8}")
K = {}
for name, P in (("honest", MODELS[0.05]),
                ("leaky", [train_model(X_leak, y_leak, SEED + k, ORDER) for k in (1, 2, 3)])):
    K[name] = []
    for k, p in enumerate(P, 1):
        a, b = subset(p, TWINS), subset(p, CLEAN)
        K[name].append(a - b)
        print(f"{name:<11} {k:>5} {accuracy(p, X_test, y_test):>7.4f} {a:>7.4f} {b:>7.4f} "
              f"{a - b:>+8.4f}")
d, s = sum(K["honest"]) / 3, sum(K["leaky"]) / 3
print(f"mean of the twinned-clean gap: honest {d:+.4f}, leaky {s:+.4f}; "
      f"shift {s - d:+.4f}")

138 rows of the test set have a twin in the training set, 114 do not
training      run     all twinned   clean     diff
honest          1  0.8016  0.7899  0.8158  -0.0259
honest          2  0.8095  0.8043  0.8158  -0.0114
honest          3  0.8016  0.7899  0.8158  -0.0259
leaky           1  0.8095  0.8116  0.8070  +0.0046
leaky           2  0.8095  0.8043  0.8158  -0.0114
leaky           3  0.8095  0.8116  0.8070  +0.0046
mean of the twinned-clean gap: honest -0.0211, leaky -0.0008; shift +0.0203

The overall numbers cannot be read on their own: the three runs of leaky training sit at 0.8095, the three runs of honest training between 0.8016 and 0.8095. The difference stays inside the run range and at this scale leakage is not visible from the overall number. In the Data Preparation and Feature Engineering course a leakage of the same kind had inflated the held-out set number by 0.0476.

The subset comparison cancels run-to-run variability, because the two subsets are read from the same run. The difference between the 138 rows whose twin is in the training set and the 114 rows with no twin is, in honest training, −0.0211 on average: the twinned group is slightly harder to begin with. In leaky training the same difference rises to −0.0008. The shift is +0.0203, that is, five subscribers; and this shift is the trace left by the leakage that does not show in the overall number. The presence of leakage is read not from the overall number but from who inflated.

Summary

• When the batch generation order is built with four separate decisions, the test number varies between 0.8016 and 0.8095; what the order moves is 0.0079, two subscribers.
• When only the initial values change under the same order, the difference rises to 0.0159, four subscribers: run-to-run variability moves twice as much as the order decision.
• The difference augmentation adds is +0.0026, +0.0013 and −0.0079 in the mean of three runs; all three are smaller than the setup without augmentation’s run range, that is, the contribution cannot be told apart from zero.
• At noise 0.40 all three runs give 0.7937: strong augmentation erases both the initial difference and the gain.
• When augmentation is done before the split, the twins of 138 rows of the test set enter the training set; the overall number does not show the inflation but the twinned-clean gap shifts from −0.0211 to −0.0008, a shift of +0.0203.

Next Step

The batch size was held at thirty in this lesson, and the number of updates at four hundred. Both are decisions and both are tied to a resource: how many examples’ forward and backward passes can stay in memory at the same time in one update. The next lesson addresses hardware usage: the relation between the batch size and the number of stored values in memory is computed, and that the batch size needs to change together with the learning rate is shown with numbers.