--- title: 'Data Loading Pipeline' source: 'https://academia.sh/en/courses/deep-learning/data-loading-pipeline' course: 'Deep Learning' language: en updated: '2026-10-21T05:03:25+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds the batch generation order with four separate decisions and compares the difference the order moves with the difference two runs of the same network move' - 'Applies data augmentation at three noise scales and measures the difference it adds with the number of updates held constant' - 'Shows that augmentation produces leakage into the held-out set when done before the split and measures the inflation with a subset comparison independent of run-to-run variability' --- # Data Loading Pipeline Measuring the batch generation order, data augmentation and the leakage augmentation produces separately: with the same network and the same 400 updates, four batch orders give four numbers between 0.8016 and 0.8095 on the test set, while the difference rises to 0.0159 when only the initial values change under the same order — the order decision moves half as much as run-to-run variability. The difference augmentation adds is +0.0026, +0.0013 and −0.0079 in the mean of three runs; all three are smaller than the run range of a single order. When augmentation is done before the split, the twins of 138 rows of the test set enter the training set and the twinned-clean subset gap shifts from −0.0211 to −0.0008: a shift of +0.0203. The previous lesson computed the gradient by two routes and showed that the two gave the same number to machine precision. One line of that lesson's training loop passed without question: the training set was shuffled every epoch and split into batches of thirty. This line determines not **what** the network sees but **in which order** it sees it. The data loading pipeline consists of three decisions: which examples, in which order, how many together. The third is the batch size and is left to the next lesson. This lesson measures the first two and adds a fourth: **data augmentation**. The measure is two numbers: the difference augmentation adds, and the inflation in the case where augmentation **produces leakage**. - **TP7.** The dataset, the network, the baseline and the hand-written backpropagation are inherited from the previous lesson: the fictional subscriber table, the 300-row training set, the 8-8-1 network, 81 parameters, a baseline of 0.7579 on the test set. - **TP8.** So that the comparisons can run, **the number of updates is held constant at 400**. Since the augmented set is twice as large, the number of epochs drops from forty to twenty; the network takes the same number of gradient steps in both setups. - **TP9.** Augmentation here is **adding noise to tabular data** and is a model: copy rows are produced by adding noise, measured in units of standard deviation, to the four continuous columns; the label is copied and not changed. Domain-specific augmentation procedures are left to the Natural Language Processing and Computer Vision course. - **TP10.** In the leakage setup **the test set is the same 252 original rows in both cases**; what changes is only which rows' twins enter the training set. Leakage was measured in the Data Preparation and Feature Engineering course, and its procedure is not repeated. - **TP11.** Run-to-run variability is not noise but a measured quantity: every comparison is printed with **at least two runs** and the difference it adds is read together with this range. ## The Dataset, the Network and the Baseline ```python # COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through # Unsupervised Learning courses is generated with the same seed (20260218); # the training set is reduced from 756 to 300 rows. The network and the hand-written # backpropagation are inherited from the previous lesson: 8-8-1, 81 parameters, 400 updates, step 0.30. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "mean"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def mix(items, seed): r, s = rng(seed), list(items) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue subscriber_id = 10001 + i r, v = rng(SEED + 7000 + subscriber_id), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean": round(mean, 2), "suspicious": int(z > 1.35)} for q in REGION[1:]: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) K = mix(DATA, SEED + 90000) TRAIN, TEST = K[:300], K[1008:] def table(rows, stats=None): X = [[float(s[a]) for a in FIELDS] for s in rows] if stats is None: # scaling is learned from the training set only stats = [] for j in range(4, 8): m = sum(v[j] for v in X) / len(X) stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X)))) for v in X: for j, m, sd in stats: v[j] = (v[j] - m) / sd return X, [s["suspicious"] for s in rows], stats X_train, y_train, STATS = table(TRAIN) X_test, y_test, _ = table(TEST, STATS) BASELINE = sum(v == 0 for v in y_test) / len(y_test) G, H = len(FIELDS), 8 def init_weights(seed): # gain-scaled initialization r = rng(seed) return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(H * G)] + [0.0] * H \ + [(r() * 2 - 1) * math.sqrt(2.0 / H) for _ in range(H)] + [0.0] def gradient(p, X, y, idx): # the hand-written backpropagation of the previous lesson g = [0.0] * len(p) for k in idx: x, hh = X[k], [0.0] * H for j in range(H): s = p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G)) hh[j] = s if s > 0 else 0.0 s = p[-1] + sum(p[H * G + H + j] * hh[j] for j in range(H)) d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k] g[-1] += d for j in range(H): g[H * G + H + j] += d * hh[j] if hh[j] > 0: g[H * G + j] += d * p[H * G + H + j] for i in range(G): g[j * G + i] += d * p[H * G + H + j] * x[i] return [v / len(idx) for v in g] STEP, BATCH_SIZE = 0.30, 30 def train_model(X, y, seed, order, updates=400): p, t = init_weights(seed), 0 while t * (len(X) // BATCH_SIZE) < updates: perm = order(t, len(X)) for b in range(0, len(perm) - BATCH_SIZE + 1, BATCH_SIZE): p = [a - STEP * v for a, v in zip(p, gradient(p, X, y, perm[b:b + BATCH_SIZE]))] t += 1 return p def accuracy(p, X, y): n = 0 for k in range(len(X)): x = X[k] o = p[-1] + sum(p[H * G + H + j] * max(0.0, p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G))) for j in range(H)) n += (o >= 0.0) == (y[k] == 1) return n / len(X) print(f"fictional: train {len(X_train)} rows (reduced from 756), test {len(X_test)}, " f"features {G}, parameters {len(init_weights(SEED))}") print(f"baseline (predicting the most frequent class, on the test set) {BASELINE:.4f}") ``` ``` fictional: train 300 rows (reduced from 756), test 252, features 8, parameters 81 baseline (predicting the most frequent class, on the test set) 0.7579 ``` ## Batch Generation Order There are four order decisions and all four are defensible. **Unordered**: rows are given in the order of the file, the same batches are built every epoch. **Once**: the set is shuffled once at the start and that order is kept for forty epochs. **Every epoch**: the set is shuffled again every epoch and the batches are built from different rows every epoch; this last one is tried with two separate seeds. ```python # The batch generation ORDER is a decision: same data, same network, same 400 updates. # Four orders, each with two initializations. ORDERS = {"unordered": lambda t, n: list(range(n)), "once": lambda t, n: mix(range(n), SEED + 700), "every epoch A": lambda t, n: mix(range(n), SEED + 700 + t), "every epoch B": lambda t, n: mix(range(n), SEED + 800 + t)} S = {} print(f"{'batch order':<13} {'run 1':>8} {'run 2':>8} {'diff':>8}") for name, o in ORDERS.items(): S[name] = [accuracy(train_model(X_train, y_train, SEED + k, o), X_test, y_test) for k in (1, 2)] print(f"{name:<13} {S[name][0]:>8.4f} {S[name][1]:>8.4f} {S[name][1] - S[name][0]:>+8.4f}") first = [v[0] for v in S.values()] widest = max(abs(v[1] - v[0]) for v in S.values()) print(f"moved by the order (same initialization, four orders) {max(first) - min(first):.4f} " f"({round((max(first) - min(first)) * len(X_test))} subscribers)") print(f"moved by the initialization (same order, two runs) at most {widest:.4f} " f"({round(widest * len(X_test))} subscribers)") ORDER = ORDERS["every epoch A"] ``` ``` batch order run 1 run 2 diff unordered 0.8056 0.7897 -0.0159 once 0.8016 0.8016 +0.0000 every epoch A 0.8095 0.7937 -0.0159 every epoch B 0.8016 0.8056 +0.0040 moved by the order (same initialization, four orders) 0.0079 (2 subscribers) moved by the initialization (same order, two runs) at most 0.0159 (4 subscribers) ``` Four orders, the same network, the same 400 updates: test numbers of 0.8056, 0.8016, 0.8095 and 0.8016. What the order moves is **0.0079**, two subscribers. This is not a finding but it is a decision: whichever order is chosen, its justification must be written, because it moves the result. Read through the second column, the same table falls into scale. When the order is held constant and only the initial values change, the difference rises to **0.0159**, four subscribers — twice what the order moves. In the "unordered" and "every epoch A" rows the second run falls four subscribers below the first, while in the "once" row the two runs give exactly the same number. Two runs coming out equal in the same row does not show that the order has no effect; it shows that **two runs are not all runs**. ## Data Augmentation ```python # Data augmentation: COPY rows with noise added to the four continuous columns; the noise # scale is in units of standard deviation. The NUMBER OF UPDATES is HELD CONSTANT at 400 — # since the augmented set is twice as large, the number of epochs is halved. def augment(X, y, seed, noise): r, X_aug, y_aug = rng(seed), list(X), list(y) for k in range(len(X)): v = list(X[k]) for j in range(4, 8): v[j] += (r() + r() + r() - 1.5) * 2 * noise X_aug.append(v) y_aug.append(y[k]) return X_aug, y_aug MODELS, MEANS = {}, {} print(f"{'augmentation':<13} {'rows':>6} {'epochs':>6} {'run1':>7} {'run2':>7} " f"{'run3':>7} {'mean':>7}") for noise in (0.0, 0.05, 0.15, 0.40): X_aug, y_aug = (X_train, y_train) if noise == 0.0 else augment(X_train, y_train, SEED + 11000, noise) MODELS[noise] = [train_model(X_aug, y_aug, SEED + k, ORDER) for k in (1, 2, 3)] r = [accuracy(p, X_test, y_test) for p in MODELS[noise]] MEANS[noise] = sum(r) / len(r) name = "none" if noise == 0.0 else f"noise {noise:.2f}" print(f"{name:<13} {len(X_aug):>6} {400 // (len(X_aug) // 30):>6} " + " ".join(f"{v:>7.4f}" for v in r) + f" {MEANS[noise]:>7.4f}") for noise in (0.05, 0.15, 0.40): print(f"added by augmentation (noise {noise:.2f}) {MEANS[noise] - MEANS[0.0]:+.4f}") ``` ``` augmentation rows epochs run1 run2 run3 mean none 300 40 0.8095 0.7937 0.8016 0.8016 noise 0.05 600 20 0.8016 0.8095 0.8016 0.8042 noise 0.15 600 20 0.8016 0.8095 0.7976 0.8029 noise 0.40 600 20 0.7937 0.7937 0.7937 0.7937 added by augmentation (noise 0.05) +0.0026 added by augmentation (noise 0.15) +0.0013 added by augmentation (noise 0.40) -0.0079 ``` The difference augmentation adds is read from the mean of three runs: **+0.0026** at noise 0.05, **+0.0013** at 0.15 and **−0.0079** at 0.40. The first two are positive and the third negative; but all three are smaller than the setup without augmentation's own run range (0.7937–0.8095, that is, 0.0158). The honest sentence on this table is: **augmentation does not add a measurable difference on this data.** The contribution cannot be told apart from zero, because the difference between two runs of the same setup is larger than the contribution. Something else shows in the last row. When the noise rises to 0.40, all three runs give **exactly the same number**: 0.7937. Strong noise behaves like regularization and erases the difference the initial values opened, but together with the difference it erases it also erases the gain. The number of rows the network sees doubled; since the number of epochs halved, the gradient steps it took stayed the same, and the diversity augmentation gained did not turn out larger than the signal it lost. ## Where Augmentation Leaks Augmentation is an order decision. The right order is **split, then augment**: copies are derived only from training rows. The wrong order is **augment, then split**: the noisy twin of a test row falls into the training set and the network has nearly seen that row. ```python # LEAKAGE: if augmentation is done BEFORE the split, the noisy twin of a test row enters # the training set (the Data Preparation and Feature Engineering course, data leakage). The test set is the SAME # 252 original rows in both setups; the shift is read from the subset comparison. def leaky(noise, seed): r, X_aug, y_aug = rng(seed), list(X_train), list(y_train) pool = mix(range(len(X_train) + len(X_test)), seed + 5)[:len(X_train)] for k in pool: x, e = (X_train[k], y_train[k]) if k < len(X_train) else (X_test[k - len(X_train)], y_test[k - len(X_train)]) v = list(x) for j in range(4, 8): v[j] += (r() + r() + r() - 1.5) * 2 * noise X_aug.append(v) y_aug.append(e) return X_aug, y_aug, sorted(k - len(X_train) for k in pool if k >= len(X_train)) def subset(p, idx): return accuracy(p, [X_test[k] for k in idx], [y_test[k] for k in idx]) X_leak, y_leak, TWINS = leaky(0.05, SEED + 11000) CLEAN = [k for k in range(len(X_test)) if k not in set(TWINS)] print(f"{len(TWINS)} rows of the test set have a twin in the training set, " f"{len(CLEAN)} do not") print(f"{'training':<11} {'run':>5} {'all':>7} {'twinned':>7} {'clean':>7} {'diff':>8}") K = {} for name, P in (("honest", MODELS[0.05]), ("leaky", [train_model(X_leak, y_leak, SEED + k, ORDER) for k in (1, 2, 3)])): K[name] = [] for k, p in enumerate(P, 1): a, b = subset(p, TWINS), subset(p, CLEAN) K[name].append(a - b) print(f"{name:<11} {k:>5} {accuracy(p, X_test, y_test):>7.4f} {a:>7.4f} {b:>7.4f} " f"{a - b:>+8.4f}") d, s = sum(K["honest"]) / 3, sum(K["leaky"]) / 3 print(f"mean of the twinned-clean gap: honest {d:+.4f}, leaky {s:+.4f}; " f"shift {s - d:+.4f}") ``` ``` 138 rows of the test set have a twin in the training set, 114 do not training run all twinned clean diff honest 1 0.8016 0.7899 0.8158 -0.0259 honest 2 0.8095 0.8043 0.8158 -0.0114 honest 3 0.8016 0.7899 0.8158 -0.0259 leaky 1 0.8095 0.8116 0.8070 +0.0046 leaky 2 0.8095 0.8043 0.8158 -0.0114 leaky 3 0.8095 0.8116 0.8070 +0.0046 mean of the twinned-clean gap: honest -0.0211, leaky -0.0008; shift +0.0203 ``` The overall numbers cannot be read on their own: the three runs of leaky training sit at 0.8095, the three runs of honest training between 0.8016 and 0.8095. The difference stays inside the run range and **at this scale leakage is not visible from the overall number**. In the Data Preparation and Feature Engineering course a leakage of the same kind had inflated the held-out set number by 0.0476. The subset comparison cancels run-to-run variability, because the two subsets are read **from the same run**. The difference between the 138 rows whose twin is in the training set and the 114 rows with no twin is, in honest training, **−0.0211** on average: the twinned group is slightly harder to begin with. In leaky training the same difference rises to **−0.0008**. The shift is **+0.0203**, that is, five subscribers; and this shift is the trace left by the leakage that does not show in the overall number. The presence of leakage is read not from the overall number but from **who inflated**. ## Summary - When the batch generation order is built with four separate decisions, the test number varies between 0.8016 and 0.8095; what the order moves is 0.0079, two subscribers. - When only the initial values change under the same order, the difference rises to 0.0159, four subscribers: run-to-run variability moves twice as much as the order decision. - The difference augmentation adds is +0.0026, +0.0013 and −0.0079 in the mean of three runs; all three are smaller than the setup without augmentation's run range, that is, the contribution cannot be told apart from zero. - At noise 0.40 all three runs give 0.7937: strong augmentation erases both the initial difference and the gain. - When augmentation is done before the split, the twins of 138 rows of the test set enter the training set; the overall number does not show the inflation but the twinned-clean gap shifts from −0.0211 to −0.0008, a shift of +0.0203. ## Next Step The batch size was held at thirty in this lesson, and the number of updates at four hundred. Both are decisions and both are tied to a resource: how many examples' forward and backward passes can stay in memory **at the same time** in one update. The next lesson addresses hardware usage: the relation between the batch size and the number of stored values in memory is computed, and that the batch size needs to change together with the learning rate is shown with numbers.