--- title: 'Deep Learning' source: 'https://academia.sh/en/courses/deep-learning' language: en updated: '2026-10-21T05:03:22+00:00' license: 'CC BY-SA 4.0' lessons: 24 outcomes: - 'Writes the forward and backward pass of a neural network by hand and verifies the gradient with a numerical derivative' - 'Applies convolutional, recurrent and attention-based architectures to suitable data structures and measures what they add' - 'Stabilizes the training process with regularization and optimization settings and reports run-to-run variance' --- # Deep Learning Measuring whether the number a network produces comes from its capacity or from its training, together with a second run of the same network: a single-layer network being the same as a linear model, saturation and the number of dead units, the parameter and multiplication counts as a layer widens, the number the choice of loss moves, verifying backpropagation with a numerical derivative, the loss of three optimizers on the same budget, the gradient vanishing as depth grows, the contribution of dropout and early stopping, the learning rate that batch normalization allows, the parameter savings of convolution over a fully connected layer, the information pooling discards, skip connections carrying the gradient to the first layer, the information the hidden state carries over a long sequence, where the attention weights fall, the necessity of positional encoding, the difference between a learned embedding and an unlearned projection, how many examples transfer adds with, the difference between automatic differentiation and a hand-written gradient, the relation between batch size and memory, and run-to-run variance misleading the search. ## Neural Network Fundamentals Measuring what each part of a fully connected network contributes and how much a second run of the same network moves: the single-layer network being the same thing as the linear model, what adding a layer changes, saturation shrinking the gradient and the dead unit count, the growth of parameter and multiplication counts as layers widen, the choice of loss giving two different results in the same network and the loss not being the same thing as the metric, backpropagation checked against the numerical derivative, the loss and stability reached by three optimizers within the same epoch budget, the vanishing and exploding gradient as depth grows, the contribution of dropout and early stopping and which set early stopping is read from, the learning rate that batch normalization allows. ### Perceptron and Multilayer Network Source: https://academia.sh/en/courses/deep-learning/perceptron-and-multilayer-network First measurement of whether a network's number comes from its capacity or from its training: initialized from zero, the single-layer network produces all three numbers of the Supervised Learning course's linear model at its three settings, down to the last digit. On the reduced 300-row training set, the two-layer network with 113 parameters reaches 0.7857 across four candidates, with a baseline of 0.7579; but a second run of the same network, with only the initial values changed, gives 0.7698, and the band of five runs is 0.7659-0.7857. Run-to-run variability is five subscribers, the difference capacity adds is one subscriber; the two-layer network whose activation is left linear cannot get out of the single-layer band despite its 113 parameters. Across twenty-one lessons, eleven of them unlabeled, every method worked on the assumption that the features were given. Scaling, encoding, generation and reduction were always decided by hand, and the cost of each decision was measured separately. What a procedure that learns these decisions from the data itself would gain, and what it would make invisible, was never asked. The neural network is a claimed answer to that question: instead of designing the intermediate representation by hand, determine the weights of the hidden layers by training as well. This lesson builds the smallest form of that claim and sets up the three numbers of the course: the **baseline**, **the network's number**, and **the second run of the same network, with only the initial values changed**. The third is the signature of the course, because a network's number can come from its training rather than its capacity. One more thing is announced up front: on this fictional tabular data the network is not expected to beat the decision tree or the linear model, and this will not be hidden. - **NN1.** The dataset, the split and the seed carry over unchanged from the Introduction to Machine Learning, Data Preparation and Feature Engineering, Supervised Learning and Unsupervised Learning courses: the fictional subscriber table, 756/252/252, seed 20260218. The target is the **suspicious** label and the baseline is saying the most frequent class. - **NN2.** The feature set is the twelve columns left by the Data Preparation and Feature Engineering course and is fixed throughout the course. The network **takes these columns as given**; the scaling, encoding and derivation decisions are still made by hand. - **NN3.** The columns are scaled with the mean and dispersion learned from the training set. The Data Preparation and Feature Engineering course had measured that this step added +0.0437 to a gradient-based model; here it is assumed. - **NN4.** In this course the training set is **reduced to 300 rows**. The reason is run time: every lesson trains the same network at least twice. The cost of the reduction is measured in this lesson. - **NN5.** The solution is **gradient descent**; its procedure was built in the Linear Models topic of the Supervised Learning course and is not repeated here. How the gradient is distributed across layers is the subject of the Backpropagation lesson. - **NN6.** The weights are started with **gain-scaled initialization**: the starting range is scaled with the square root of the number of inputs. The biases start from zero. - **NN7.** The metric is accuracy, the test set is 252 subscribers and one subscriber is worth 0.0040. The number of candidates tried is **4**, and the choice is made on the validation set. #### Weight, Bias and Unit A **unit** is built from three parts: each input gets a **weight**, a constant is added to the weighted sum, and the result is passed through an **activation function**. In this course the added constant is called the **bias**. The distinction is necessary: the **bias** in the bias–variance trade-off of the Introduction to Machine Learning course is an error component, whereas the **bias** here is a learned parameter and shifts the threshold of the unit. Units standing side by side form a **layer**. This lesson compares two architectures: the **single-layer** network that connects the input directly to the output, and the **two-layer** network that carries a layer of eight units in between. ```python # neural_net.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split; # the target is the label of suspected illicit use or fault. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): # self-written generator, no library x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def scaled(x): # FIT on train, APPLY to every set return [(x[a] - MEAN[a]) / STD[a] for a in FIELDS] def make_set(data): return [scaled(x) for x in data], [float(x["suspicious"]) for x in data] X_FULL, Y_FULL = make_set(TRAIN) # full training set, 756 rows X_TR, Y_TR = make_set(TRAIN[:300]) # REDUCED training set, 300 rows X_VA, Y_VA = make_set(VAL) X_TE, Y_TE = make_set(TEST) print(f"train {len(X_FULL)}, reduced train {len(X_TR)}, validation {len(X_VA)}, " f"test {len(X_TE)}, columns {len(FIELDS)}") print(f"suspicious rate in full train {sum(Y_FULL) / len(Y_FULL):.4f}, " f"in reduced {sum(Y_TR) / len(Y_TR):.4f}, in test {sum(Y_TE) / len(Y_TE):.4f}") print(f"baseline (most frequent class) on test {BASELINE:.4f}") ``` ``` train 756, reduced train 300, validation 252, test 252, columns 12 suspicious rate in full train 0.2222, in reduced 0.2200, in test 0.2421 baseline (most frequent class) on test 0.7579 ``` The comparison line comes from the Supervised Learning course and was measured with the full 756 rows: the gradient-based linear model and the decision tree tuned on validation give **0.8016** on the test set, and the untuned decision tree 0.7857. What the network has to beat is not the baseline but this number. #### A Single-Layer Network Is Not a New Model A network with no hidden layer carries twelve weights and one bias, feeds the weighted sum to an S-shaped function and produces a probability. This description is word for word the same as that of the logistic regression of the Supervised Learning course, and the sameness can be tested: if the weights are started from zero and trained with the same learning rate and number of epochs, the network must reproduce the table of that lesson. ```python def squash(z): # S-shaped function, overflow-protected if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def init_weights(seed, sizes): # gain-scaled initialization, bias zero r, net = rng(seed), [] for a, b in zip(sizes, sizes[1:]): scale = math.sqrt(2.0 / a) net.append(([[(r() * 2 - 1) * scale for _ in range(a)] for _ in range(b)], [0.0] * b)) return net def forward(net, x, linear=False): # FORWARD PASS: layer by layer a, acts = x, [x] for k, (W, bias) in enumerate(net): z = [sum(w * v for w, v in zip(s, a)) + b for s, b in zip(W, bias)] a = [squash(z[0])] if k == len(net) - 1 else ( z[:] if linear else [t if t > 0 else 0.0 for t in z]) acts.append(a) return acts def train_model(net, X, Y, rate, epochs, linear=False): # MODEL: full-batch gradient descent n = len(X) for _ in range(epochs): gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net] gb = [[0.0] * len(bias) for _, bias in net] for i in range(n): acts = forward(net, X[i], linear) d = [acts[-1][0] - Y[i]] for k in range(len(net) - 1, -1, -1): W, a = net[k][0], acts[k] for j, dj in enumerate(d): if dj: s = gW[k][j] for p in range(len(a)): s[p] += dj * a[p] gb[k][j] += dj if k: prev = [0.0] * len(a) for j, dj in enumerate(d): if dj: s = W[j] for p in range(len(a)): prev[p] += dj * s[p] d = prev if linear else [prev[p] if a[p] > 0 else 0.0 for p in range(len(a))] for k, (W, bias) in enumerate(net): for j in range(len(W)): s, g = W[j], gW[k][j] for p in range(len(s)): s[p] -= rate * g[p] / n bias[j] -= rate * gb[k][j] / n return net def accuracy(net, X, Y, linear=False): return sum((forward(net, x, linear)[-1][0] > 0.5) == (y > 0.5) for x, y in zip(X, Y)) / len(X) def param_count(sizes): return sum(a * b + b for a, b in zip(sizes, sizes[1:])) print(f"single-layer network parameter count {param_count([12, 1])}, " f"two-layer (8 units) {param_count([12, 8, 1])}") print(f"{'learning rate':>14} {'epoch':>5} {'train':>9} {'validation':>10} {'test':>8}") for rate, epochs in ((0.1, 100), (0.1, 300), (0.1, 600)): net = train_model([([[0.0] * 12], [0.0])], X_FULL, Y_FULL, rate, epochs) # from zero print(f"{rate:>14} {epochs:>5} {accuracy(net, X_FULL, Y_FULL):>9.4f} " f"{accuracy(net, X_VA, Y_VA):>10.4f} {accuracy(net, X_TE, Y_TE):>8.4f}") ``` ``` single-layer network parameter count 13, two-layer (8 units) 113 learning rate epoch train validation test 0.1 100 0.8175 0.8056 0.8016 0.1 300 0.8201 0.8373 0.7937 0.1 600 0.8188 0.8373 0.8016 ``` All nine numbers of the three rows are the same, down to the last digit, as the first three rows of the logistic regression table of the Supervised Learning course. The single-layer network is not a new model family but a known model written in another vocabulary. Nothing the name "neural network" adds appears in these rows. #### What Makes a Layer Nonlinear The hidden layer brings 113 parameters, against 13 for the single-layer network. But capacity is not only the number of parameters. If the activation of the hidden layer is left as the identity function, that is, if the weighted sum is passed through as it is, the composition of two matrix multiplications is again a single matrix multiplication. The **set of functions the network can represent** does not change; the same function is merely written with more parameters. The measurement shows this: three architectures, the same setting, two runs. The activation of the hidden layer is once the identity function and once the **rectified linear unit**, the function that cuts everything below zero to zero. ```python ARCHITECTURES = (("single-layer", [12, 1], False), ("two-layer linear", [12, 8, 1], True), ("two-layer rectified", [12, 8, 1], False)) print(f"{'architecture':<24} {'params':>9} {'run':>6} {'train':>9} " f"{'validation':>10} {'test':>8}") BAND = {} for name, sizes, lin in ARCHITECTURES: BAND[name] = [] for run in (1, 2): net = train_model(init_weights(SEED + run, sizes), X_TR, Y_TR, 0.3, 100, lin) s = accuracy(net, X_TE, Y_TE, lin) BAND[name].append(s) print(f"{name:<24} {param_count(sizes):>9} {run:>6} " f"{accuracy(net, X_TR, Y_TR, lin):>9.4f} {accuracy(net, X_VA, Y_VA, lin):>10.4f} {s:>8.4f}") print() for name, v in BAND.items(): print(f"{name:<24} test band {min(v):.4f}-{max(v):.4f}") ``` ``` architecture params run train validation test single-layer 13 1 0.8433 0.7976 0.7817 single-layer 13 2 0.8300 0.8016 0.7698 two-layer linear 113 1 0.8233 0.8056 0.7698 two-layer linear 113 2 0.8267 0.8016 0.7738 two-layer rectified 113 1 0.8400 0.8214 0.7857 two-layer rectified 113 2 0.8367 0.8056 0.7698 single-layer test band 0.7698-0.7817 two-layer linear test band 0.7698-0.7738 two-layer rectified test band 0.7698-0.7857 ``` The network with linear activation stays inside the band of the single-layer network despite its 113 parameters: 0.7698–0.7738 against 0.7698–0.7817. Eight times the parameters is not enough to produce a separate number. Nonlinear separation comes not from the number of layers but from the **activation function**; 0.7857 is the only number that rises above the others, and it belongs to the network with rectified units. #### The Signature of the Course: The Second Run The upper end, 0.7857, is a number that can be reported on its own, and when it is reported it misleads. The same network, the same data, the same setting, with only the initial values changed, is trained five times. ```python CANDIDATES = [] print(f"{'rate':>5} {'epoch':>5} {'validation':>10} {'test':>8}") for rate in (0.3, 0.5): for epochs in (100, 200): net = train_model(init_weights(SEED + 1, [12, 8, 1]), X_TR, Y_TR, rate, epochs) CANDIDATES.append((rate, epochs, accuracy(net, X_VA, Y_VA), accuracy(net, X_TE, Y_TE))) print(f"{rate:>5} {epochs:>5} {CANDIDATES[-1][2]:>10.4f} {CANDIDATES[-1][3]:>8.4f}") CHOSEN = max(CANDIDATES, key=lambda t: t[2]) print(f"candidates tried {len(CANDIDATES)}, chosen (rate {CHOSEN[0]}, epochs {CHOSEN[1]})\n") RUNS = {} print(f"{'run':>6} {'single-layer':>14} {'two-layer':>14}") for run in range(1, 6): row = [] for sizes in ([12, 1], [12, 8, 1]): net = train_model(init_weights(SEED + run, sizes), X_TR, Y_TR, CHOSEN[0], CHOSEN[1]) row.append(accuracy(net, X_TE, Y_TE)) RUNS[run] = row print(f"{run:>6} {row[0]:>14.4f} {row[1]:>14.4f}") single, two = [RUNS[k][0] for k in RUNS], [RUNS[k][1] for k in RUNS] print(f"\nbaseline {BASELINE:.4f}, network {two[0]:.4f}, " f"second run {two[1]:.4f}, difference {round(abs(two[0] - two[1]) * len(X_TE))} subscribers") for name, v in (("single-layer", single), ("two-layer", two)): print(f"{name:<13} band {min(v):.4f}-{max(v):.4f}, run-to-run variability " f"{max(v) - min(v):.4f} = {round((max(v) - min(v)) * len(X_TE))} subscribers") print(f"capacity adds {max(two) - max(single):.4f} " f"(subscribers: {round((max(two) - max(single)) * len(X_TE))}); cost of the reduction " f"(0.8016 at 756 rows) {0.8016 - max(single):.4f} " f"(subscribers: {round((0.8016 - max(single)) * len(X_TE))})") ``` ``` rate epoch validation test 0.3 100 0.8214 0.7857 0.3 200 0.8135 0.7817 0.5 100 0.8175 0.7857 0.5 200 0.7937 0.7738 candidates tried 4, chosen (rate 0.3, epochs 100) run single-layer two-layer 1 0.7817 0.7857 2 0.7698 0.7698 3 0.7738 0.7738 4 0.7698 0.7817 5 0.7698 0.7659 baseline 0.7579, network 0.7857, second run 0.7698, difference 4 subscribers single-layer band 0.7698-0.7817, run-to-run variability 0.0119 = 3 subscribers two-layer band 0.7659-0.7857, run-to-run variability 0.0198 = 5 subscribers capacity adds 0.0040 (subscribers: 1); cost of the reduction (0.8016 at 756 rows) 0.0199 (subscribers: 5) ``` The last two lines set the rule of the course. The difference that capacity adds is **one subscriber**, while the movement of the same capacity from run to run is **five subscribers**. The network's 0.7857 comes not from capacity but from the initial values of that run; the fifth run gives 0.7659, and that number is only two subscribers above the baseline. The band of the single-layer network is narrower, three subscribers, because the problem it solves has a single best point; when a hidden layer is added, that guarantee disappears. The second claim can also be read from the table. The best run of the network, 0.7857, equals the untuned decision tree and is four subscribers below the 0.8016 of the two families of the Supervised Learning course. Part of the gap comes from the reduction of the training set: the same single-layer network gives 0.8016 with 756 rows and 0.7817 with 300. The rest is about the data itself: in a subscriber table with twelve columns, there is little intermediate representation to learn. #### Summary - A unit is built from a weight, a bias and an activation function; the bias is a layer parameter and is separate from bias, the error component of the Introduction to Machine Learning course. - The network with no hidden layer, initialized from zero, produces all nine numbers of the linear model of the Supervised Learning course down to the last digit: 0.8016, 0.7937 and 0.8016. - If the activation of the hidden layer is left as the identity function, the network with 113 parameters cannot get out of the band of the one with 13; nonlinear separation comes from the activation, not from the number of layers. - The triple: baseline 0.7579, network's number 0.7857, second run 0.7698. The band of the five runs is 0.7659–0.7857; run-to-run variability is five subscribers, the difference capacity adds is one subscriber. - The network stays at 0.7857: equal to the untuned decision tree, four subscribers below the 0.8016 of the two families of the Supervised Learning course; five subscribers of the gap come from reducing the training set to 300 rows. #### Next Step It was measured that nonlinear separation comes from the activation, but the two functions used, the rectified linear unit that cuts everything below zero and the S-shaped squashing at the output, were not chosen with any justification. The next lesson treats that choice not as a preference but as **a number**: a squashing function flattens at the ends and its gradient falls to nearly zero, while a cutting function silences completely a unit whose input stays below zero. How many of the eight units are never activated on any example and by how many times saturation shrinks the gradient: three activations side by side, each with two runs. ### Activation Functions Source: https://academia.sh/en/courses/deep-learning/activation-functions Measuring the choice of activation through saturation, dead units and the gradient that reaches the first layer: the derivative of the sigmoid falls from 0.1966 at z=1 to 0.0025 at z=6, a ratio of 79.7; for the hyperbolic tangent the ratio is 17,088. In the same network the mean gradient reaching the first layer is 1.0456 with the sigmoid and 4.4498 with the rectified linear unit — 4.26 times. Three activations with two runs each: sigmoid 0.7937 and 0.7976, rectified 0.7857 and 0.7698, hyperbolic tangent 0.7659 and 0.7698; baseline 0.7579. When the learning rate is raised to 20, four of the rectified units die, and at 30 all eight do, and the mean derivative becomes exactly 0.0000; with the sigmoid and the hyperbolic tangent no unit dies, but the mean derivative falls from 0.1971 to 0.0295. The previous lesson measured that nonlinear separation comes from the activation function, but the two functions it used were not chosen with any justification. What an activation adds to a network is not only its curve; that function also determines the gradient that flows backward during training. The gradient is multiplied layer by layer through the chain rule to reach the earlier layers, and one of the factors is the **derivative** of the activation. The measure of this lesson lies here. If the derivative of a function falls to nearly zero in one region of the input, a unit that stays in that region receives no gradient and its learning stops. This has two separate forms, and they are confused: the derivative **shrinks but does not become zero** — **saturation**; or the derivative **is exactly zero** — a **dead unit**. The two are measured separately. - **NN8.** The set, the split, the seed and the twelve columns are as in `01`; the training set is again **reduced to 300 rows**. The baseline on the test set is 0.7579. - **NN9.** The network is the network of `01`: 12 inputs, a hidden layer of eight units, a single output, 113 parameters. The only thing that changes is **the activation of the hidden layer**. - **NN10.** The output layer is a sigmoid in all three setups, because the output has to be a probability. The comparison is made only through the hidden layer. - **NN11.** The derivative is written in terms of the **output** of the activation: $a(1-a)$ for the sigmoid, $1-a^2$ for the hyperbolic tangent, and for the rectified unit 1 when the output is positive and 0 otherwise. - **NN12.** A unit is **not counted as active** on a row if its derivative in the hidden layer stays below 5% of its ceiling. A unit that is not active on any of the 300 rows of the training set is **dead**. - **NN13.** The gradient reaching the first layer is the mean absolute magnitude of the weight gradients of **the first epoch**. Since the three setups start from the same initial values, the numbers are comparable. - **NN14.** The learning rate sweep is not a tuning search; it is made to show **at which rate** a dead unit appears, and it makes no choice. #### Saturation Is a Number When saturation is described in words, it remains a description of a curve. Once it is turned into a number, how sharp a boundary it is becomes visible: by how many times the derivative of the same function differs at two points of the input. ```python # activations.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split; # the training set is reduced to 300 rows as in 01. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def make_set(data): return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in data], [float(x["suspicious"]) for x in data]) X_TR, Y_TR = make_set(TRAIN[:300]) # REDUCED training set X_VA, Y_VA = make_set(VAL) X_TE, Y_TE = make_set(TEST) def squash(z): # S-shaped function, overflow-protected if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) ACTIVATIONS = { # (function, derivative — its input is the activation output) "sigmoid": (squash, lambda a: a * (1.0 - a)), "hyperbolic tangent": (math.tanh, lambda a: 1.0 - a * a), "rectified": (lambda z: z if z > 0.0 else 0.0, lambda a: 1.0 if a > 0.0 else 0.0), } Z = (0.0, 1.0, 2.0, 4.0, 6.0, 8.0) print(f"train {len(X_TR)}, validation {len(X_VA)}, test {len(X_TE)}, " f"baseline {BASELINE:.4f}") print("\n" + f"{'derivative':<20}" + "".join(f"{'z=' + str(int(z)):>10}" for z in Z)) for name, (f, deriv) in ACTIVATIONS.items(): print(f"{name:<20}" + "".join(f"{deriv(f(z)):>10.4f}" for z in Z)) for name, (f, deriv) in ACTIVATIONS.items(): print(f"{name:<20} at z=1 {deriv(f(1.0)):.6f}, at z=6 {deriv(f(6.0)):.6f}, " f"ratio {deriv(f(1.0)) / max(deriv(f(6.0)), 1e-12):.1f}") ``` ``` train 300, validation 252, test 252, baseline 0.7579 derivative z=0 z=1 z=2 z=4 z=6 z=8 sigmoid 0.2500 0.1966 0.1050 0.0177 0.0025 0.0003 hyperbolic tangent 1.0000 0.4200 0.0707 0.0013 0.0000 0.0000 rectified 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 sigmoid at z=1 0.196612, at z=6 0.002467, ratio 79.7 hyperbolic tangent at z=1 0.419974, at z=6 0.000025, ratio 17088.4 rectified at z=1 1.000000, at z=6 1.000000, ratio 1.0 ``` The derivative of the sigmoid does not exceed 0.25 anywhere. This alone is a limit: the gradient coming from an input of twelve columns falls to one quarter at best as it passes the hidden layer. When the input rises to 6, the derivative becomes 0.0025 and the ratio reaches 79.7. The hyperbolic tangent is more generous at the ceiling — its derivative starts from 1 — but it saturates faster: the ratio between the same two points is 17,088. The derivative of the rectified unit is a constant 1 in the positive region and exactly 0 in the negative region. #### Three Activations, Two Runs The table alone does not say which is better, because in which region of the input the network actually works has not been measured. The same network is trained with three activations, from the same initial values, with two runs each. ```python def init_weights(seed, sizes): # gain-scaled initialization, bias zero r, net = rng(seed), [] for a, b in zip(sizes, sizes[1:]): scale = math.sqrt(2.0 / a) net.append(([[(r() * 2 - 1) * scale for _ in range(a)] for _ in range(b)], [0.0] * b)) return net def forward(net, x, f): # FORWARD PASS, output layer is always a sigmoid a, acts = x, [x] for k, (W, bias) in enumerate(net): z = [sum(w * v for w, v in zip(s, a)) + b for s, b in zip(W, bias)] a = [squash(z[0])] if k == len(net) - 1 else [f(t) for t in z] acts.append(a) return acts def train_model(net, X, Y, rate, epochs, name): # MODEL: full-batch gradient descent f, deriv = ACTIVATIONS[name] n, first = len(X), [0.0] for t in range(epochs): gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net] gb = [[0.0] * len(bias) for _, bias in net] for i in range(n): acts = forward(net, X[i], f) d = [acts[-1][0] - Y[i]] for k in range(len(net) - 1, -1, -1): W, a = net[k][0], acts[k] for j, dj in enumerate(d): if dj: s = gW[k][j] for p in range(len(a)): s[p] += dj * a[p] gb[k][j] += dj if k: prev = [0.0] * len(a) for j, dj in enumerate(d): if dj: s = W[j] for p in range(len(a)): prev[p] += dj * s[p] d = [prev[p] * deriv(a[p]) for p in range(len(a))] if t == 0: # mean gradient magnitude reaching the first layer first[0] = sum(abs(v) for s in gW[0] for v in s) / (len(gW[0]) * len(gW[0][0])) for k, (W, bias) in enumerate(net): for j in range(len(W)): s, g = W[j], gW[k][j] for p in range(len(s)): s[p] -= rate * g[p] / n bias[j] -= rate * gb[k][j] / n return net, first[0] def accuracy(net, X, Y, name): f = ACTIVATIONS[name][0] return sum((forward(net, x, f)[-1][0] > 0.5) == (y > 0.5) for x, y in zip(X, Y)) / len(X) def unit_stats(net, X, name): # active rows per unit and mean derivative f, deriv = ACTIVATIONS[name] W, bias = net[0] ceiling = 0.25 if name == "sigmoid" else 1.0 counts, total = [0] * len(W), 0.0 for x in X: for j in range(len(W)): t = deriv(f(sum(w * v for w, v in zip(W[j], x)) + bias[j])) total += t if t > 0.05 * ceiling: counts[j] += 1 return counts, total / (len(X) * len(W)) RESULTS = {} print(f"{'activation':<20} {'run':>6} {'validation':>10} {'test':>8} " f"{'mean_deriv':>10} {'first_layer_grad':>17}") for name in ACTIVATIONS: RESULTS[name] = [] for run in (1, 2): net, first = train_model(init_weights(SEED + run, [12, 8, 1]), X_TR, Y_TR, 0.3, 100, name) counts, mean_deriv = unit_stats(net, X_TR, name) RESULTS[name].append((accuracy(net, X_VA, Y_VA, name), accuracy(net, X_TE, Y_TE, name), mean_deriv, first)) print(f"{name:<20} {run:>6} {RESULTS[name][-1][0]:>10.4f} {RESULTS[name][-1][1]:>8.4f} " f"{mean_deriv:>10.4f} {first:>17.4f}") CHOSEN = max(ACTIVATIONS, key=lambda a: RESULTS[a][0][0]) print(f"\nactivation chosen on validation: {CHOSEN}") print(f"baseline {BASELINE:.4f}, network number {RESULTS[CHOSEN][0][1]:.4f}, " f"second run {RESULTS[CHOSEN][1][1]:.4f}") print(f"first-layer gradient: {CHOSEN} {RESULTS[CHOSEN][0][3]:.4f}, rectified " f"{RESULTS['rectified'][0][3]:.4f}, ratio " f"{RESULTS['rectified'][0][3] / RESULTS[CHOSEN][0][3]:.2f} times") ``` ``` activation run validation test mean_deriv first_layer_grad sigmoid 1 0.8294 0.7937 0.1971 1.0456 sigmoid 2 0.8333 0.7976 0.2100 1.5007 hyperbolic tangent 1 0.7976 0.7659 0.5913 3.1693 hyperbolic tangent 2 0.8016 0.7698 0.6587 4.8762 rectified 1 0.8214 0.7857 0.5183 4.4498 rectified 2 0.8056 0.7698 0.5012 5.2971 activation chosen on validation: sigmoid baseline 0.7579, network number 0.7937, second run 0.7976 first-layer gradient: sigmoid 1.0456, rectified 4.4498, ratio 4.26 times ``` The gradient measure confirms the table: the mean gradient magnitude reaching the first layer is 1.0456 with the sigmoid and 4.4498 with the rectified unit — **4.26 times**. This difference comes directly from the derivative ceiling and is the counterpart in the network of the saturation described in words. But the test column says the opposite of what is expected. The saturating function gives the highest numbers: 0.7937 and 0.7976. The reason is in the mean derivative column: the mean derivative of the sigmoid units is 0.1971, 79% of its ceiling of 0.2500. The network is not working in the saturated region, because with scaled inputs the weighted sum stays small. Saturation is not a property of a function but a consequence of **the region in which that function is run**. At this scale a small gradient is not an obstacle; it will be one when the layers deepen and the same factor stacks up. The triple is again in place: baseline 0.7579, network number 0.7937, second run 0.7976. The difference between the two runs is one subscriber; but the largest difference between the three activations is 0.7937 against 0.7659, that is, seven subscribers. In this lesson the choice produces a difference above the run-to-run noise. #### Dead Unit and Saturated Unit The way to tell the two forms of degradation apart is to increase the learning rate. As the step grows, the weights are thrown further and the units enter the region where the derivative is small. ```python print(f"{'activation':<20} {'rate':>6} {'test':>8} {'dead units':>10} " f"{'mean_deriv':>10} active rows per unit") for name in ACTIVATIONS: for rate in (0.3, 20.0, 30.0): net, _ = train_model(init_weights(SEED + 1, [12, 8, 1]), X_TR, Y_TR, rate, 100, name) counts, mean_deriv = unit_stats(net, X_TR, name) print(f"{name:<20} {rate:>6} {accuracy(net, X_TE, Y_TE, name):>8.4f} " f"{sum(1 for c in counts if c == 0):>10} {mean_deriv:>10.4f} {counts}") ``` ``` activation rate test dead units mean_deriv active rows per unit sigmoid 0.3 0.7937 0 0.1971 [300, 300, 300, 300, 300, 300, 300, 300] sigmoid 20.0 0.7579 0 0.0536 [176, 127, 118, 119, 132, 169, 141, 127] sigmoid 30.0 0.6071 0 0.0295 [89, 68, 66, 70, 53, 95, 85, 71] hyperbolic tangent 0.3 0.7659 0 0.5913 [296, 299, 291, 291, 283, 289, 288, 282] hyperbolic tangent 20.0 0.7381 0 0.0528 [28, 37, 62, 27, 35, 36, 11, 47] hyperbolic tangent 30.0 0.6667 0 0.0259 [5, 26, 17, 22, 15, 18, 15, 24] rectified 0.3 0.7857 0 0.5183 [151, 185, 143, 142, 154, 134, 132, 203] rectified 20.0 0.7579 4 0.0825 [0, 30, 91, 33, 44, 0, 0, 0] rectified 30.0 0.7540 8 0.0000 [0, 0, 0, 0, 0, 0, 0, 0] ``` The distinction is in the last column. With the sigmoid and the hyperbolic tangent **no unit dies**: even at rate 30 every unit stays active on a part of the 300 rows, between 53 and 95. The mean derivative falls from 0.1971 to 0.0295 — a six-fold shrinkage — and the number drops to 0.6071, far below the baseline. But the derivative is not zero; the network can keep learning, even if slowly. With the rectified unit the situation is different. At rate 20, **four** of the eight units die; at rate 30, **all eight** do, and the mean derivative becomes exactly **0.0000**. The gradient flowing back through a dead unit is multiplied by zero; its weights are never updated again. This is an irreversible loss and it shows in the table as a number: when eight of eight units are dead, the output of the network is constant and the test number is 0.7540, that is, **below even the baseline**. With four units dead at rate 20, the number is exactly 0.7579, that is, the network tells everyone "not suspicious". #### Summary - Saturation is a number: the derivative of the sigmoid is 0.1966 at z=1 and 0.0025 at z=6, a ratio of 79.7; for the hyperbolic tangent the same ratio is 17,088. The derivative of the rectified unit is a constant 1 in the positive region. - In the same network the mean gradient reaching the first layer is 1.0456 with the sigmoid and 4.4498 with the rectified unit — 4.26 times. The difference comes directly from the derivative ceiling. - The triple: baseline 0.7579, network number 0.7937 (sigmoid), second run 0.7976. The largest difference between the activations is seven subscribers, the difference between the two runs is one subscriber. - Saturation is not a property of a function but a consequence of the region in which it is run: the mean derivative of the sigmoid units is 0.1971, that is 79% of its ceiling, because with scaled inputs the sum stays small. - A dead unit and a saturated unit are different things. At rate 30 no unit dies with the sigmoid and the hyperbolic tangent, while with the rectified unit eight of eight die and the mean derivative becomes 0.0000; the number falls to 0.7540, below the baseline. #### Next Step In the two lessons the network always stayed the same: twelve inputs, eight units, a single output. **How many units** the hidden layer would carry was not chosen, and what would change if it were chosen was not measured. The next lesson opens the forward pass layer by layer and ties two separate numbers to this choice: how the **parameter count** and the **number of multiplications** made for a single example grow as the unit count increases, and how much the held-out set count responds to that growth. The matrix multiplication is written by hand, because the cost of a layer can be counted only when it is written out explicitly. ### Forward Propagation Source: https://academia.sh/en/courses/deep-learning/forward-propagation Measuring the width of the hidden layer with two cost numbers and one return number: in a network with 12 inputs, when the hidden layer grows from 2 to 32 units, the parameter count grows from 29 to 449 (15.5 times), the multiplications per example from 26 to 416 (16.0 times), and the total multiplications of a hundred-epoch training from 780,000 to 12,480,000. The return stays small beside this: the test band of ten runs is 0.7540-0.7857, that is, eight subscribers; the difference between the good runs of each width is five subscribers and the largest difference between two runs at a single width is also five subscribers. The width chosen from validation is 4 units, that is, 57 parameters, and the test number of that candidate is 0.7540, below the baseline; its second run is 0.7738. In the two lessons the network always stayed the same: twelve inputs, eight units, a single output. How many units the hidden layer would carry was never chosen, and what would change if it were chosen was not measured. This lesson ties two cost numbers and one return number to that choice. The cost side is found by calculation and does not change from run to run: the **parameter count** — the number of weights and biases the network holds in memory — and the **multiplications per example** — the number of multiplications made for one subscriber to pass through the network once. Both are read from the structure of the forward pass. The return side can be found only by measuring and, by the rule of this course, is written with **two runs**. - **NN15.** The set, the split, the seed, the twelve columns and the training set **reduced to 300 rows** are as in `01`–`02`. The baseline on the test set is 0.7579. - **NN16.** The activation of the hidden layer in this lesson is the **rectified linear unit** and is held fixed; `02` measured three activations, and here the only variable is the width. - **NN17.** The output layer is a sigmoid, because the output has to be a probability. The threshold is 0.5, and that the threshold is a decision was measured in the Logistic Regression lesson of the Supervised Learning course. - **NN18.** The parameter count is computed over layer pairs as $\sum (a_i b_i + b_i)$ and the multiplications per example as $\sum a_i b_i$; $a_i$ is the input and $b_i$ the output width. Additions and activation calls are not counted, because the measure is the number of multiplications. - **NN19.** The total multiplications in training is the multiplications per example times 300 rows and 100 epochs; the backward pass is not counted, the measure is only that of the forward pass. - **NN20.** The number of widths tried is **5** (2, 4, 8, 16, 32) and the choice is made from the validation set, by looking at the sum of the two runs. Choosing by looking at a single run is not accepted in this course. - **NN21.** The learning rate is 0.3 and the number of epochs is 100; this is the setting chosen from validation in `01` and is held fixed throughout the width sweep. #### Two Cost Numbers The cost of a layer is read from its shape. A layer going from twelve inputs to eight units carries 96 weights and eight biases; the layer going from eight units to a single output carries eight weights and one bias. The total is 113. The same calculation is made for multiplications, and the biases stay out, because a bias is added, not multiplied. ```python # forward_propagation.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same # split; the training set is reduced to 300 rows as in 01. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def make_set(data): return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in data], [float(x["suspicious"]) for x in data]) X_TR, Y_TR = make_set(TRAIN[:300]) # REDUCED training set X_VA, Y_VA = make_set(VAL) X_TE, Y_TE = make_set(TEST) def squash(z): # S-shaped function, overflow-protected if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) UNITS = (2, 4, 8, 16, 32) COST = {} print(f"{'units':>6} {'shape':>12} {'params':>10} {'mults/example':>13} " f"{'total mults in training':>23}") for h in UNITS: sizes = [len(FIELDS), h, 1] params = sum(a * b + b for a, b in zip(sizes, sizes[1:])) mults = sum(a * b for a, b in zip(sizes, sizes[1:])) COST[h] = (params, mults) print(f"{h:>6} {str(tuple(sizes)):>12} {params:>10} {mults:>13} " f"{mults * len(X_TR) * 100:>23}") p0, c0 = COST[UNITS[0]] p1, c1 = COST[UNITS[-1]] print(f"\nunits {UNITS[0]} to {UNITS[-1]}: params {p0} -> {p1} " f"({p1 / p0:.1f} times), mults {c0} -> {c1} ({c1 / c0:.1f} times)") ``` ``` units shape params mults/example total mults in training 2 (12, 2, 1) 29 26 780000 4 (12, 4, 1) 57 52 1560000 8 (12, 8, 1) 113 104 3120000 16 (12, 16, 1) 225 208 6240000 32 (12, 32, 1) 449 416 12480000 units 2 to 32: params 29 -> 449 (15.5 times), mults 26 -> 416 (16.0 times) ``` Both columns grow **in direct proportion** to the width, because the twelve inputs are connected to each unit separately. When the width grows sixteen-fold, the parameters grow 15.5-fold and the multiplications 16.0-fold. The last column shows that the measure is not time: a hundred-epoch training does 780,000 multiplications in the narrowest network and 12,480,000 in the widest. These numbers do not depend on which machine they are run on. #### Matrix Multiplication by Hand The forward pass is the repetition of two steps: a **matrix multiplication** and an **activation**. Each row of the matrix multiplication belongs to one unit and pairs the weights of that unit with the input vector. When it is written by hand, where the cost comes from becomes visible: two nested loops, the outer one turning as many times as the number of units, the inner one as many times as the number of inputs. ```python def rectified(z): # activation that cuts everything below zero to zero return z if z > 0.0 else 0.0 def init_weights(seed, sizes): # gain-scaled initialization, bias zero r, net = rng(seed), [] for a, b in zip(sizes, sizes[1:]): scale = math.sqrt(2.0 / a) net.append(([[(r() * 2 - 1) * scale for _ in range(a)] for _ in range(b)], [0.0] * b)) return net def matmul(W, bias, a): # MATRIX MULTIPLICATION by hand: each row is a unit return [sum(w * v for w, v in zip(row, a)) + b for row, b in zip(W, bias)] def forward(net, x): # FORWARD PASS: layer by layer a = x for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(t) for t in z] return a def forward_trace(net, x): # the same pass, printing every layer a, mults = x, 0 print(f" input layer width {len(a):>3}, first three {[round(v, 3) for v in a[:3]]}") for k, (W, bias) in enumerate(net): mults += len(W) * len(a) z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(t) for t in z] name = "output layer" if k == len(net) - 1 else f"hidden layer {k + 1}" print(f" {name:<18} width {len(a):>3}, first three {[round(v, 3) for v in a[:3]]}") return a[0], mults for h in (4, 8): net = init_weights(SEED + 1, [len(FIELDS), h, 1]) print(f"{h} units in the hidden layer, first subscriber of the test set:") output, mults = forward_trace(net, X_TE[0]) print(f" multiplications made {mults}, in the table {COST[h][1]}, " f"output {output:.4f}, true label {int(Y_TE[0])}") ``` ``` 4 units in the hidden layer, first subscriber of the test set: input layer width 12, first three [-0.008, -0.63, 1.399] hidden layer 1 width 4, first three [0.047, 0.0, 0.673] output layer width 1, first three [0.407] multiplications made 52, in the table 52, output 0.4068, true label 0 8 units in the hidden layer, first subscriber of the test set: input layer width 12, first three [-0.008, -0.63, 1.399] hidden layer 1 width 8, first three [0.047, 0.0, 0.673] output layer width 1, first three [0.492] multiplications made 104, in the table 104, output 0.4923, true label 0 ``` The counted multiplications agree exactly with the calculation in the table: 52 and 104. The shape of the pass is visible too — 12 values drop to 4 values, 4 values to 1 value. The output of the second unit of the hidden layer is 0.0; since its weighted sum of the input stayed negative, the rectified unit cut it off. For the same subscriber the two networks give 0.4068 and 0.4923; both are below the 0.5 threshold and, since the true label is 0, both are correct. #### How Much Does Width Return While the cost grows sixteen-fold, what the return is can be found only by training. Five widths, two runs each. ```python def train_model(net, X, Y, rate, epochs): # MODEL: full-batch gradient descent n = len(X) for _ in range(epochs): gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net] gb = [[0.0] * len(bias) for _, bias in net] for i in range(n): a, acts = X[i], [X[i]] for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(t) for t in z] acts.append(a) d = [acts[-1][0] - Y[i]] for k in range(len(net) - 1, -1, -1): W, a = net[k][0], acts[k] for j, dj in enumerate(d): if dj: s = gW[k][j] for p in range(len(a)): s[p] += dj * a[p] gb[k][j] += dj if k: prev = [0.0] * len(a) for j, dj in enumerate(d): if dj: s = W[j] for p in range(len(a)): prev[p] += dj * s[p] d = [prev[p] if a[p] > 0 else 0.0 for p in range(len(a))] for k, (W, bias) in enumerate(net): for j in range(len(W)): s, g = W[j], gW[k][j] for p in range(len(s)): s[p] -= rate * g[p] / n bias[j] -= rate * gb[k][j] / n return net def accuracy(net, X, Y): return sum((forward(net, x)[0] > 0.5) == (y > 0.5) for x, y in zip(X, Y)) / len(X) RESULTS = {} print(f"{'units':>6} {'params':>10} {'run':>6} {'validation':>10} {'test':>8}") for h in UNITS: RESULTS[h] = [] for run in (1, 2): net = train_model(init_weights(SEED + run, [len(FIELDS), h, 1]), X_TR, Y_TR, 0.3, 100) RESULTS[h].append((accuracy(net, X_VA, Y_VA), accuracy(net, X_TE, Y_TE))) print(f"{h:>6} {COST[h][0]:>10} {run:>6} {RESULTS[h][-1][0]:>10.4f} " f"{RESULTS[h][-1][1]:>8.4f}") CHOSEN = max(UNITS, key=lambda h: sum(t[0] for t in RESULTS[h])) all_runs = [t[1] for h in UNITS for t in RESULTS[h]] best_runs = [max(t[1] for t in RESULTS[h]) for h in UNITS] run_gap = max(abs(RESULTS[h][0][1] - RESULTS[h][1][1]) for h in UNITS) print(f"\nwidth chosen on validation {CHOSEN} units ({COST[CHOSEN][0]} parameters)") print(f"baseline {BASELINE:.4f}, network number {RESULTS[CHOSEN][0][1]:.4f}, " f"second run {RESULTS[CHOSEN][1][1]:.4f}") print(f"band of ten runs {min(all_runs):.4f}-{max(all_runs):.4f} " f"({round((max(all_runs) - min(all_runs)) * len(X_TE))} subscribers)") print(f"what width adds (the good run of each width) {min(best_runs):.4f}-{max(best_runs):.4f} " f"= {round((max(best_runs) - min(best_runs)) * len(X_TE))} subscribers") print(f"largest run difference at a single width {run_gap:.4f} " f"= {round(run_gap * len(X_TE))} subscribers") ``` ``` units params run validation test 2 29 1 0.8135 0.7579 2 29 2 0.8016 0.7659 4 57 1 0.8254 0.7540 4 57 2 0.8135 0.7738 8 113 1 0.8214 0.7857 8 113 2 0.8056 0.7698 16 225 1 0.8135 0.7659 16 225 2 0.7937 0.7778 32 449 1 0.8056 0.7817 32 449 2 0.8056 0.7698 width chosen on validation 4 units (57 parameters) baseline 0.7579, network number 0.7540, second run 0.7738 band of ten runs 0.7540-0.7857 (8 subscribers) what width adds (the good run of each width) 0.7659-0.7857 = 5 subscribers largest run difference at a single width 0.0198 = 5 subscribers ``` The last two lines should be read side by side. A 15.5-fold difference in parameters produces **five subscribers** between the good runs of each width. At a single width, with nothing changed except the initial values, the largest difference is again **five subscribers**. The difference between doubling the width and retraining the network cannot be told apart to this extent. The validation column carries a separate warning. When the choice is made from validation, the winner is **4 units**, that is, one of the narrowest of the five candidates; with 57 parameters. The test number of that candidate is 0.7540 and this is **below the baseline** — saying the most frequent class gives 0.7579. The second run of the same candidate gives 0.7738; there are five subscribers between the two runs and the validation set cannot choose between them, because on validation too it moves between 0.8254 and 0.8135. The difference that the tuning budget inflates had been measured in the Hyperparameter Search lesson of the Supervised Learning course; what is added here is that the movement **within** the same candidate spoils the choice. #### Summary - The forward pass is the repetition of a matrix multiplication and an activation. Each row of the matrix multiplication belongs to one unit and, when it is written by hand, the cost is read from two nested loops. - When the hidden layer grows from 2 to 32 units, the parameters grow from 29 to 449 (15.5 times), the multiplications per example from 26 to 416 (16.0 times), and the total multiplications of a hundred-epoch training from 780,000 to 12,480,000. All three numbers are independent of the machine. - When a single subscriber is traced, the counted multiplications agree exactly with the calculation in the table: 52 with 4 units, 104 with 8 units; 12 values drop first to the width of the hidden layer, then to a single probability. - The triple: baseline 0.7579, network number 0.7540, second run 0.7738. The width chosen by validation is 4 units and its first run stays below the baseline. - A 15.5-fold difference in parameters amounts to five subscribers, and the difference between two runs at the same width also amounts to five subscribers; all ten runs stand in a band between 0.7540 and 0.7857, a band of eight subscribers. #### Next Step In three lessons the output of the network was always a probability, and that probability was always compared with the target in the same way: the difference was taken and passed backward. This comparison is itself a choice and has never been discussed. The next lesson treats the **loss function** not as a preference but as a number: if the same network, the same data, is trained with two separate losses, do they give two separate results, and how many subscribers is the difference. In the same lesson a second distinction is built — the loss that governs training and the metric that reports performance are not the same thing, and a change that lowers the loss can worsen the metric. ### Loss Functions Source: https://academia.sh/en/courses/deep-learning/loss-functions Measuring how the choice of loss moves the gradient the network sees and the number it reaches: when the network says 0.99 and the true label is 0, the gradient that cross-entropy gives to the output layer is 0.9900 and that of squared error 0.0196 — a ratio of 50.5. When the same network is trained with two losses on four candidates, validation picks squared error and 25 epochs; the test number of that candidate is 0.7619, its second run 0.7897, a difference of seven subscribers. What the choice of loss adds is three subscribers. While the training loss always falls from 0.4861 to 0.3279 and the training accuracy rises from 0.78 to 0.86, the test accuracy peaks at 0.7937 at epoch 50 and falls to 0.7817; in the same run the number of suspicious subscribers caught rises from 1 to 25. In three lessons the output of the network was always a probability, and that probability was always compared with the target in the same way: the difference was taken and passed back. This comparison is a choice. The number passed back is the gradient of the **loss function** with respect to the output layer, and when the loss changes, that number changes too. The loss does two jobs at once: during training it tells the network which way to go, and when training is over it is read, or one tries to read it, as the number that reports performance. It was not designed for the second job, and the second measure of the lesson shows this. - **NN22.** The set, the split, the seed, the twelve columns and the training set **reduced to 300 rows** are as in `01`–`03`; the network is 12–8–1, 113 parameters. The baseline on the test set is 0.7579. - **NN23.** The classification losses are **cross-entropy** and **squared error**. Both are computed over the same sigmoid output; the only thing that changes is the gradient given to the output layer. - **NN24.** The output gradient of cross-entropy is $a-y$; the derivative of the sigmoid cancels against the derivative of the loss. The gradient of squared error is $2(a-y)\,a(1-a)$ and it keeps carrying the derivative of the sigmoid. - **NN25.** In cross-entropy the probability is bounded from below by $10^{-12}$ as overflow protection. The bound does not affect the number, because no probability the network produces comes near that value. - **NN26.** The regression losses are **not trained** in this lesson; only the gradient behavior is shown by calculation. In the Regression Metrics lesson of the Supervised Learning course two fits were measured: explained variance 0.4770 and 0.4751. - **NN27.** The number of candidates tried is **4**: two losses times two epoch budgets (25 and 100). The choice is made from the validation set, by looking at the sum of the two runs. - **NN28.** The second metric is **the number of suspicious subscribers caught**: the real suspicious subscribers flagged at the 0.5 threshold, out of the 61 suspicious ones in the test set. The design of the metric was measured in the Classification Metrics lesson of the Supervised Learning course. #### Choosing the Loss Changes the Gradient The difference between the two losses shows not in the loss value but in the **derivative**. The network says 0.99 to a subscriber and the true label is 0. This is one of the worst errors possible; how the loss reacts to it can be measured. ```python # loss.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split; # the training set is reduced to 300 rows as in 01. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def make_set(data): return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in data], [float(x["suspicious"]) for x in data]) X_TR, Y_TR = make_set(TRAIN[:300]) # REDUCED training set X_VA, Y_VA = make_set(VAL) X_TE, Y_TE = make_set(TEST) def squash(z): # S-shaped function, overflow-protected if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def rectified(z): return z if z > 0.0 else 0.0 def init_weights(seed, sizes): # gain-scaled initialization, bias zero r, net = rng(seed), [] for a, b in zip(sizes, sizes[1:]): scale = math.sqrt(2.0 / a) net.append(([[(r() * 2 - 1) * scale for _ in range(a)] for _ in range(b)], [0.0] * b)) return net def matmul(W, bias, a): # MATRIX MULTIPLICATION by hand: each row is a unit return [sum(w * v for w, v in zip(row, a)) + b for row, b in zip(W, bias)] def forward(net, x): # FORWARD PASS: layer by layer a = x for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(t) for t in z] return a LOSSES = { # (loss, gradient at the output layer) "cross-entropy": ( lambda a, y: -(math.log(max(a, 1e-12)) if y else math.log(max(1 - a, 1e-12))), lambda a, y: a - y), "squared error": ( lambda a, y: (a - y) ** 2, lambda a, y: 2.0 * (a - y) * a * (1.0 - a)), } print(f"train {len(X_TR)}, validation {len(X_VA)}, test {len(X_TE)}, " f"baseline {BASELINE:.4f}\n") print("classification, true label 0:") print(f"{'output':>7} {'cross loss':>13} {'cross grad':>12} {'squared loss':>12} " f"{'squared grad':>12} {'ratio':>7}") for a in (0.01, 0.10, 0.50, 0.90, 0.99): cl, cg = LOSSES["cross-entropy"] sl, sg = LOSSES["squared error"] print(f"{a:>7.2f} {cl(a, 0.0):>13.4f} {cg(a, 0.0):>12.4f} {sl(a, 0.0):>12.4f} " f"{sg(a, 0.0):>12.4f} {cg(a, 0.0) / sg(a, 0.0):>7.1f}") print("\nregression, residual r:") print(f"{'r':>7} {'squared loss':>12} {'squared grad':>12} {'absolute loss':>13} " f"{'absolute grad':>13} {'ratio':>7}") for r in (0.5, 1.0, 2.0, 5.0, 10.0): print(f"{r:>7.1f} {r * r:>12.4f} {2 * r:>12.4f} {r:>13.4f} {1.0:>13.4f} " f"{2 * r:>7.1f}") ``` ``` train 300, validation 252, test 252, baseline 0.7579 classification, true label 0: output cross loss cross grad squared loss squared grad ratio 0.01 0.0101 0.0100 0.0001 0.0002 50.5 0.10 0.1054 0.1000 0.0100 0.0180 5.6 0.50 0.6931 0.5000 0.2500 0.2500 2.0 0.90 2.3026 0.9000 0.8100 0.1620 5.6 0.99 4.6052 0.9900 0.9801 0.0196 50.5 regression, residual r: r squared loss squared grad absolute loss absolute grad ratio 0.5 0.2500 1.0000 0.5000 1.0000 1.0 1.0 1.0000 2.0000 1.0000 1.0000 2.0 2.0 4.0000 4.0000 2.0000 1.0000 4.0 5.0 25.0000 10.0000 5.0000 1.0000 10.0 10.0 100.0000 20.0000 10.0000 1.0000 20.0 ``` The last row decides the classification question. When the network says 0.99 and is wrong, cross-entropy sends 0.9900 to the output layer and squared error 0.0196 — there is a **50.5-fold** difference between them. Because the gradient of squared error carries the derivative of the sigmoid, the more confidently the network is wrong, the weaker the correction becomes. The saturation measured in `02` meets us here once more, this time through the loss. The regression table shows the same logic in the opposite direction. When the residual is 10 units, the gradient of squared error is 20 and that of absolute error 1. Squared error brings large residuals to the front and a single outlying observation pulls the fit toward itself; absolute error gives every residual the same weight. Which one is right is a question not of the data but of the cost of the error. #### The Same Network, Two Losses Whether the difference in derivative turns into a number can be understood only by training. The same network, the same starting values, the same learning rate; the only things that change are the loss and the epoch budget. ```python def train_model(net, X, Y, rate, epochs, loss_name, watch=()): # MODEL: full-batch gradient descent loss_fn, output_grad = LOSSES[loss_name] n, trace = len(X), [] for t in range(epochs): gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net] gb = [[0.0] * len(bias) for _, bias in net] total = 0.0 for i in range(n): a, acts = X[i], [X[i]] for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(q) for q in z] acts.append(a) total += loss_fn(acts[-1][0], Y[i]) d = [output_grad(acts[-1][0], Y[i])] for k in range(len(net) - 1, -1, -1): W, a = net[k][0], acts[k] for j, dj in enumerate(d): if dj: s = gW[k][j] for p in range(len(a)): s[p] += dj * a[p] gb[k][j] += dj if k: prev = [0.0] * len(a) for j, dj in enumerate(d): if dj: s = W[j] for p in range(len(a)): prev[p] += dj * s[p] d = [prev[p] if a[p] > 0 else 0.0 for p in range(len(a))] for k, (W, bias) in enumerate(net): for j in range(len(W)): s, g = W[j], gW[k][j] for p in range(len(s)): s[p] -= rate * g[p] / n bias[j] -= rate * gb[k][j] / n if (t + 1) in watch: trace.append((t + 1, total / n, accuracy(net, X_TR, Y_TR), accuracy(net, X_VA, Y_VA), accuracy(net, X_TE, Y_TE), caught(net, X_TE, Y_TE))) return net, trace def accuracy(net, X, Y): return sum((forward(net, x)[0] > 0.5) == (y > 0.5) for x, y in zip(X, Y)) / len(X) def caught(net, X, Y): # real suspicious subscribers flagged at the 0.5 threshold return sum(1 for x, y in zip(X, Y) if y > 0.5 and forward(net, x)[0] > 0.5) CANDIDATES = {} print(f"{'loss':<20} {'epoch':>5} {'run':>6} {'train_loss':>10} {'validation':>10} " f"{'test':>8}") for loss_name in LOSSES: for run in (1, 2): _, trace = train_model(init_weights(SEED + run, [12, 8, 1]), X_TR, Y_TR, 0.3, 100, loss_name, watch=(25, 100)) for epoch, loss, _, val, test, _ in trace: CANDIDATES.setdefault((loss_name, epoch), []).append((val, test)) print(f"{loss_name:<20} {epoch:>5} {run:>6} {loss:>10.4f} {val:>10.4f} {test:>8.4f}") CHOSEN = max(CANDIDATES, key=lambda k: sum(t[0] for t in CANDIDATES[k])) print(f"\ncandidates tried {len(CANDIDATES)}, chosen from validation: {CHOSEN[0]}, " f"{CHOSEN[1]} epochs") print(f"baseline {BASELINE:.4f}, network number {CANDIDATES[CHOSEN][0][1]:.4f}, " f"second run {CANDIDATES[CHOSEN][1][1]:.4f}, difference " f"{round(abs(CANDIDATES[CHOSEN][0][1] - CANDIDATES[CHOSEN][1][1]) * len(X_TE))} subscribers") best = {k: max(t[1] for t in CANDIDATES[k]) for k in CANDIDATES} print(f"loss choice (best run of each candidate) {min(best.values()):.4f}-{max(best.values()):.4f} " f"= {round((max(best.values()) - min(best.values())) * len(X_TE))} subscribers") ``` ``` loss epoch run train_loss validation test cross-entropy 25 1 0.4341 0.8135 0.7778 cross-entropy 100 1 0.3601 0.8214 0.7857 cross-entropy 25 2 0.4152 0.8333 0.7778 cross-entropy 100 2 0.3648 0.8056 0.7698 squared error 25 1 0.1556 0.8135 0.7619 squared error 100 1 0.1312 0.8214 0.7857 squared error 25 2 0.1536 0.8373 0.7897 squared error 100 2 0.1232 0.8056 0.7738 candidates tried 4, chosen from validation: squared error, 25 epochs baseline 0.7579, network number 0.7619, second run 0.7897, difference 7 subscribers loss choice (best run of each candidate) 0.7778-0.7897 = 3 subscribers ``` The loss columns cannot be compared directly, because the two losses produce numbers on two different scales: 0.4341 against 0.1556. What can be compared is the test column, and the difference between the two losses is **three subscribers**, while the difference between the two runs of the same candidate is **seven subscribers**. The epoch budget makes the gradient difference visible. At twenty-five epochs squared error stays at 0.7619 while cross-entropy reaches 0.7778; at a hundred epochs both rise to 0.7857. The weak gradient of squared error does not stop learning, it **delays** it. #### Loss Is Not a Metric Until now the loss only steered the training. The table below follows the same run epoch by epoch and puts three numbers side by side: the training loss, the test accuracy and the number of suspicious subscribers caught. ```python WATCH = (10, 25, 50, 100, 200) print(f"suspicious subscribers in the test set {int(sum(Y_TE))}") print(f"{'run':>6} {'epoch':>5} {'train_loss':>10} {'on_train':>9} {'on_test':>9} " f"{'caught':>10}") TRACES = {} for run in (1, 2): _, trace = train_model(init_weights(SEED + run, [12, 8, 1]), X_TR, Y_TR, 0.3, WATCH[-1], "cross-entropy", watch=WATCH) TRACES[run] = trace for epoch, loss, acc_train, _, acc_test, n_caught in trace: print(f"{run:>6} {epoch:>5} {loss:>10.4f} {acc_train:>9.4f} {acc_test:>9.4f} " f"{n_caught:>10}") for run in (1, 2): trace = TRACES[run] peak = max(trace, key=lambda t: t[4]) print(f"run {run}: loss {trace[0][1]:.4f} -> {trace[-1][1]:.4f} (always falling), " f"test highest at epoch {peak[0]}: {peak[4]:.4f}, " f"at epoch {WATCH[-1]}: {trace[-1][4]:.4f}") print(f" caught suspicious {trace[0][5]} -> {trace[-1][5]}, " f"highest {max(t[5] for t in trace)}") ``` ``` suspicious subscribers in the test set 61 run epoch train_loss on_train on_test caught 1 10 0.4861 0.7800 0.7619 1 1 25 0.4341 0.7933 0.7778 7 1 50 0.3899 0.8133 0.7937 20 1 100 0.3601 0.8400 0.7857 21 1 200 0.3279 0.8600 0.7817 25 2 10 0.4995 0.7967 0.7817 10 2 25 0.4152 0.8233 0.7778 16 2 50 0.3805 0.8233 0.7738 23 2 100 0.3648 0.8367 0.7698 25 2 200 0.3518 0.8400 0.7738 25 run 1: loss 0.4861 -> 0.3279 (always falling), test highest at epoch 50: 0.7937, at epoch 200: 0.7817 caught suspicious 1 -> 25, highest 25 run 2: loss 0.4995 -> 0.3518 (always falling), test highest at epoch 10: 0.7817, at epoch 200: 0.7738 caught suspicious 10 -> 25, highest 25 ``` The loss column is one-directional in both runs: it always falls. The training accuracy is one-directional too, from 0.78 to 0.86. The test accuracy is not. In the first run it peaks at 0.7937 at epoch 50 and drops to 0.7817 at epoch 200; in the second run the peak is at epoch 10 and after that it only goes down. The epochs that keep pushing the loss down make the accuracy on the held-out set worse. The last column completes the distinction. The number of suspicious subscribers caught **rises** in both runs: from 1 to 25 and from 10 to 25. At the same moment of the same training, one metric gets worse while another gets better. At epoch 10 the network catches only one of the 61 suspicious subscribers, yet its accuracy is 0.7619; at epoch 200 it catches 25 of them and its accuracy is 0.7817. The sentence "the network was best at epoch 50" is true for the one who reads accuracy and false for the one who reads the suspicious subscribers caught. At which epoch to stop is a question not of the loss but of the metric and of the cost behind the metric; that calculation is the subject of the Threshold Selection and Curves lesson of the Supervised Learning course. #### Summary - The choice of loss is decisive not in the loss value but in the derivative. When the network says 0.99 and is wrong, the output gradient of cross-entropy is 0.9900 and that of squared error 0.0196 — a ratio of 50.5. - In regression the same logic works in the opposite direction: when the residual is 10 units, the gradient of squared error is 20 and that of absolute error 1. Squared error brings large residuals to the front, absolute error gives all of them equal weight. - The triple: baseline 0.7579, network number 0.7619, second run 0.7897. What the choice of loss adds is three subscribers, the run difference of the same candidate is seven subscribers. - The weak gradient of squared error does not stop learning, it delays it: at 25 epochs they separate at 0.7619 and 0.7778, at 100 epochs both come to 0.7857. - Loss is not a metric. While the training loss always falls from 0.4861 to 0.3279, the test accuracy peaks at epoch 50 and drops, and the number of suspicious subscribers caught rises from 1 to 25. #### Next Step In four lessons the gradient always arrived ready-made. How the number at the output layer is found was written in this lesson, but **how that number is distributed** over the weights of the hidden layer was never opened. The next lesson opens that box: the chain rule is applied layer by layer over the network and the resulting gradient is compared with the **numerical derivative**. The measure is two numbers — the largest difference between the hand-computed gradient and the numerical derivative, and how many forward passes the two methods take. ### Backpropagation Source: https://academia.sh/en/courses/deep-learning/backpropagation Applying the chain rule by hand over the network and verifying the resulting gradient with the numerical derivative: across all 113 parameters the largest absolute difference is 2.193e-10 at initialization and 6.441e-11 after a hundred epochs of training. When the step size is chosen as 0.001, the difference in the trained network jumps to 2.518e-01; this is not a calculation error, it is the kink of the rectified unit. The cost: backpropagation produces all 113 gradients with one forward and one backward pass per example, the numerical derivative asks for 4,520 passes for the same job, and the ratio grows with the parameter count: 58, 226, 898. The triple does not change — baseline 0.7579, network number 0.7857, second run 0.7698. In four lessons the gradient always arrived ready-made. How the number at the output layer is found was written in `04`, but how that number is distributed over the weights of the hidden layer was never opened. This lesson writes that distribution and tests what it wrote with a separate method. The rule of the distribution is the chain rule. The effect of a weight on the loss is the product of two parts: the value **coming back** to the unit it feeds and the **input** it multiplies. The value coming back is also a product — the sum of what comes from the next layer weighted by the weights, times the derivative of the activation. When a single rule is applied layer by layer, the gradients of all parameters come out. - **NN29.** The set, the split, the seed, the twelve columns and the training set **reduced to 300 rows** are as in `01`–`04`; the network is 12–8–1, 113 parameters. The baseline on the test set is 0.7579. - **NN30.** The loss is **cross-entropy**. The value arriving at the output layer is $a-y$, because the derivative of the sigmoid cancels against the derivative of the loss; this was shown in `04`. - **NN31.** The activation of the hidden layer is the rectified linear unit. Its derivative is 1 when the input is positive and 0 otherwise; it is undefined at zero and 0 is chosen in the code. - **NN32.** The gradients are **summed** over the examples. Because the numerical derivative is computed over the mean loss, its result is multiplied by the number of examples and the two methods come to the same scale. - **NN33.** The numerical derivative is the **central difference**: the parameter is nudged once up and once down and the difference is divided by two steps. It is more accurate than the one-sided difference and asks for twice the passes. - **NN34.** The verification is made on a **subset of 20 rows**; the reason is the number of passes. - **NN35.** The number of passes is in terms of **forward passes**. The backward pass of backpropagation is of the same order as a forward pass and is not counted separately; the ratio is for this reason a lower bound. #### The Chain Rule over the Network Backpropagation consists of two runs. In the forward pass the intermediate values are **stored**, because the weight gradient needs them. In the backward pass one starts from the output and does two jobs at every layer: the gradients of that layer are written, then the value coming back is carried to the previous layer. ```python # backpropagation.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split; # the training set is reduced to 300 rows as in 01. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def make_set(data): return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in data], [float(x["suspicious"]) for x in data]) X_TR, Y_TR = make_set(TRAIN[:300]) # REDUCED training set X_VA, Y_VA = make_set(VAL) X_TE, Y_TE = make_set(TEST) def squash(z): # S-shaped function, overflow-protected if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def rectified(z): return z if z > 0.0 else 0.0 def init_weights(seed, sizes): # gain-scaled initialization, bias zero r, net = rng(seed), [] for a, b in zip(sizes, sizes[1:]): scale = math.sqrt(2.0 / a) net.append(([[(r() * 2 - 1) * scale for _ in range(a)] for _ in range(b)], [0.0] * b)) return net def matmul(W, bias, a): # MATRIX MULTIPLICATION by hand: each row is a unit return [sum(w * v for w, v in zip(row, a)) + b for row, b in zip(W, bias)] def forward(net, x): # FORWARD PASS: layer by layer a = x for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(t) for t in z] return a def loss(a, y): # cross-entropy return -(math.log(max(a, 1e-12)) if y else math.log(max(1 - a, 1e-12))) def backpropagation(net, X, Y, trace=False): # MODEL: chain rule layer by layer gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net] gb = [[0.0] * len(bias) for _, bias in net] total = 0.0 for i in range(len(X)): a, acts = X[i], [X[i]] # FORWARD PASS: intermediate values are stored for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(q) for q in z] acts.append(a) total += loss(acts[-1][0], Y[i]) d = [acts[-1][0] - Y[i]] # output layer: loss and sigmoid cancel for k in range(len(net) - 1, -1, -1): W, a = net[k][0], acts[k] # a: the INPUT of this layer if trace: print(f" layer {k + 1}: incoming width {len(d)}, first two " f"{[round(v, 5) for v in d[:2]]}") for j, dj in enumerate(d): # weight gradient = incoming x input s = gW[k][j] for p in range(len(a)): s[p] += dj * a[p] gb[k][j] += dj # bias gradient = incoming if k: # carry to the previous layer, with the activation derivative prev = [0.0] * len(a) for j, dj in enumerate(d): s = W[j] for p in range(len(a)): prev[p] += dj * s[p] d = [prev[p] if a[p] > 0 else 0.0 for p in range(len(a))] return gW, gb, total / len(X) net = init_weights(SEED + 1, [12, 8, 1]) print(f"single subscriber, true label {int(Y_TR[0])}, network output {forward(net, X_TR[0])[0]:.5f}") gW, gb, k0 = backpropagation(net, X_TR[:1], Y_TR[:1], trace=True) print(f" loss {k0:.5f}") print(f" output layer weight gradient, first three " f"{[round(v, 5) for v in gW[1][0][:3]]}") print(f" hidden layer unit 1 weight gradient, first three " f"{[round(v, 5) for v in gW[0][0][:3]]}") print(f" hidden layer bias gradients, first three {[round(v, 5) for v in gb[0][:3]]}") ``` ``` single subscriber, true label 0, network output 0.49369 layer 2: incoming width 1, first two [0.49369] layer 1: incoming width 8, first two [0.0, 0.00155] loss 0.68061 output layer weight gradient, first three [0.0, 0.15433, 0.18243] hidden layer unit 1 weight gradient, first three [0.0, 0.0, 0.0] hidden layer bias gradients, first three [0.0, 0.00155, -0.03503] ``` The value arriving at the output layer is 0.49369, that is, the output of the network minus the true label. Eight values go down to the hidden layer and the first is exactly 0.0. The reason had been measured in `02`: that unit is not active for this subscriber, the rectified function has cut its input, its derivative is 0 and the chain breaks there. The result shows in the last two lines — the twelve weight gradients of the first unit are zero and so is its bias gradient. This subscriber cannot teach that unit anything. #### Verification with the Numerical Derivative A derivative written by hand can be silently wrong: the code runs, the loss falls, numbers are produced, and the error shows itself only as performance staying a little low. The independent test is to measure the loss twice with a small nudge of the parameter and take the difference. It is slow, but because it knows nothing about the structure of the network, it does not make the same error as the chain rule. ```python def mean_loss(net, X, Y): return sum(loss(forward(net, x)[0], y) for x, y in zip(X, Y)) / len(X) def numerical_derivative(net, X, Y, eps): # central difference, two passes per parameter gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net] gb = [[0.0] * len(bias) for _, bias in net] passes = 0 for k, (W, bias) in enumerate(net): for j in range(len(W)): for p in range(len(W[j]) + 1): target, index = (W[j], p) if p < len(W[j]) else (bias, j) old = target[index] target[index] = old + eps plus = mean_loss(net, X, Y) target[index] = old - eps minus = mean_loss(net, X, Y) target[index] = old value = (plus - minus) / (2 * eps) * len(X) passes += 2 * len(X) if p < len(W[j]): gW[k][j][p] = value else: gb[k][j] = value return gW, gb, passes def check(net, X, Y, eps): # hand-written gradient against the numerical derivative aW, ab, _ = backpropagation(net, X, Y) nW, nb, passes = numerical_derivative(net, X, Y, eps) diffs = [] for k in range(len(net)): for j in range(len(aW[k])): for p in range(len(aW[k][j])): diffs.append((abs(aW[k][j][p] - nW[k][j][p]), abs(aW[k][j][p]))) diffs.append((abs(ab[k][j] - nb[k][j]), abs(ab[k][j]))) return len(diffs), max(a for a, _ in diffs), max(a / max(b, 1e-12) for a, b in diffs), passes def train_model(net, X, Y, rate, epochs): # MODEL: full-batch gradient descent n = len(X) for _ in range(epochs): gW, gb, _ = backpropagation(net, X, Y) for k, (W, bias) in enumerate(net): for j in range(len(W)): s, g = W[j], gW[k][j] for p in range(len(s)): s[p] -= rate * g[p] / n bias[j] -= rate * gb[k][j] / n return net SUB_X, SUB_Y = X_TR[:20], Y_TR[:20] # verification subset, 20 rows TRAINED = train_model(init_weights(SEED + 1, [12, 8, 1]), X_TR, Y_TR, 0.3, 100) print(f"{'network':<24} {'eps':>7} {'params':>10} {'max absolute':>16} " f"{'max relative':>15}") for label, net in (("at initialization", init_weights(SEED + 1, [12, 8, 1])), ("after 100 epochs", TRAINED)): for eps in (1e-3, 1e-5, 1e-7): count, max_abs, max_rel, passes = check(net, SUB_X, SUB_Y, eps) print(f"{label:<24} {eps:>7g} {count:>10} {max_abs:>16.3e} {max_rel:>15.3e}") ``` ``` network eps params max absolute max relative at initialization 0.001 113 2.199e-07 1.583e-07 at initialization 1e-05 113 2.193e-10 4.567e-08 at initialization 1e-07 113 2.591e-08 4.932e-06 after 100 epochs 0.001 113 2.518e-01 2.881e+00 after 100 epochs 1e-05 113 6.441e-11 1.227e-09 after 100 epochs 1e-07 113 5.694e-09 2.697e-07 ``` The first three rows show the error of the numerical derivative itself. When the step is large the truncation error dominates, when it is small the rounding error; the best agreement is in the middle, at 1e-05, with 2.193e-10. That is, across all 113 parameters the two methods give the same number up to about ten significant digits. This is the proof that the hand-written chain rule is correct. The fourth row is a trap and must not be passed over without measuring. In the trained network, when the step is chosen as 0.001, the largest absolute difference jumps to 2.518e-01 and the relative difference to 2.881. This is not a calculation error; the same network at the same step gives 6.441e-11 at 1e-05. The cause is the **kink** of the rectified unit: a nudge of 0.001 is enough to carry the weighted sum from one side of zero to the other, and the two ends of the central difference fall on two different linear pieces of the function. When a verification fails, the first place to look is not the code but the step size. #### Cost: How Many Passes If the two methods produce the same number, which one to use depends on the cost. Backpropagation produces the gradients of **all** parameters in one forward and one backward pass. The numerical derivative asks for two separate passes for each parameter. ```python print(f"{'units':>6} {'params':>10} {'backpropagation':>15} {'numerical derivative':>20} " f"{'ratio':>7}") for h in (2, 8, 32): sizes = [12, h, 1] P = sum(a * b + b for a, b in zip(sizes, sizes[1:])) print(f"{h:>6} {P:>10} {len(SUB_X):>15} {2 * P * len(SUB_X):>20} {2 * P:>7}") print(f"\n{'run':>6} {'train_loss':>10} {'validation':>10} {'test':>8}") TEST_ACCS = [] for run in (1, 2): net = train_model(init_weights(SEED + run, [12, 8, 1]), X_TR, Y_TR, 0.3, 100) _, _, loss_value = backpropagation(net, X_TR, Y_TR) val_acc = sum((forward(net, x)[0] > 0.5) == (y > 0.5) for x, y in zip(X_VA, Y_VA)) / len(X_VA) test_acc = sum((forward(net, x)[0] > 0.5) == (y > 0.5) for x, y in zip(X_TE, Y_TE)) / len(X_TE) TEST_ACCS.append(test_acc) print(f"{run:>6} {loss_value:>10.4f} {val_acc:>10.4f} {test_acc:>8.4f}") print(f"baseline {BASELINE:.4f}, network number {TEST_ACCS[0]:.4f}, " f"second run {TEST_ACCS[1]:.4f}, difference " f"{round(abs(TEST_ACCS[0] - TEST_ACCS[1]) * len(X_TE))} subscribers") ``` ``` units params backpropagation numerical derivative ratio 2 29 20 1160 58 8 113 20 4520 226 32 449 20 17960 898 run train_loss validation test 1 0.3597 0.8214 0.7857 2 0.3646 0.8056 0.7698 baseline 0.7579, network number 0.7857, second run 0.7698, difference 4 subscribers ``` The backpropagation column is 20 in all three rows — as many as the number of examples. The numerical derivative column grows together with the parameter count: 1,160, 4,520, 17,960. The ratio is twice the parameter count and rises to 898 in the network with 32 units. The cost of backpropagation is independent of the parameter count; the work is done by the intermediate values stored in the forward pass. For this reason the numerical derivative is not a training method but a tool of **inspection**. The last row confirms the number of the four lessons. In this section the training was carried out with `backpropagation` and the triple is the same output as in `01`: 0.7579 — 0.7857 — 0.7698. The reason is plain; the code running since `01` was already backpropagation, only it had not been named. #### Summary - The gradient of a weight is the product of the value coming back to the unit it feeds and the input that weight multiplies; the value coming back is carried from the next layer and multiplied by the derivative of the activation. - All gradients of an inactive unit are zero: in the example subscriber, the twelve weight gradients and the bias gradient of the first unit of the hidden layer all come out as 0.0. - Verification with the central difference agrees in all 113 parameters: the largest absolute difference is 2.193e-10 at initialization and 6.441e-11 after a hundred epochs of training. When the step is chosen too large or too small, the error of the numerical derivative itself grows. - In the trained network the difference jumps to 2.518e-01 at step 0.001; this is the kink of the rectified unit, not a calculation error. The same network gives 6.441e-11 at 1e-05. - Backpropagation produces all 113 gradients with one forward and one backward pass per example; the numerical derivative asks for 4,520 passes for the same job and the ratio grows with the parameter count: 58, 226, 898. #### Next Step The gradient is now computed correctly and cheaply. But what to do with it was never asked in five lessons: in every epoch the gradient was multiplied by the same constant number and subtracted from the weights, always of the same size, without ever looking at what the previous epochs had done. Using the same gradient in another way — remembering the direction of the previous steps or giving each parameter its own step size — what loss it reaches in the same epoch budget and how much it moves between two runs was not measured. The next lesson treats the gradient not as a result but as an input. ### Optimization Algorithms Source: https://academia.sh/en/courses/deep-learning/optimization-algorithms Using the same gradient with three different step rules and measuring that a falling loss does not mean a falling error: in the shared 40-epoch budget momentum descent brings the training loss down to 0.2777 and the adaptive step to 0.2923, plain descent stays at 0.3296; the test accuracies, on the other hand, are 0.7500, 0.7222 and 0.7698. In the three settings read from validation the network number is 0.7857, the second run of the same network is 0.7698 and the baseline is 0.7579. When four runs are made changing only the initialization, the test spread of a single optimizer rises to 0.0317, that is, eight subscribers; the difference between optimizers, on the other hand, moves between 0.0516 and 0.0079 depending on the run, and none of the twelve runs passes the 0.8016 of the linear model. The previous lesson showed that the gradient is computed correctly: the number given by backpropagation was compared with the numerical derivative and the difference was printed. But the gradient gives a **direction**, not a **step**. In five lessons it was never asked by which rule the weights are moved in that direction; because it was never asked, there seems to be a single rule: subtract a constant multiple of the gradient from the weight. This lesson fills that gap. With the same network, the same data, the same gradient and the same epoch budget, **three different step rules** are run. The measure is twofold: the job of the optimizer is to lower the training loss, the job of the model is to be right on the held-out set. The finding of this lesson is that the two **can go in opposite directions**. - **NN46.** The set, the split and the seed are as in the previous lessons; the table is **fictional**. So that the run time stays within the lesson, the training set has been **reduced from 756 rows to 300 rows**; the validation and test sets did not change (252 and 252) and the baseline on the test set is **0.7579**. - **NN47.** The network is fixed: 12 inputs, 8 hidden units, 1 output. The only thing that changes is the step rule. There is the rectified linear unit in the hidden layer, a sigmoid at the output and cross-entropy as the loss; the initialization is **gain-scaled** and the biases start from zero. - **NN48.** The epoch budget is 40 and the batch size is 20; all three spend the same number of updates. - **NN49.** The three rules are: **plain descent** (a constant multiple of the gradient is subtracted from the weight), **momentum descent** (gradients accumulate with a decay factor of 0.9) and the **adaptive step** (a squared-gradient accumulation is kept for each weight and the step is divided by the root of that accumulation). - **NN50.** The batch split order is the same in every run. The only thing that changes from run to run is the **initialization**. - **NN51.** The number of candidates tried is **9**: three optimizers times three learning rates, with the choice from the validation set. - **NN52.** The comparison lines come from the Supervised Learning course and are not retrained: the linear model **0.8016**, the decision tree **0.7857**. Both had been measured on the same test set. - **NN53.** The measure of speed is not time but the **number of epochs**: in which epoch the loss first falls below the 0.40 threshold. The threshold is arbitrary and the same for the three optimizers. #### From Gradient to Step The setup produces the fictional table, separates the sets and defines the network. The last coefficient of every weight row is the **bias**. ```python # optimization.py — MODEL: the same FICTIONAL subscriber table, the same seed, the same split; # the training set is reduced to 300 rows in this topic. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue sub_id = 10001 + i r, v = rng(SEED + 7000 + sub_id), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + sub_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (sub_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for q in REGION: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) r, S = rng(SEED + 90000), list(range(len(DATA))) for i in range(len(S) - 1, 0, -1): j = int(r() * (i + 1)) S[i], S[j] = S[j], S[i] K = [DATA[i] for i in S] TRAIN, VAL, TEST = K[:300], K[756:1008], K[1008:] # training REDUCED from 756 to 300 FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} def scale(s): # LEARNING from training, APPLICATION to every set return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in s], [float(x["suspicious"]) for x in s]) X_TR, Y_TR = scale(TRAIN) X_VA, Y_VA = scale(VAL) X_TE, Y_TE = scale(TEST) BASELINE = sum(y == 0.0 for y in Y_TE) / len(Y_TE) TREE, LINEAR = 198 / len(TEST), 202 / len(TEST) # the two numbers of the Supervised Learning course def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def init_weights(seed, shape, gain=1.0): # gain-scaled; the last column of every row is the BIAS r, layers = rng(seed), [] for fan_in, fan_out in shape: limit = math.sqrt(6.0 * gain / fan_in) layers.append([[(r() * 2 - 1) * limit for _ in range(fan_in)] + [0.0] for _ in range(fan_out)]) return layers def zeros(layers): return [[[0.0] * len(w) for w in W] for W in layers] def forward(layers, x): # rectified linear unit in the hidden layers, a, trace = x, [] # sigmoid at the output for i, W in enumerate(layers): trace.append(a) z = [sum(p * q for p, q in zip(w, a)) + w[-1] for w in W] a = [squash(z[0])] if i == len(layers) - 1 else [t if t > 0 else 0.0 for t in z] trace.append(a) return trace def gradient(layers, X, Y, idx): # BACKPROPAGATION, cross-entropy loss G, loss = zeros(layers), 0.0 for i in idx: trace = forward(layers, X[i]) p = min(max(trace[-1][0], 1e-12), 1 - 1e-12) loss -= Y[i] * math.log(p) + (1 - Y[i]) * math.log(1 - p) d = [p - Y[i]] for L in range(len(layers) - 1, -1, -1): W, a = layers[L], trace[L] for k, dk in enumerate(d): gk = G[L][k] for j, aj in enumerate(a): gk[j] += dk * aj gk[-1] += dk if L: d = [sum(d[k] * W[k][j] for k in range(len(W))) * (a[j] > 0) for j in range(len(a))] for W in G: for w in W: w[:] = [v / len(idx) for v in w] return G, loss / len(idx) def accuracy(layers, X, Y): return sum((forward(layers, x)[-1][0] > 0.5) == (y > 0.5) for x, y in zip(X, Y)) / len(X) def order(epoch): # batch split order is the SAME in every run r, s = rng(SEED + 500 + epoch), list(range(len(X_TR))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s LAYERS0 = init_weights(SEED + 1, [(len(FIELDS), 8), (8, 1)]) print(f"train {len(TRAIN)} (reduced from 756), validation {len(VAL)}, " f"test {len(TEST)}, columns {len(FIELDS)}") print(f"network {len(FIELDS)}-8-1, parameters {sum(len(w) for W in LAYERS0 for w in W)}, " f"epochs 40, batch 20, updates {40 * len(X_TR) // 20}") print(f"baseline {BASELINE:.4f}; comparison lines linear model {LINEAR:.4f}, " f"decision tree {TREE:.4f}; initialized network {accuracy(LAYERS0, X_TE, Y_TE):.4f}") ``` ``` train 300 (reduced from 756), validation 252, test 252, columns 12 network 12-8-1, parameters 113, epochs 40, batch 20, updates 600 baseline 0.7579; comparison lines linear model 0.8016, decision tree 0.7857; initialized network 0.3333 ``` The last number gives the starting point: the untrained network scores 0.3333 on the test set, far below the baseline. All 113 parameters are still noise, and the job of the 600 updates is to carry this number upward. #### Three Optimizers in the Same Epoch Budget The three rules sit in a single function. Plain descent uses the gradient directly. Momentum descent accumulates past gradients; if the direction is consistent the step grows, if the direction oscillates the steps cancel each other. The adaptive step keeps a separate squared-gradient accumulation for each weight and divides the step of that weight by the root of the accumulation; a weight that keeps seeing large gradients moves with a small step, a weight that sees rare gradients with a large one. ```python def apply_step(layers, G, M, V, method, rate): # a single update step for W, gW, mW, vW in zip(layers, G, M, V): for w, g, m, v in zip(W, gW, mW, vW): for j in range(len(w)): if method == "momentum": # accumulated direction m[j] = 0.9 * m[j] + g[j] w[j] -= rate * m[j] elif method == "adaptive": # squared-gradient accumulation v[j] = 0.9 * v[j] + 0.1 * g[j] * g[j] w[j] -= rate * g[j] / (math.sqrt(v[j]) + 1e-8) else: # plain descent w[j] -= rate * g[j] def train_model(method, rate, seed, epochs=40, batch=20): layers = init_weights(seed, [(len(FIELDS), 8), (8, 1)]) M, V, curve = zeros(layers), zeros(layers), [] for t in range(epochs): s, total = order(t), 0.0 for b in range(0, len(s), batch): G, k = gradient(layers, X_TR, Y_TR, s[b:b + batch]) apply_step(layers, G, M, V, method, rate) total += k * batch curve.append(total / len(s)) return layers, curve def epochs_to_target(curve, threshold=0.40): # first epoch the loss falls below the threshold for t, k in enumerate(curve): if k < threshold: return str(t + 1) return "never" METHODS = ("plain", "momentum", "adaptive") RESULTS = {} print(f"{'optimizer':>12} {'rate':>6} {'train_loss':>13} {'epochs_to_target':>16} " f"{'validation':>10} {'test':>8}") for m in METHODS: for rate in (0.02, 0.05, 0.15): layers, curve = train_model(m, rate, SEED + 1) RESULTS[(m, rate)] = (curve[-1], accuracy(layers, X_VA, Y_VA), accuracy(layers, X_TE, Y_TE)) print(f"{m:>12} {rate:>6} {curve[-1]:>13.4f} {epochs_to_target(curve):>16} " f"{RESULTS[(m, rate)][1]:>10.4f} {RESULTS[(m, rate)][2]:>8.4f}") CHOSEN = {m: max((0.02, 0.05, 0.15), key=lambda a: RESULTS[(m, a)][1]) for m in METHODS} lowest = min(RESULTS.values(), key=lambda t: t[0]) highest = max(RESULTS.values(), key=lambda t: t[2]) print(f"\ncandidates tried {len(RESULTS)}; rates read from validation " + ", ".join(f"{m} {CHOSEN[m]}" for m in METHODS) + f"\nlowest loss {lowest[0]:.4f} (test {lowest[2]:.4f}), " + f"highest test {highest[2]:.4f} (loss {highest[0]:.4f})") ``` ``` optimizer rate train_loss epochs_to_target validation test plain 0.02 0.4042 never 0.8135 0.7857 plain 0.05 0.3664 19 0.8056 0.7897 plain 0.15 0.3296 8 0.7817 0.7698 momentum 0.02 0.3186 6 0.7976 0.7579 momentum 0.05 0.2811 4 0.7817 0.7540 momentum 0.15 0.2777 3 0.7817 0.7500 adaptive 0.02 0.2948 4 0.7659 0.7341 adaptive 0.05 0.2923 3 0.7500 0.7222 adaptive 0.15 0.3038 8 0.7341 0.7460 candidates tried 9; rates read from validation plain 0.02, momentum 0.02, adaptive 0.02 lowest loss 0.2777 (test 0.7500), highest test 0.7897 (loss 0.3664) ``` The loss column shows the optimizers doing their job. In the same 600 updates momentum descent goes down to 0.2777 and the adaptive step to 0.2923; the best of plain descent stays at 0.3296. The difference is sharper in epochs: momentum descent crosses the 0.40 threshold in the **third** epoch, plain descent at the smallest rate does not cross it at all in forty epochs. The test column does not give the counterpart. The setting that produces the lowest loss gives **0.7500** on the test set; the highest test number, **0.7897**, is produced by plain descent, which brings the loss down only to 0.3664. The two columns go in opposite directions, because with 113 parameters on 300 rows there is a way to pull the loss lower: memorizing the training rows. Momentum descent and the adaptive step get there faster. **The optimizer lowers the loss, it does not lower the generalization error.** For this reason the choice is read from the validation set, and in all three the smallest rate, 0.02, wins. #### Run Variability Swallows the Method Difference Each row of the table above is **a single run**. The block below runs the three optimizers four times each with the rates read from validation; the data, the order, the budget and the rate are fixed, only the initialization changes. ```python print(f"{'optimizer':>12} {'loss 4 runs':>32} {'test 4 runs':>32}") SPREAD = {} for m in METHODS: runs = [train_model(m, CHOSEN[m], SEED + t) for t in (1, 2, 3, 4)] SPREAD[m] = ([curve[-1] for _, curve in runs], [accuracy(l, X_TE, Y_TE) for l, _ in runs]) print(f"{m:>12} " + " ".join(f"{v:>7.4f}" for v in SPREAD[m][0]) + " " + " ".join(f"{v:>7.4f}" for v in SPREAD[m][1])) for m in METHODS: s = SPREAD[m][1] print(f"{m:>12} test spread {max(s) - min(s):.4f} " f"({round((max(s) - min(s)) * len(TEST))} subscribers)") best = max(METHODS, key=lambda m: SPREAD[m][1][0]) print(f"\nbaseline {BASELINE:.4f}, network number {SPREAD[best][1][0]:.4f} ({best}), " f"second run of the same network {SPREAD[best][1][1]:.4f}") print("difference between the best and the worst optimizer per run: " + ", ".join(f"{max(SPREAD[m][1][t] for m in METHODS) - min(SPREAD[m][1][t] for m in METHODS):.4f}" for t in range(4))) print(f"runs beating the decision tree " f"{sum(v > TREE + 1e-9 for _, s in SPREAD.values() for v in s)}/12, the linear model " f"{sum(v > LINEAR + 1e-9 for _, s in SPREAD.values() for v in s)}/12") ``` ``` optimizer loss 4 runs test 4 runs plain 0.4042 0.3874 0.3980 0.3972 0.7857 0.7698 0.7897 0.7817 momentum 0.3186 0.3373 0.3244 0.3177 0.7579 0.7659 0.7817 0.7738 adaptive 0.2948 0.3064 0.2922 0.2976 0.7341 0.7619 0.7619 0.7659 plain test spread 0.0198 (5 subscribers) momentum test spread 0.0238 (6 subscribers) adaptive test spread 0.0317 (8 subscribers) baseline 0.7579, network number 0.7857 (plain), second run of the same network 0.7698 difference between the best and the worst optimizer per run: 0.0516, 0.0079, 0.0278, 0.0159 runs beating the decision tree 1/12, the linear model 0/12 ``` The loss columns are narrow: each optimizer stays in its own band across four runs and the three bands do not touch each other. The effect of the optimizer on the loss is **stable** at this scale. The test columns are not narrow. The adaptive step wanders between 0.7341 and 0.7659; this is 0.0317, that is, eight subscribers. The line before the last is the real finding of the lesson: the difference between the optimizers is 0.0516 in the first run and **0.0079** in the second. The same three methods, the same settings, the same data — only the initialization changed and the distance between them narrowed by more than a factor of six. To look at the first run and say "plain descent beats the adaptive step by 0.0516" is not a measured result. The triple is as follows: baseline 0.7579, network number 0.7857, the second run of the same network 0.7698; the difference from the baseline is 0.0278, that is, seven subscribers. The last line gives the second claim of the course: **only one** of the twelve runs passes the 0.7857 of the decision tree and **none** passes the 0.8016 of the linear model. On this tabular data the network does not pass two simple families, and this is not hidden. #### Summary - Backpropagation gives a direction; the step rule is a separate decision, and three rules are compared in the same budget of 600 updates. - Momentum descent brings the training loss down to 0.2777 and the adaptive step to 0.2923; plain descent stays at 0.3296 and cannot reach the 0.40 threshold at all at the smallest rate. - The test column turns around: the lowest loss gives 0.7500, the highest test number gives 0.7897. The optimizer lowers the loss, it does not lower the generalization error. - In four runs that change only the initialization, the test spread of a single optimizer is 0.0317, that is, eight subscribers; the difference between optimizers, on the other hand, moves between 0.0516 and 0.0079 depending on the run. - The triple: baseline 0.7579, network number 0.7857, second run 0.7698. One of the twelve runs passes the decision tree, none passes the linear model. #### Next Step The network of this lesson had two layers and the gradient was chained only once as it came back. When the number of layers grows, the question does not stay clean: the gradient is filtered by a factor at every layer on its way from the output to the input, and the product of those factors either goes down to zero or overflows. The next lesson ties this to a number — as the network deepens, how many times smaller the gradient magnitude seen by the first layer becomes than that of the last layer, at which initialization gain it turns around and explodes, and which of these two extremes gradient clipping and the learning rate schedule correct, and by how much. ### Training Stability Source: https://academia.sh/en/courses/deep-learning/training-stability Measuring the vanishing and exploding of the gradient as it passes through the layers: with saturating activation the gradient magnitude of the first layer goes from 1.274e-02 with one hidden layer down to 5.539e-05 with six layers, that is, 230 times lower, and the ratio to the last layer rises from 18.8 to 3,977.2; with the rectified linear unit the ratio at the same depth stays at 0.7022, while when the initialization gain is raised to 16 it grows 134.99 times. With a learning rate of 10 the gradient magnitude of the network with three hidden layers shoots up to 2.800e+07, 15 of the 24 units die and both runs give exactly the baseline, 0.7579. Gradient clipping clips 400 of the 450 updates and saves the training, but the test falls to 0.6984, below the baseline; clipping and the schedule together bring the loss down to 0.2722 and raise the run difference to 0.0516, that is, thirteen subscribers. The previous lesson compared three step rules in a network with two layers. There the gradient was chained once on its way back, so every layer saw a gradient of a similar magnitude. When the number of layers grows this stops being true: while the error at the output moves toward the input it is filtered by a factor at every layer, and the **product** of those factors decides what is left for the first layer. If the factors are smaller than one the product goes down to zero, if they are larger than one it overflows. This lesson measures that product. The two extremes are counted separately — the **vanishing gradient** and the **exploding gradient** — and then two corrections are tried: **gradient clipping** and the **learning rate schedule**. The measure is again twofold: rescuing the training and improving the held-out set number are not the same thing. - **NN54.** The set, the split, the seed and the scale are the same as in the previous lesson; the table is **fictional**, the training set has been **reduced to 300 rows** and the baseline on the test set is **0.7579**. - **NN55.** Every hidden layer has 8 units: one layer comes to 113 parameters, two layers to 185, four layers to 329, six layers to 473. - **NN56.** The gradient magnitude is the **root of the mean of the squares** of the gradient entries in a layer; because layers carry different numbers of weights, the mean is taken, not the sum. - **NN57.** The measurement is made **right after initialization**, on a single batch of 60 rows; the question is not the result of the training but how the gradient is distributed across the layers. Two activations stand side by side: the **sigmoid** (saturating) and the **rectified linear unit** (non-saturating). Gain is the factor that sets the width of the initialization range. - **NN58.** In the stability experiment the network has three hidden layers, makes **450 updates** with 30 epochs and a batch size of 20, and uses plain descent. The number of settings tried is **5**; this is not a search. - **NN59.** The **gradient clipping** threshold is 0.10 and is applied once to the gradient magnitude of the whole network: if the magnitude exceeds the threshold the gradient is reduced by the same ratio, and the direction is kept. The **schedule** halves the learning rate every six epochs. - **NN60.** A **dead unit** is a unit that gives an output greater than zero on no training row; across the three hidden layers there are 24 units in total. #### The Gradient on Its Way Through the Layers The setup is the same as in the previous lesson; the only difference is that the forward pass and backpropagation take the **activation as a variable**. The derivative of the sigmoid is computed from the output itself, the derivative of the rectified linear unit is zero or one. ```python # stability.py — MODEL: the same FICTIONAL subscriber table, the same seed, the same split; # the training set is reduced to 300 rows in this topic. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue sub_id = 10001 + i r, v = rng(SEED + 7000 + sub_id), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + sub_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (sub_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for q in REGION: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) r, S = rng(SEED + 90000), list(range(len(DATA))) for i in range(len(S) - 1, 0, -1): j = int(r() * (i + 1)) S[i], S[j] = S[j], S[i] K = [DATA[i] for i in S] TRAIN, VAL, TEST = K[:300], K[756:1008], K[1008:] # training REDUCED from 756 to 300 FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} def scale(s): # LEARNING from training, APPLICATION to every set return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in s], [float(x["suspicious"]) for x in s]) X_TR, Y_TR = scale(TRAIN) X_TE, Y_TE = scale(TEST) BASELINE = sum(y == 0.0 for y in Y_TE) / len(Y_TE) TREE, LINEAR = 198 / len(TEST), 202 / len(TEST) # the two numbers of the Supervised Learning course def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def init_weights(seed, shape, gain=1.0): # gain-scaled; the last column of every row is the BIAS r, layers = rng(seed), [] for fan_in, fan_out in shape: limit = math.sqrt(6.0 * gain / fan_in) layers.append([[(r() * 2 - 1) * limit for _ in range(fan_in)] + [0.0] for _ in range(fan_out)]) return layers def zeros(layers): return [[[0.0] * len(w) for w in W] for W in layers] def forward(layers, x, activation="rectified"): a, trace = x, [] for i, W in enumerate(layers): trace.append(a) z = [sum(p * q for p, q in zip(w, a)) + w[-1] for w in W] if i == len(layers) - 1: a = [squash(z[0])] elif activation == "sigmoid": a = [squash(t) for t in z] else: a = [t if t > 0 else 0.0 for t in z] trace.append(a) return trace def gradient(layers, X, Y, idx, activation="rectified"): # BACKPROPAGATION G, loss = zeros(layers), 0.0 slope = (lambda v: v * (1 - v)) if activation == "sigmoid" else (lambda v: v > 0) for i in idx: trace = forward(layers, X[i], activation) p = min(max(trace[-1][0], 1e-12), 1 - 1e-12) loss -= Y[i] * math.log(p) + (1 - Y[i]) * math.log(1 - p) d = [p - Y[i]] for L in range(len(layers) - 1, -1, -1): W, a = layers[L], trace[L] for k, dk in enumerate(d): gk = G[L][k] for j, aj in enumerate(a): gk[j] += dk * aj gk[-1] += dk if L: d = [sum(d[k] * W[k][j] for k in range(len(W))) * slope(a[j]) for j in range(len(a))] for W in G: for w in W: w[:] = [v / len(idx) for v in w] return G, loss / len(idx) def order(epoch): # batch split order is the SAME in every run r, s = rng(SEED + 500 + epoch), list(range(len(X_TR))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s def layer_shape(depth, hidden=8): # depth hidden layers, then a single output return [(len(FIELDS), hidden)] + [(hidden, hidden)] * (depth - 1) + [(hidden, 1)] print(f"train {len(TRAIN)} (reduced from 756), test {len(TEST)}, columns {len(FIELDS)}") print("hidden layers and parameters: " + ", ".join( f"{d} -> {sum(len(w) for W in init_weights(SEED, layer_shape(d)) for w in W)}" for d in (1, 2, 3, 4, 6))) print(f"baseline {BASELINE:.4f}; comparison lines linear model {LINEAR:.4f}, " f"decision tree {TREE:.4f}") ``` ``` train 300 (reduced from 756), test 252, columns 12 hidden layers and parameters: 1 -> 113, 2 -> 185, 3 -> 257, 4 -> 329, 6 -> 473 baseline 0.7579; comparison lines linear model 0.8016, decision tree 0.7857 ``` #### The Numbers of Vanishing and Exploding Gradients At every depth the network is initialized, the gradient is computed on a single batch and the gradient magnitude of the **first layer** is printed next to that of the **last layer**. The last column gives how many times the first layer falls relative to the single-layer network of the same family. ```python def magnitude(W): # root of the mean of squares return math.sqrt(sum(v * v for w in W for v in w) / sum(len(w) for w in W)) print(f"{'activation':>14} {'gain':>7} {'depth':>9} {'first_layer':>12} " f"{'last_layer':>12} {'last/first':>12} {'vs_first':>10}") FIRST = {} for activation, gain in (("sigmoid", 1.0), ("rectified", 2.0), ("rectified", 16.0)): for depth in (1, 2, 4, 6): layers = init_weights(SEED + 1, layer_shape(depth), gain) G, _ = gradient(layers, X_TR, Y_TR, range(60), activation) b = [magnitude(W) for W in G] FIRST.setdefault((activation, gain), b[0]) print(f"{activation:>14} {gain:>7} {depth:>9} {b[0]:>12.3e} {b[-1]:>12.3e} " f"{b[-1] / b[0]:>12.1f} {b[0] / FIRST[(activation, gain)]:>10.4f}") ``` ``` activation gain depth first_layer last_layer last/first vs_first sigmoid 1.0 1 1.274e-02 2.393e-01 18.8 1.0000 sigmoid 1.0 2 2.820e-03 4.822e-02 17.1 0.2213 sigmoid 1.0 4 1.930e-04 1.318e-01 682.6 0.0151 sigmoid 1.0 6 5.539e-05 2.203e-01 3977.2 0.0043 rectified 2.0 1 9.817e-02 5.694e-01 5.8 1.0000 rectified 2.0 2 5.689e-02 2.176e-01 3.8 0.5796 rectified 2.0 4 2.513e-02 5.219e-01 20.8 0.2560 rectified 2.0 6 6.893e-02 3.273e-01 4.7 0.7022 rectified 16.0 1 3.065e-01 1.707e+00 5.6 1.0000 rectified 16.0 2 5.330e-01 2.059e+00 3.9 1.7387 rectified 16.0 4 1.769e+00 3.761e+01 21.3 5.7719 rectified 16.0 6 4.138e+01 1.813e+02 4.4 134.9930 ``` The first four rows are the number of the vanishing gradient. With saturating activation the gradient magnitude seen by the first layer is 1.274e-02 with one layer and 5.539e-05 with six layers — by the last column **0.0043**, that is, **230 times** lower; the ratio to the last layer rises from 18.8 to 3,977.2. The last layer of the six-layer network sees a gradient of the same magnitude as that of the single-layer network; what disappears is only **the share of the layers near the input**. If this network is trained, the greater part of the 473 parameters stays where it was initialized. In the middle four rows the same depths are measured with non-saturating activation and at six layers the ratio is **0.7022**; there is almost no vanishing. What makes it vanish is not depth but **the magnitude of the factor**. The last four rows are the opposite extreme: when the gain is raised to 16 the factors become larger than one and the product runs upward. At six layers the gradient magnitude of the first layer is **134.99 times** that of the single-layer one. A vanishing gradient freezes the weights, an exploding gradient throws them away in a single update. #### What Clipping and the Schedule Correct The explosion does not come only from initialization; a large learning rate produces the same result. The block below trains a network with three hidden layers first with a safe step and then with a step two hundred times larger, and tries the two corrections one by one and together. ```python def norm(G): # gradient magnitude of the whole network return math.sqrt(sum(v * v for W in G for w in W for v in w)) def dead_units(layers): # a unit that never activates on any training row alive = [[False] * len(W) for W in layers[:-1]] for x in X_TR: trace = forward(layers, x) for L in range(len(layers) - 1): for j, v in enumerate(trace[L + 1]): if v > 0: alive[L][j] = True return sum(not c for W in alive for c in W) def train_model(rate, seed, clip=0.0, schedule=False, depth=3, epochs=30, batch=20): layers, clipped, peak, curve = init_weights(seed, layer_shape(depth), 2.0), 0, 0.0, [] for t in range(epochs): a = rate * 0.5 ** (t // 6) if schedule else rate s, total = order(t), 0.0 for b in range(0, len(s), batch): G, k = gradient(layers, X_TR, Y_TR, s[b:b + batch]) n = norm(G) peak = max(peak, n) if clip and n > clip: # direction is kept, magnitude is pulled to the threshold clipped += 1 for W in G: for w in W: w[:] = [v * clip / n for v in w] for W, gW in zip(layers, G): for w, g in zip(W, gW): for j in range(len(w)): w[j] -= a * g[j] total += k * batch curve.append(total / len(s)) return layers, curve[-1], clipped, peak SETTINGS = (("rate 0.05", 0.05, 0.0, False), ("rate 10 bare", 10.0, 0.0, False), ("rate 10 clipping", 10.0, 0.10, False), ("rate 10 schedule", 10.0, 0.0, True), ("rate 10 both", 10.0, 0.10, True)) print(f"{'setting':>16} {'loss 2 runs':>16} {'test 2 runs':>16} {'dead':>7} " f"{'clipped':>9} {'max_gradient':>14}") OUTCOME = {} for name, rate, clip, sched in SETTINGS: runs = [train_model(rate, SEED + s, clip, sched) for s in (1, 2)] OUTCOME[name] = [sum((forward(k, x)[-1][0] > 0.5) == (y > 0.5) for x, y in zip(X_TE, Y_TE)) / len(X_TE) for k, _, _, _ in runs] print(f"{name:>16} {runs[0][1]:>7.4f} {runs[1][1]:>8.4f} {OUTCOME[name][0]:>8.4f} " f"{OUTCOME[name][1]:>7.4f} {dead_units(runs[0][0]):>6}/24 {runs[0][2]:>5}/450 " f"{runs[0][3]:>14.3e}") print(f"\nbaseline {BASELINE:.4f}, network number {OUTCOME['rate 0.05'][0]:.4f}, " f"second run of the same network {OUTCOME['rate 0.05'][1]:.4f}") for name in ("rate 10 bare", "rate 10 clipping", "rate 10 both"): print(f"{name:>16} run difference {abs(OUTCOME[name][0] - OUTCOME[name][1]):.4f} " f"(subscribers {round(abs(OUTCOME[name][0] - OUTCOME[name][1]) * len(TEST))})") ``` ``` setting loss 2 runs test 2 runs dead clipped max_gradient rate 0.05 0.3666 0.3776 0.7817 0.7778 0/24 0/450 4.599e+00 rate 10 bare 0.6210 0.6210 0.7579 0.7579 15/24 0/450 2.800e+07 rate 10 clipping 0.4333 0.3578 0.6984 0.6944 1/24 400/450 7.982e+00 rate 10 schedule 0.5287 0.5287 0.7579 0.7579 15/24 0/450 2.800e+07 rate 10 both 0.2722 0.2473 0.7698 0.7183 0/24 422/450 6.445e+00 baseline 0.7579, network number 0.7817, second run of the same network 0.7778 rate 10 bare run difference 0.0000 (subscribers 0) rate 10 clipping run difference 0.0040 (subscribers 1) rate 10 both run difference 0.0516 (subscribers 13) ``` The second row is the explosion within the training. The gradient magnitude shoots up to **2.800e+07** at some point, that update throws the weights away and **15** of the 24 units never activate again on any training row. The result is the same in both runs: 0.7579, **exactly the baseline**. The network now says "not suspicious" to everyone. That the two initializations give the same number is not good news here — both have fallen to the same dead point. The fourth row says what the schedule does by itself: nothing. The step is halved every six epochs but the explosion happens **in the first epoch**. A schedule does not undo damage that took place before it. In the third row gradient clipping does its job: 400 of the 450 updates are clipped, the largest gradient it sees falls to 7.982 and the dead units go from 15 to 1. But the test number is **0.6984**, clearly below the baseline. Clipping rescued the training, it did not rescue the model. In the fifth row the two together bring the loss to the lowest of the table, 0.2722; the test rises to 0.7698 but in the second run it falls to 0.7183 — **0.0516**, thirteen subscribers. The widest run gap is in the setting that looks best. The triple is as follows: baseline 0.7579, network number 0.7817, second run 0.7778. The shallow setting with the safe step is higher than all of the rescued deep settings, and none of them comes close to the 0.8016 of the linear model. #### Summary - With saturating activation the gradient magnitude of the first layer at six layers falls to 0.0043 times that of the single-layer one, that is, 230 times lower; the ratio to the last layer rises from 18.8 to 3,977.2. - With non-saturating activation the ratio at the same depth stays at 0.7022; at gain 16 the same measure rises to 134.99 times and explodes. What makes it vanish is not depth but the magnitude of the factor. - At a learning rate of 10 the gradient magnitude shoots up to 2.800e+07, 15 of the 24 units die and both runs give exactly the baseline; the schedule by itself does not correct this. - Gradient clipping clips 400 of the 450 updates and brings the dead units down to 1 but the test falls to 0.6984; clipping and the schedule together produce the lowest loss and the widest run gap. #### Next Step In these two lessons the network went wrong in the same direction every time: as the training loss fell, the held-out set number tended to fall. With 300 rows and a few hundred parameters the name of this is known. The next lesson ties that name to a number. Weight decay was measured in Introduction to Machine Learning and Supervised Learning and will not be repeated; what will be added are two methods **specific to the network** — **dropout**, which randomly silences a part of the units during training, and **early stopping**, which cuts the training short. In the second the real question is not the method but the source: **which set** the stopping epoch is read from. ### Regularization Source: https://academia.sh/en/courses/deep-learning/regularization Measuring the contribution of two network-specific regularization methods and the set from which early stopping is read: the gap between training and test of the bare network is 0.0887, with dropout at rate 0.4 it falls to 0.0330 and the test goes from 0.7579 to 0.7937; the second run of the same setting gives 0.7738 and there the bare network beats the dropout network. Weight decay had been swept in Introduction to Machine Learning and Supervised Learning, here it is applied with a single coefficient and brings the gap down to 0.0702. In early stopping the validation loss sees its bottom at the tenth and the fourteenth epoch; when the stopping epoch is read from training the test comes out as 0.7579 and 0.7817, from validation 0.7698 and 0.7976, from test 0.7738 and 0.8016 — the inflation from reading off the test is 0.0040 in both runs, that is, one subscriber. The previous two lessons pointed in the same direction: as the training loss fell, the held-out set number tended to fall. This is a condition with a known name — when a few hundred parameters meet a training set of 300 rows, the network memorizes the rows. Overfitting itself was measured in the Generalization topic of Introduction to Machine Learning and is not repeated here. The question here is narrower: which countermeasures are **specific to the network**, and how many subscribers does each one earn? **Weight decay** is not the first answer to this question and it is not swept again in this course. Regularization Intuition (`04`) in the Generalization topic of Introduction to Machine Learning had moved the penalty coefficient from 0.001 to 1000 and measured the weight magnitude going from 10.040 to 0.601, and Regularization Penalties (`03`) in Supervised Learning had measured the squared, absolute and mixed penalties separating on sparsity. Here the penalty is used with a **single coefficient** and only as a comparison line. The two methods this lesson adds are **dropout** and **early stopping**. A distinction of terms: **dropout** in this course is temporarily silencing a part of the hidden units during training. The downsampling in the Chart Design topic of Visualization, Reporting and Analysis at Scale was a drawing technique and reduced the number of points; the two share only a name. - **NN61.** The set, the split, the seed and the scale are as in the previous lessons; the table is **fictional**, the training set has been **reduced to 300 rows** and the baseline on the test set is **0.7579**. - **NN62.** In this lesson the network has 16 hidden units and carries 225 parameters. It was widened because dropout needs units to close. - **NN63.** Training is 45 epochs, batch size 20, plain descent, learning rate 0.05. These settings were not searched for; they come from the previous lesson. - **NN64.** **Dropout** is applied with a mask redrawn for every mini-batch: the output of a closed unit is zero and the output of the remaining ones is scaled up by $1/(1-p)$. Because the scale is applied in training, **no correction is needed at inference**; outside training the network runs without a mask. - **NN65.** The mask is per mini-batch, not per row. This is a decision and it keeps the number of masks at 45 times 15, that is, 675. - **NN66.** The **weight decay** coefficient is 0.01 and it is added to the gradient as the coefficient times the weight. **Biases are not penalized**: the bias carries the position of the decision boundary, not its curvature. - **NN67.** The criterion of **early stopping** is the **validation loss**, not accuracy; accuracy measures in steps of 0.0040 on a set of 252 rows and leaves the bottom unclear. - **NN68.** The number of settings tried is **5**; this is not a search but the four countermeasures placed side by side with the bare network. The comparison lines come from Supervised Learning: the linear model **0.8016**, the decision tree **0.7857**. #### Two Methods Specific to the Network The setup is the same as in the previous lessons; what is added is that the forward pass takes a **mask** and backpropagation can put a **penalty term** on the gradient. The output of a masked unit is zero, so its gradient is zero as well; the gradient of the remaining units is scaled by the scaling factor in the mask. ```python # regularization.py — MODEL: the same FICTIONAL subscriber table, the same seed, the same split; # the training set is reduced to 300 rows in this topic. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue sub_id = 10001 + i r, v = rng(SEED + 7000 + sub_id), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + sub_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (sub_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for q in REGION: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) r, S = rng(SEED + 90000), list(range(len(DATA))) for i in range(len(S) - 1, 0, -1): j = int(r() * (i + 1)) S[i], S[j] = S[j], S[i] K = [DATA[i] for i in S] TRAIN, VAL, TEST = K[:300], K[756:1008], K[1008:] # training REDUCED from 756 to 300 FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} def scale(s): # LEARNING from training, APPLICATION to every set return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in s], [float(x["suspicious"]) for x in s]) X_TR, Y_TR = scale(TRAIN) X_VA, Y_VA = scale(VAL) X_TE, Y_TE = scale(TEST) BASELINE = sum(y == 0.0 for y in Y_TE) / len(Y_TE) TREE, LINEAR = 198 / len(TEST), 202 / len(TEST) # the two numbers of the Supervised Learning course def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def init_weights(seed, shape, gain=1.0): # gain-scaled; the last column of every row is the BIAS r, layers = rng(seed), [] for fan_in, fan_out in shape: limit = math.sqrt(6.0 * gain / fan_in) layers.append([[(r() * 2 - 1) * limit for _ in range(fan_in)] + [0.0] for _ in range(fan_out)]) return layers def zeros(layers): return [[[0.0] * len(w) for w in W] for W in layers] def make_mask(layers, rate, seed): # DROPOUT: a closed unit is 0, the rest 1/(1-rate) r = rng(seed) return [[0.0 if r() < rate else 1.0 / (1.0 - rate) for _ in W] for W in layers[:-1]] def forward(layers, x, mask=None): a, trace = x, [] for i, W in enumerate(layers): trace.append(a) z = [sum(p * q for p, q in zip(w, a)) + w[-1] for w in W] if i == len(layers) - 1: a = [squash(z[0])] else: a = [t if t > 0 else 0.0 for t in z] if mask: a = [v * m for v, m in zip(a, mask[i])] trace.append(a) return trace def gradient(layers, X, Y, idx, mask=None, penalty=0.0): # BACKPROPAGATION G, loss = zeros(layers), 0.0 for i in idx: trace = forward(layers, X[i], mask) p = min(max(trace[-1][0], 1e-12), 1 - 1e-12) loss -= Y[i] * math.log(p) + (1 - Y[i]) * math.log(1 - p) d = [p - Y[i]] for L in range(len(layers) - 1, -1, -1): W, a = layers[L], trace[L] for k, dk in enumerate(d): gk = G[L][k] for j, aj in enumerate(a): gk[j] += dk * aj gk[-1] += dk if L: mk = mask[L - 1] if mask else None d = [sum(d[k] * W[k][j] for k in range(len(W))) * (0.0 if a[j] <= 0 else (mk[j] if mk else 1.0)) for j in range(len(a))] for W in G: for w in W: w[:] = [v / len(idx) for v in w] if penalty: # WEIGHT DECAY: the bias is not penalized for W, gW in zip(layers, G): for w, g in zip(W, gW): for j in range(len(w) - 1): g[j] += penalty * w[j] return G, loss / len(idx) def loss_of(layers, X, Y): t = 0.0 for x, y in zip(X, Y): p = min(max(forward(layers, x)[-1][0], 1e-12), 1 - 1e-12) t -= y * math.log(p) + (1 - y) * math.log(1 - p) return t / len(X) def accuracy(layers, X, Y): return sum((forward(layers, x)[-1][0] > 0.5) == (y > 0.5) for x, y in zip(X, Y)) / len(X) def order(epoch): # batch split order is the SAME in every run r, s = rng(SEED + 500 + epoch), list(range(len(X_TR))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s LAYERS0 = init_weights(SEED + 1, [(len(FIELDS), 16), (16, 1)]) print(f"network {len(FIELDS)}-16-1, parameters {sum(len(w) for W in LAYERS0 for w in W)}, " f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, epochs 45, batch 20") print(f"baseline {BASELINE:.4f}; comparison lines linear model {LINEAR:.4f}, " f"decision tree {TREE:.4f}") ``` ``` network 12-16-1, parameters 225, train 300, validation 252, test 252, epochs 45, batch 20 baseline 0.7579; comparison lines linear model 0.8016, decision tree 0.7857 ``` #### Dropout and Weight Decay The training loop is in a single function and takes both countermeasures as switches. The measure is three columns: training accuracy, test accuracy and the **gap** between the two. The gap is the direct measure of overfitting; if a regularization method is working it narrows the gap first. ```python def train_model(seed, rate=0.0, penalty=0.0, watch=False, step=0.05, epochs=45, batch=20, hidden=16): layers, curve = init_weights(seed, [(len(FIELDS), hidden), (hidden, 1)]), [] for t in range(epochs): s = order(t) for bi, b in enumerate(range(0, len(s), batch)): mk = make_mask(layers, rate, SEED + 900 + t * 100 + bi) if rate else None G, _ = gradient(layers, X_TR, Y_TR, s[b:b + batch], mk, penalty) for W, gW in zip(layers, G): for w, g in zip(W, gW): for j in range(len(w)): w[j] -= step * g[j] if watch: # the held-out set is read at the end of every epoch curve.append((loss_of(layers, X_TR, Y_TR), loss_of(layers, X_VA, Y_VA), accuracy(layers, X_TE, Y_TE))) return layers, curve SETTINGS = (("bare", 0.0, 0.0), ("dropout 0.2", 0.2, 0.0), ("dropout 0.4", 0.4, 0.0), ("weight decay", 0.0, 0.01), ("dropout 0.4 + decay", 0.4, 0.01)) print(f"{'setting':>21} {'train':>15} {'test':>15} {'gap':>7} {'run':>7}") OUTCOME = {} for name, rate, penalty in SETTINGS: nets = [train_model(SEED + s, rate, penalty)[0] for s in (1, 2)] tr = [accuracy(k, X_TR, Y_TR) for k in nets] OUTCOME[name] = [accuracy(k, X_TE, Y_TE) for k in nets] print(f"{name:>21} {tr[0]:>7.4f} {tr[1]:>7.4f} {OUTCOME[name][0]:>7.4f} {OUTCOME[name][1]:>7.4f} " f"{tr[0] - OUTCOME[name][0]:>7.4f} {abs(OUTCOME[name][0] - OUTCOME[name][1]):>7.4f}") print(f"\nsettings tried {len(SETTINGS)}; baseline {BASELINE:.4f}, network number " f"{OUTCOME['dropout 0.4'][0]:.4f}, second run of the same network {OUTCOME['dropout 0.4'][1]:.4f}") print(f"runs beating the decision tree {sum(v > TREE + 1e-9 for n in OUTCOME.values() for v in n)}/10, " f"the linear model {sum(v > LINEAR + 1e-9 for n in OUTCOME.values() for v in n)}/10") ``` ``` setting train test gap run bare 0.8467 0.8467 0.7579 0.7817 0.0887 0.0238 dropout 0.2 0.8267 0.8300 0.7778 0.7937 0.0489 0.0159 dropout 0.4 0.8267 0.8133 0.7937 0.7738 0.0330 0.0198 weight decay 0.8400 0.8433 0.7698 0.7937 0.0702 0.0238 dropout 0.4 + decay 0.8233 0.8133 0.7897 0.7738 0.0337 0.0159 settings tried 5; baseline 0.7579, network number 0.7937, second run of the same network 0.7738 runs beating the decision tree 4/10, the linear model 0/10 ``` The gap column behaves regularly. In the bare network there are **0.0887** between training and test — twenty-two subscribers. With dropout rate 0.2 this goes down to 0.0489, with 0.4 to **0.0330**. A narrowing of almost three times shows that dropout is doing its job: because every mini-batch works with different units, no unit can memorize a single training row on its own. Weight decay brings the same gap down only to 0.0702; in a setting where the penalty is not swept this is an expected result, and what the sweep earns was measured in Introduction to Machine Learning. The test column confirms the gap in the first run: from 0.7579 to 0.7937, nine subscribers. But the second run does not confirm it. There the bare network gives 0.7817 and the dropout network 0.7738 — **the ranking flips**. Someone looking at the first run says "dropout 0.4 beats the bare network by 0.0358", someone looking at the second run says "it trails by 0.0079". Both ran the same code. The last line sustains the second claim of the course: four of the ten runs beat the decision tree, **none** beats the linear model. #### Which Set Early Stopping Is Read From Early stopping is not a setting but a **reading point**: after how many epochs the training will be cut. The real question is which set is looked at to find the epoch. The block below puts three sources side by side. ```python for s in (1, 2): _, curve = train_model(SEED + s, watch=True) val = min(range(len(curve)), key=lambda t: curve[t][1]) # the bottom of the validation loss tst = max(range(len(curve)), key=lambda t: curve[t][2]) # the peak of the test print(f"run {s}: training loss {curve[0][0]:.4f} at epoch 1, {curve[-1][0]:.4f} at the last " f"epoch; validation loss {curve[0][1]:.4f} -> {curve[-1][1]:.4f}") print(f" stopping epoch read from training 45: test {curve[-1][2]:.4f}") print(f" stopping epoch read from validation {val + 1}: validation loss " f"{curve[val][1]:.4f}, test {curve[val][2]:.4f}") print(f" stopping epoch read from test {tst + 1}: test {curve[tst][2]:.4f}, " f"inflation {curve[tst][2] - curve[val][2]:.4f} " f"(subscribers {round((curve[tst][2] - curve[val][2]) * len(TEST))})") ``` ``` run 1: training loss 0.5183 at epoch 1, 0.3549 at the last epoch; validation loss 0.5244 -> 0.4505 stopping epoch read from training 45: test 0.7579 stopping epoch read from validation 10: validation loss 0.4475, test 0.7698 stopping epoch read from test 11: test 0.7738, inflation 0.0040 (subscribers 1) run 2: training loss 0.6826 at epoch 1, 0.3478 at the last epoch; validation loss 0.6677 -> 0.4244 stopping epoch read from training 45: test 0.7817 stopping epoch read from validation 14: validation loss 0.4129, test 0.7976 stopping epoch read from test 11: test 0.8016, inflation 0.0040 (subscribers 1) ``` The two loss curves part ways. The training loss never rises across forty-five epochs: in the first run it goes from 0.5183 to 0.3549. The validation loss, on the other hand, sees its bottom at the tenth epoch with 0.4475, turns upward and rises to 0.4505 at the last epoch. A stopping epoch cannot be chosen by looking at the training set, because there is no place to stop there — the bottom of the curve is always at the last epoch. The epoch read from the validation set is 10 in the first run and 14 in the second. Their counterparts on the test are 0.7698 and 0.7976; had forty-five epochs been run they would have been 0.7579 and 0.7817. The contribution of early stopping is 0.0119 in the first run and 0.0159 in the second — that is, three and four subscribers. In the first run the bare network had fallen exactly to the baseline; early stopping lifts it above the baseline. The last lines are the real warning of the lesson. Had the epoch been chosen by looking at the test set, it would have been 11 in both runs and the reported numbers would have been 0.7738 and 0.8016. The inflation is **0.0040**, that is, one subscriber. It is small, but it is an unmeasured gain: the moment the test set is read once, it stops being held out. The same logic was built in the pipelines lesson of Data Preparation and Feature Engineering to keep a transformation from being learned from the whole set; the early stopping epoch carries as much information as a transformation. #### Summary - Weight decay had been swept in Introduction to Machine Learning and Supervised Learning; here it is applied with a single coefficient and brings the gap between training and test from 0.0887 down to 0.0702. - Dropout brings the same gap down to 0.0330 at rate 0.4; because every mini-batch works with different units, a single unit cannot memorize a single row. - Because the scaling is done in training, no correction is needed at inference; outside training the network runs without a mask. - The test column shows dropout ahead by 0.0358 in the first run and behind by 0.0079 in the second; four of the ten runs beat the decision tree, none beats the linear model. - The validation loss sees its bottom at the 10th and the 14th epoch; when the stopping epoch is read from validation the test comes out as 0.7698 and 0.7976, from test as 0.7738 and 0.8016 — the inflation is 0.0040 in both runs, one subscriber. #### Next Step Both methods of this lesson touched the **output** of the network: one silenced units, the other shrank weights. The **input** of the layers was never touched. Yet input scaling was the only preparation step used throughout the course and it was applied only to the **first** layer; the numbers the second layer sees drift as training goes on and nobody holds them in place. The next lesson ties this to a measure: when layer inputs are rescaled at every batch, what does the network do at a learning rate that is unstable without normalization. The same lesson also opens a source of leakage, because normalization **does not use the same number** in training and in inference. ### Batch Normalization Source: https://academia.sh/en/courses/deep-learning/batch-normalization Measuring the learning rate that scaling the layer inputs within the mini-batch allows and the source of leakage it opens: the network without normalization carries the loss to 8.6577 at learning rate 20 and at 60 kills 14 of 16 units and gives exactly the baseline; the network with normalization holds the loss at 0.7206 and 2.3263 at the same rates and not a single unit dies at any rate — the largest rate that keeps the loss under 1.0 goes from 5 to 20, a fourfold increase. When inference is made with running averages the test is 0.7698, when it is made with the statistics of the test batch itself it is 0.7937; the inflation is 0.0238, that is, six subscribers. When only the order of the test rows is changed the same model gives 0.7817, seven subscribers change class and the probability of a single subscriber moves by 0.2273. Both methods of the previous lesson touched the output of the network; the **input** of the layers was never touched. Yet scaling was the only preparation step of this course and it was applied only to the **first** layer: twelve columns were scaled with the mean and variability learned from the training set. The numbers the hidden layer sees have no such guarantee; because the weights change at every update, the input scale of the second layer drifts as well. Batch normalization fills this gap: the inputs of a layer are rescaled at every step with the mean and variability of **the mini-batch they are in**. The measure of this lesson is **the learning rate that normalization allows**. The second measure is its cost: normalization does not use the same number in training and in inference, and this difference is a source of leakage. - **NN69.** The set, the split, the seed and the scale are as in the previous lessons; the table is **fictional**, the training set has been **reduced to 300 rows** and the baseline on the test set is **0.7579**. - **NN70.** The network is 12-16-1 with 225 parameters, as in the previous lesson. Training is 30 epochs, batch size 20, plain descent. The only things that change are the learning rate and whether there is normalization. - **NN71.** Normalization is applied to the **pre-activation** values of the hidden layer; the learned scale and shift coefficients are **not used**. To avoid dividing by zero, $10^{-8}$ is added to the variability and this threshold moves nothing at the visible digit. - **NN72.** The **derivative of the normalization itself** is written into backpropagation: because the mean and the deviation come from all the rows of the batch, the gradient of one row depends on the other rows as well. - **NN73.** **At inference** there are no batch statistics; a running average with coefficient 0.9 is accumulated during training and that is used at inference. The dead unit is the definition of the previous lesson. - **NN74.** The number of settings tried is **8**: two normalization states times four learning rates. This is not a search; the rates were chosen to show the limit of stability. The comparison lines come from Supervised Learning: the linear model **0.8016**, the decision tree **0.7857**. #### Scaling Within the Mini-Batch The code of this lesson writes the forward pass and backpropagation **batch by batch, not row by row**, because the statistic to be normalized comes from the whole of the batch. The normalization derivative in backpropagation is a consequence of this: the gradient of one row contains the values of the other rows of the same batch. ```python # normalization.py — MODEL: the same FICTIONAL subscriber table, the same seed, the same split; # the training set is reduced to 300 rows in this topic. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue sub_id = 10001 + i r, v = rng(SEED + 7000 + sub_id), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + sub_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (sub_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for q in REGION: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) r, S = rng(SEED + 90000), list(range(len(DATA))) for i in range(len(S) - 1, 0, -1): j = int(r() * (i + 1)) S[i], S[j] = S[j], S[i] K = [DATA[i] for i in S] TRAIN, TEST = K[:300], K[1008:] # training REDUCED from 756 to 300 FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} def scale(s): # LEARNING from training, APPLICATION to every set return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in s], [float(x["suspicious"]) for x in s]) X_TR, Y_TR = scale(TRAIN) X_TE, Y_TE = scale(TEST) BASELINE = sum(y == 0.0 for y in Y_TE) / len(Y_TE) TREE, LINEAR = 198 / len(TEST), 202 / len(TEST) # the two numbers of the Supervised Learning course def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def init_weights(seed, shape, gain=1.0): # gain-scaled; the last column of every row is the BIAS r, layers = rng(seed), [] for fan_in, fan_out in shape: limit = math.sqrt(6.0 * gain / fan_in) layers.append([[(r() * 2 - 1) * limit for _ in range(fan_in)] + [0.0] for _ in range(fan_out)]) return layers def order(epoch): # batch split order is the SAME in every run r, s = rng(SEED + 500 + epoch), list(range(len(X_TR))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s def forward_batch(W1, W2, Xb, stats=None, norm=False): # THE WHOLE OF ONE MINI-BATCH Z = [[sum(p * q for p, q in zip(w, x)) + w[-1] for w in W1] for x in Xb] mu = sd = None if norm: n, m = len(Z), len(W1) if stats is None: # TRAINING: the batch's own statistics mu = [sum(z[j] for z in Z) / n for j in range(m)] sd = [math.sqrt(sum((z[j] - mu[j]) ** 2 for z in Z) / n + 1e-8) for j in range(m)] else: # INFERENCE: running averages mu, sd = stats Z = [[(z[j] - mu[j]) / sd[j] for j in range(m)] for z in Z] A = [[t if t > 0 else 0.0 for t in z] for z in Z] P = [squash(sum(p * q for p, q in zip(W2[0], a)) + W2[0][-1]) for a in A] return Z, mu, sd, A, P def gradient_batch(W1, W2, Xb, Yb, norm=False): # BACKPROPAGATION Z, mu, sd, A, P = forward_batch(W1, W2, Xb, norm=norm) n, m, loss = len(Xb), len(W1), 0.0 g1 = [[0.0] * len(W1[0]) for _ in W1] g2 = [[0.0] * len(W2[0])] dZ = [[0.0] * m for _ in range(n)] for i in range(n): p = min(max(P[i], 1e-12), 1 - 1e-12) loss -= Yb[i] * math.log(p) + (1 - Yb[i]) * math.log(1 - p) d = p - Yb[i] for j in range(m): g2[0][j] += d * A[i][j] / n dZ[i][j] = d * W2[0][j] * (Z[i][j] > 0) g2[0][-1] += d / n if norm: # the derivative of the normalization itself for j in range(m): s1 = sum(dZ[i][j] for i in range(n)) s2 = sum(dZ[i][j] * Z[i][j] for i in range(n)) for i in range(n): dZ[i][j] = (n * dZ[i][j] - s1 - Z[i][j] * s2) / (n * sd[j]) for i in range(n): for j in range(m): if dZ[i][j]: for k, xk in enumerate(Xb[i]): g1[j][k] += dZ[i][j] * xk / n g1[j][-1] += dZ[i][j] / n return g1, g2, loss / n, mu, sd def predict(W1, W2, X, stats=None, norm=False, batch=0): if not batch: return forward_batch(W1, W2, X, stats, norm)[4] return [p for b in range(0, len(X), batch) for p in forward_batch(W1, W2, X[b:b + batch], None, norm)[4]] def accuracy(P, Y): return sum((p > 0.5) == (y > 0.5) for p, y in zip(P, Y)) / len(Y) def dead_units(W1, W2, run_mean, run_std, norm): # a unit that never activates on any training row A = forward_batch(W1, W2, X_TR, (run_mean, run_std) if norm else None, norm)[3] return sum(all(a[j] <= 0 for a in A) for j in range(len(W1))) W1, W2 = init_weights(SEED + 1, [(len(FIELDS), 16), (16, 1)]) print(f"network {len(FIELDS)}-16-1, parameters {sum(len(w) for w in W1 + W2)}, train {len(TRAIN)}, " f"test {len(TEST)}, epochs 30, batch 20") print(f"baseline {BASELINE:.4f}; comparison lines linear model {LINEAR:.4f}, " f"decision tree {TREE:.4f}") ``` ``` network 12-16-1, parameters 225, train 300, test 252, epochs 30, batch 20 baseline 0.7579; comparison lines linear model 0.8016, decision tree 0.7857 ``` #### The Learning Rate That Normalization Allows The training loop also accumulates the running averages, because at inference there is no batch. ```python def train_model(seed, rate, norm=False, epochs=30, batch=20, hidden=16): W1, W2 = init_weights(seed, [(len(FIELDS), hidden), (hidden, 1)]) run_mean, run_std, last = [0.0] * hidden, [1.0] * hidden, [] for t in range(epochs): s, total = order(t), 0.0 for b in range(0, len(s), batch): i = s[b:b + batch] g1, g2, k, mu, sd = gradient_batch(W1, W2, [X_TR[q] for q in i], [Y_TR[q] for q in i], norm) for w, g in zip(W1, g1): for j in range(len(w)): w[j] -= rate * g[j] for j in range(len(W2[0])): W2[0][j] -= rate * g2[0][j] if norm: # running averages are accumulated for inference run_mean = [0.9 * a + 0.1 * b for a, b in zip(run_mean, mu)] run_std = [0.9 * a + 0.1 * b for a, b in zip(run_std, sd)] total += k * len(i) last.append(total / len(s)) return W1, W2, run_mean, run_std, last[-1] print(f"{'normalization':>15} {'rate':>6} {'loss 2 runs':>16} {'test 2 runs':>16} " f"{'dead':>11}") OUTCOME = {} for norm in (False, True): for rate in (0.05, 5.0, 20.0, 60.0): r = [train_model(SEED + s, rate, norm) for s in (1, 2)] n = [accuracy(predict(a, b, X_TE, (c, d), norm), Y_TE) for a, b, c, d, _ in r] OUTCOME[(norm, rate)] = n print(f"{'yes' if norm else 'no':>15} {rate:>6} {r[0][4]:>7.4f} {r[1][4]:>8.4f} " f"{n[0]:>8.4f} {n[1]:>7.4f} {dead_units(*r[0][:4], norm):>7}/16 " f"{dead_units(*r[1][:4], norm):>2}/16") print(f"\nsettings tried {len(OUTCOME)}; baseline {BASELINE:.4f}, network number " f"{OUTCOME[(True, 0.05)][0]:.4f}, second run of the same network {OUTCOME[(True, 0.05)][1]:.4f}") ``` ``` normalization rate loss 2 runs test 2 runs dead no 0.05 0.3740 0.3671 0.7659 0.7817 0/16 0/16 no 5.0 0.3841 0.3717 0.6905 0.5992 1/16 2/16 no 20.0 8.6577 8.1051 0.6270 0.6190 0/16 0/16 no 60.0 6.0788 6.0788 0.7579 0.7579 14/16 16/16 yes 0.05 0.4020 0.3922 0.7698 0.7857 0/16 0/16 yes 5.0 0.3809 0.4014 0.7381 0.7579 0/16 0/16 yes 20.0 0.7206 0.6698 0.6984 0.6865 0/16 0/16 yes 60.0 2.3263 1.9161 0.6429 0.6587 0/16 0/16 settings tried 8; baseline 0.7579, network number 0.7698, second run of the same network 0.7857 ``` The loss column gives the number in the title of the lesson. The network without normalization is still standing at 5, at 20 the loss goes up to 8.6577; the network with normalization stays at 0.7206 at the same 20 and does not exceed 2.3263 even at 60. The largest rate that keeps the loss under 1.0 is **5** without normalization and **20** with it: **fourfold**. This is what normalization allows. The dead unit column shows the difference more sharply. The network without normalization loses 14 of 16 units at 60 and **all of them** in the second run, and both runs give exactly the baseline. The network with normalization loses **not a single unit** at any rate: rescaling the input at every step keeps the pre-activation values from being pushed permanently below zero. The test column carries a warning: normalization makes the large rate **possible** but does not make it **useful**, because the highest number in both families is at the smallest rate. The triple is this — baseline 0.7579, network number 0.7698, second run of the same network **0.7857**; the second is 0.0159 higher than the first and lands exactly on the number of the decision tree. #### Different in Training, Different at Inference The cost of normalization is **the splitting of behavior in two**. In training the output of a row depends on the nineteen rows next to it; at inference it should not. The block below makes the inference in three ways. ```python W1, W2, RUN_MEAN, RUN_STD, _ = train_model(SEED + 1, 0.05, True) P_A = predict(W1, W2, X_TE, (RUN_MEAN, RUN_STD), True) # the right way: running averages P_B = predict(W1, W2, X_TE, norm=True, batch=20) # the batch's own statistics r, S2 = rng(SEED + 4242), list(range(len(X_TE))) for i in range(len(S2) - 1, 0, -1): j = int(r() * (i + 1)) S2[i], S2[j] = S2[j], S2[i] P_C = [0.0] * len(X_TE) for k, i in enumerate(predict(W1, W2, [X_TE[q] for q in S2], norm=True, batch=20)): P_C[S2[k]] = i print(f"inference with running averages {accuracy(P_A, Y_TE):.4f}") print(f"inference with batch statistics {accuracy(P_B, Y_TE):.4f} " f"inflation {accuracy(P_B, Y_TE) - accuracy(P_A, Y_TE):.4f} " f"(subscribers {round((accuracy(P_B, Y_TE) - accuracy(P_A, Y_TE)) * len(TEST))})") print(f"same path, test order changed {accuracy(P_C, Y_TE):.4f}") print(f"subscribers who changed class: between running averages and batch statistics " f"{sum((a > 0.5) != (b > 0.5) for a, b in zip(P_A, P_B))}, " f"only when the order changes {sum((a > 0.5) != (b > 0.5) for a, b in zip(P_B, P_C))}") print(f"largest movement in the probability of a single subscriber " f"{max(abs(a - b) for a, b in zip(P_B, P_C)):.4f}") ``` ``` inference with running averages 0.7698 inference with batch statistics 0.7937 inflation 0.0238 (subscribers 6) same path, test order changed 0.7817 subscribers who changed class: between running averages and batch statistics 6, only when the order changes 7 largest movement in the probability of a single subscriber 0.2273 ``` The second line looks like good news at first glance: 0.7937 instead of 0.7698, six more subscribers correct. But those six subscribers do not come from learning. The mean of the test batch was computed from the test rows themselves; the prediction of every subscriber contains the numbers of the other nineteen subscribers in the same batch. The third line proves it. The model is the same, the weights are the same, the test set is the same — only the **order** of the rows changed and the number went down from 0.7937 to 0.7817. Seven subscribers changed class, the probability of one subscriber moved by **0.2273**. The prediction of a model should depend on no one but the person being predicted; here it does. The pipelines lesson of Data Preparation and Feature Engineering had measured that learning a transformation from the whole set produced an inflation between 0.0397 and 0.0476; batch normalization makes the same mistake possible **silently**. There is only one right way: running averages at inference. #### Summary - Batch normalization scales the pre-activation values of a layer with the mean and variability of the mini-batch it is in; its derivative also comes from the whole of the batch. - The largest learning rate that keeps the loss under 1.0 is 5 without normalization and 20 with it — fourfold. The network without normalization carries the loss to 8.6577 at 20. - At rate 60, 14 of the 16 units of the network without normalization die, and in the second run all of them, and the result is exactly the baseline; with normalization no unit dies at any rate. - The large rate becomes possible but not useful: the highest test number is at the smallest rate in both families. The triple is baseline 0.7579, network number 0.7698, second run 0.7857. - When inference is made with the statistics of the batch itself the number inflates to 0.7937; when only the test order changes it goes down to 0.7817 and one probability moves by 0.2273. #### Next Step Nine lessons built a network piece by piece: nonlinear separation, activation, forward propagation, loss, backpropagation, optimizer, stability, regularization and normalization. In every lesson three numbers stood side by side and the third — the second run of the same network — never came out the same as the first. The rule of the course was confirmed in the last four lessons: in each lesson the capacity was held constant and only one setting was changed, and the optimizer, the learning rate, dropout and normalization moved the number between 0.5992 and 0.7937. The second claim of the course also holds: no run beat 0.8016 of the linear model. There is one side of this network that did not change in nine lessons. Every input column was connected to every hidden unit and every connection carried its own weight — the network stayed **fully connected**. This is behaving as if the input columns had no relation to one another: if the order of the columns were shuffled the network would give the same result. Yet most real inputs have **structure**; in a series of meter readings neighboring periods, in a coverage grid neighboring cells are neighbors. The next topic writes this structure into the network itself and asks the first question with a parameter count: how many weights does a fully connected layer need to do the same job, and how many does a layer that knows about neighborhood need. ## Image and Sequence Architectures Counting what architectures that use the structure of the input add and what they still miss: how many weights convolution needs to do the same job as a fully connected layer, the information pooling throws away and its resistance to shifts, the gradient reaching the first layer as depth grows and the skip connection correcting it, how much information the hidden state can carry as the sequence lengthens, how many steps back the gated units carry a dependency, how many of the attention weights go to the correct step and why a high weight is not an explanation, the difference the number of heads adds and the order information that disappears when positional encoding is removed, the difference between a learned embedding and a projection that is not learned, how many examples transfer adds with and the distinction between freezing and fine-tuning, the reconstruction error and the instability of the adversarial arrangement between two runs. ### Convolution Operation Source: https://academia.sh/en/courses/deep-learning/convolution-operation Counting how many weights the first layer that uses the structure of the input needs: a fully connected layer producing the same 144 outputs holds 20,880 numbers while a 3x3 convolution kernel holds 10, a ratio of 2,088 times. Two hand-written kernels give 0.8900 agreement without padding and 0.9380 with padding 1, and the difference is entirely the edge cells. On the test set the most frequent class gives 0.5850, while a fully connected network with 1,169 parameters memorizes the training set completely and reads 0.5100 and 0.5350, and a convolutional network with 23 parameters reads 0.8000 and 0.8150: the difference between the two runs is 0.0250 for the fully connected network and 0.0150 for the convolutional network. Over nine lessons the network stayed **fully connected**: every input column was tied to every unit with its own weight. In such a layer the order of the columns means nothing; if the columns were shuffled, what the network learns would not change, only the positions of the weights would. In the subscriber table this is not a loss, because there is no adjacency between household size and mean consumption. When the input is a **grid**, the situation changes. What is sought lies not in a single cell but in the arrangement of several adjacent cells. A fully connected layer does not use this: it learns a separate weight for every cell, and when the same arrangement appears somewhere else it has to learn it again. This is the measure of this lesson — with how many weights a fully connected layer does the same job, and with how many convolution does. - **AR1. The coverage grid is fictional.** In the Visualization, Reporting and Analysis at Scale course the regions were divided into a 12x12 grid of cells; here the same grid is turned into a learning input. Each cell is a coverage unit, and its value is whether the unit has a reading above the threshold. Seed 20260218. - **AR2. The label is fictional too, and structural.** A grid is labeled 1 if a **line of three adjacent cells** has been planted in it. The line stands for a leak trace running along a pipeline; scattered single cells are noise. - **AR3. The label cannot be read exactly from the grid.** About 9% of the cells are active on their own; in some grids with no line planted, a three-cell arrangement also appears by chance. - **AR4. The split is 200/100/200** and fixed in this grid set. The training set is deliberately **reduced**; the scale is kept small. - **AR5. The baseline is read from the grid set.** Saying the most frequent class on the test set gives 0.5850. The 0.7579 of the Introduction to Machine Learning, Data Preparation and Feature Engineering, and Supervised Learning courses belongs to the **subscriber table** and cannot be compared with this set. - **AR6. The kernel** is a small weight matrix slid over the grid; the **stride** is the shift between two neighboring positions, and the **padding** is the thickness of the frame of zeros added to the edge. - **AR7. Weight sharing** is the definition of convolution: the same kernel is used at all positions. This makes the parameter count independent of the number of positions. - **AR8. No library is called;** the convolution and the training loop are written inside the lesson and are **a model**. - **AR9. Every network is run twice.** In the second run only the initial values change; the data, the split, the number of epochs and the learning rate are the same. #### The Coverage Grid and the Parameter Count ```python # grid.py — MODEL: the coverage grid is FICTIONAL: the 12x12 coverage grid of the # Visualization, Reporting and Analysis at Scale course becomes a learning input # here. Each cell is a coverage unit; its value is whether it has a reading above the threshold. import math SEED, M32, SIDE = 20260218, 0xFFFFFFFF, 12 DENSITY, TRACE_RATE, RUN_LENGTH = 0.09, 0.42, 3 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def grid(subscriber_id): """A subscriber's coverage grid. Label: whether a leak trace was planted.""" r = rng(SEED + 300000 + subscriber_id) g = [[1 if r() < DENSITY else 0 for _ in range(SIDE)] for _ in range(SIDE)] trace = 1 if r() < TRACE_RATE else 0 if trace: # a line of three adjacent cells horizontal, a, b = r() < 0.5, int(r() * SIDE), int(r() * (SIDE - RUN_LENGTH + 1)) for k in range(RUN_LENGTH): if horizontal: g[a][b + k] = 1 else: g[b + k][a] = 1 return g, trace def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s RAW = [grid(i) for i in range(500)] DATA = [RAW[i] for i in shuffle(500, SEED + 90000)] TRAIN, VAL, TEST = DATA[:200], DATA[200:300], DATA[300:] BASELINE = max(sum(y for _, y in TEST), len(TEST) - sum(y for _, y in TEST)) / len(TEST) example = next(g for g, y in DATA if y == 1 and any( all(g[i][j + k] for k in range(RUN_LENGTH)) for i in range(SIDE) for j in range(SIDE - RUN_LENGTH + 1))) print("grid with a horizontal line planted (1 = reading above threshold), label 1:") for row in example: print(" " + "".join("#" if x else "." for x in row)) print(f"grid {SIDE}x{SIDE} = {SIDE * SIDE} cells, mean active cells " f"{sum(sum(sum(s) for s in g) for g, _ in DATA) / len(DATA):.2f}") print(f"split {len(TRAIN)}/{len(VAL)}/{len(TEST)}, most frequent class on the test set " f"{BASELINE:.4f}") FC_HIDDEN = 8 fc_params = SIDE * SIDE * FC_HIDDEN + FC_HIDDEN + FC_HIDDEN + 1 fc_layer = (SIDE * SIDE) ** 2 + SIDE * SIDE print(f"\n{'layer':<34}{'weight':>9}{'bias':>7}{'total':>9}") print(f"{'fully connected 144 -> 144':<34}{(SIDE * SIDE) ** 2:>9}{SIDE * SIDE:>7}" f"{fc_layer:>9}") print(f"{'convolution 3x3, single kernel':<34}{9:>9}{1:>7}{10:>9}") print(f"ratio for the same output size {fc_layer / 10:.0f}x") ``` ``` grid with a horizontal line planted (1 = reading above threshold), label 1: #..#........ ....#....... ..#...#..... ............ #.#......... .....#..#... #....#.....# .####....##. .......##... ..........#. ..........#. ............ grid 12x12 = 144 cells, mean active cells 13.96 split 200/100/200, most frequent class on the test set 0.5850 layer weight bias total fully connected 144 -> 144 20736 144 20880 convolution 3x3, single kernel 9 1 10 ratio for the same output size 2088x ``` In the example grid, three of the four adjacent cells in the eighth row are the planted line, and the fourth is a noise cell that happened to land next to it. The remaining active cells are scattered. The table at the bottom is the lesson's number. A fully connected layer that produces one output per cell holds **20,880** numbers. A single 3x3 kernel that produces the same output size holds **10**: nine weights and one bias. The ratio is **2,088 times**. The difference is not an optimization trick but an **assumption**. The fully connected layer says "every position may have its own rule" and reserves room for 144 separate rules; convolution says "the pattern sought is the same pattern wherever it is" and learns a single rule. If the assumption is wrong, convolution cannot learn. On this grid it holds, because the line is planted anywhere. #### Kernel, Stride and Padding ```python # convolution.py — the convolution operation is written by hand: the kernel is slid over the # grid and the same weights are used at every position. The input is binary, so the loop # runs only over active cells; the computed number does not change, the run gets shorter. def active_cells(g): return [(i, j) for i in range(SIDE) for j in range(SIDE) if g[i][j]] def convolve(active, K, bias, stride=1, padding=0): c = len(K) n = (SIDE + 2 * padding - c) // stride + 1 output = [[bias] * n for _ in range(n)] for (i, j) in active: for di in range(c): for dj in range(c): p, q = i + padding - di, j + padding - dj if p % stride == 0 and q % stride == 0 and 0 <= p // stride < n \ and 0 <= q // stride < n and p >= 0 and q >= 0: output[p // stride][q // stride] += K[di][dj] return output HORIZONTAL = [[0, 0, 0], [1, 1, 1], [0, 0, 0]] VERTICAL = [[0, 1, 0], [0, 1, 0], [0, 1, 0]] def fires(g, padding): active = active_cells(g) return (max(max(r) for r in convolve(active, HORIZONTAL, -2.5, 1, padding)) > 0 or max(max(r) for r in convolve(active, VERTICAL, -2.5, 1, padding)) > 0) print("horizontal kernel output (+ = line found), padding 1, 12x12:") for row in convolve(active_cells(example), HORIZONTAL, -2.5, 1, 1): print(" " + "".join("+" if v > 0 else "." for v in row)) for padding in (0, 1): agree = sum(1 for g, y in DATA if fires(g, padding) == (y == 1)) print(f"two hand-written kernels, padding {padding}: agreement with label " f"{agree / len(DATA):.4f}") print(f"\n{'stride':>7}{'padding':>8}{'output':>8}{'weight':>9}{'mults':>9}" f"{'fully connected mults':>23}") for stride, padding in ((1, 0), (1, 1), (2, 0), (2, 1), (3, 1)): n = (SIDE + 2 * padding - 3) // stride + 1 print(f"{stride:>7}{padding:>8}{f'{n}x{n}':>8}{10:>9}{n * n * 9:>9}" f"{n * n * SIDE * SIDE:>23}") ``` ``` horizontal kernel output (+ = line found), padding 1, 12x12: ............ ............ ............ ............ ............ ............ ............ ..++........ ............ ............ ............ ............ two hand-written kernels, padding 0: agreement with label 0.8900 two hand-written kernels, padding 1: agreement with label 0.9380 stride padding output weight mults fully connected mults 1 0 10x10 10 900 14400 1 1 12x12 10 1296 20736 2 0 5x5 10 225 3600 2 1 6x6 10 324 5184 3 1 4x4 10 144 2304 ``` When the kernel is written by hand, what it does becomes visible. The horizontal kernel takes the sum of three adjacent cells and, with a bias of $-2.5$, produces a positive value only if all three are active; on the example grid it fires at two positions of the seventh row, because there are four adjacent cells there. These two kernels, with no training at all, give **0.9380** agreement over the whole set. The difference between the two agreement numbers is the measure of **padding** alone. When working without padding, the center of the kernel cannot sit on the first and last rows, lines at the edge are not seen, and agreement drops to 0.8900. Adding one row of zeros closes a gap of **0.0480**; padding is not a tuning convenience but a decision about whether to see the information at the edge. The table below shows that the output size follows from the relation $n = \lfloor (12 + 2d - c)/s \rfloor + 1$ with stride $s$, padding $d$ and kernel size $c$. The column to watch is the **weight** column: it is 10 whatever the stride and padding. Stride changes only the number of **multiplications** — from 1,296 to 144. The fully connected layer does sixteen times more multiplications in every row, and every one of those multiplications has a weight of its own. #### Two Networks Side by Side The parameter count is a promise; the number on the held-out set is what it amounts to. Two networks are trained on the same data with the same budget; the only difference is the form of the first layer. ```python # train.py — MODEL: two networks are trained on the same data with the same epoch budget. # Output sigmoid, loss cross-entropy, update per example. PADDING, N = 1, SIDE + 2 * 1 - 3 + 1 TRAIN_ACTIVE = [(active_cells(g), y) for g, y in TRAIN] TEST_ACTIVE = [(active_cells(g), y) for g, y in TEST] def sigmoid(z): return 1 / (1 + math.exp(-z)) if z > -30 else 0.0 def fully_connected(H, seed): # 144 -> H -> 1 r, a = rng(seed), (1 / (SIDE * SIDE)) ** 0.5 return [[[(r() * 2 - 1) * a for _ in range(SIDE * SIDE)] for _ in range(H)], [0.0] * H, [(r() * 2 - 1) * (1 / H) ** 0.5 for _ in range(H)], [0.0]] def fc_forward(P, active): W, b, v, c = P h = [max(0.0, b[k] + sum(W[k][i * SIDE + j] for (i, j) in active)) for k in range(len(b))] return h, sigmoid(c[0] + sum(v[k] * h[k] for k in range(len(h)))) def convolutional(count, seed): # count kernels -> max -> 1 r, a = rng(seed), (1 / 9) ** 0.5 return [[[[(r() * 2 - 1) * a for _ in range(3)] for _ in range(3)] for _ in range(count)], [0.0] * count, [(r() * 2 - 1) * (1 / count) ** 0.5 for _ in range(count)], [0.0]] def conv_forward(P, active): Ks, b, v, c = P h, arg = [], [] for k, K in enumerate(Ks): m = convolve(active, K, b[k], 1, PADDING) best, bp, bq = m[0][0], 0, 0 for p in range(N): for q in range(N): if m[p][q] > best: best, bp, bq = m[p][q], p, q h.append(max(0.0, best)) arg.append((bp, bq)) return (h, arg), sigmoid(c[0] + sum(v[k] * h[k] for k in range(len(h)))) def train_model(P, forward, data, epochs, lr, seed, is_conv): r, n = rng(seed), len(data) W, b, v, c = P for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: active, y = data[idx] cache, p = forward(P, active) h = cache[0] if is_conv else cache d = p - y for k in range(len(b)): if h[k] > 0.0: dh = d * v[k] b[k] -= lr * dh if is_conv: bp, bq = cache[1][k] for (i, j) in active: di, dj = i + PADDING - bp, j + PADDING - bq if 0 <= di < 3 and 0 <= dj < 3: W[k][di][dj] -= lr * dh else: for (i, j) in active: W[k][i * SIDE + j] -= lr * dh v[k] -= lr * d * h[k] c[0] -= lr * d return P def accuracy(P, forward, data, is_conv): return sum(1 for active, y in data if (forward(P, active)[1] >= 0.5) == (y == 1)) / len(data) conv_params = 2 * (9 + 1) + 2 + 1 print(f"{'network':<25}{'parameters':>11}{'run':>7}{'train':>8}{'test':>8}") print(f"{'baseline':<25}{0:>11}{'-':>7}{'-':>8}{BASELINE:>8.4f}") for name, build, forward, is_conv, params in ( ("fully connected 144-8-1", lambda t: fully_connected(8, t), fc_forward, False, fc_params), ("convolutional 2x3x3", lambda t: convolutional(2, t), conv_forward, True, conv_params)): for k, seed in enumerate((SEED + 11, SEED + 12)): P = train_model(build(seed), forward, TRAIN_ACTIVE, 60, 0.10, seed + 500, is_conv) print(f"{name if k == 0 else '':<25}{params if k == 0 else '':>11}{k + 1:>7}" f"{accuracy(P, forward, TRAIN_ACTIVE, is_conv):>8.4f}" f"{accuracy(P, forward, TEST_ACTIVE, is_conv):>8.4f}") ``` ``` network parameters run train test baseline 0 - - 0.5850 fully connected 144-8-1 1169 1 1.0000 0.5100 2 0.9950 0.5350 convolutional 2x3x3 23 1 0.7850 0.8000 2 0.8150 0.8150 ``` Three numbers stand side by side. The baseline is **0.5850**. The fully connected network memorizes the training set completely with **1,169** parameters — 1.0000 in the first run — and reads **0.5100** on the test set, that is 0.0750 **below** the baseline. The convolutional network reads **0.8000** on the test set with **23** parameters, passing the baseline by 0.2150. The failure of the fully connected network is not a training defect. Two hundred grids never resemble each other in a 144-dimensional binary space; the network is wide enough to memorize each of them separately, and memorizing is easier than learning where the line is. When the line appears somewhere else, none of the memorized weights is of any use. **The third number** is written for both networks. The two runs of the fully connected network read 0.5100 and 0.5350, a difference of **0.0250**; those of the convolutional network read 0.8000 and 0.8150, a difference of **0.0150**. With capacity, data and budget fixed, changing only the initial values opens a measurable difference. Writing the number of a single run means making this uncertainty invisible. The convolutional network cannot reach even the 0.9380 of the two hand-written kernels. The cause is not capacity but the structure after the first layer: the **largest** value over the entire grid is taken for each kernel and the remaining 143 positions are discarded. This network answers the question "is there a line" and does not answer "where". #### Summary - The coverage grid is fictional: 12x12 cells, 9% activity, the label is a line of three adjacent cells; the most frequent class on the test set gives 0.5850. - A fully connected layer producing the same 144 outputs holds 20,880 numbers, a single 3x3 convolution kernel holds 10; the ratio is 2,088 times and its source is weight sharing. - The output size is computed from stride, padding and kernel size; the number of weights is independent of these three and only the number of multiplications changes (1,296 at stride 1 padding 1, 144 at stride 3). - Working without padding does not see the lines at the edge: agreement rises from 0.8900 to 0.9380, a difference of 0.0480. - A fully connected network with 1,169 parameters memorizes the training set and reads 0.5100 and 0.5350 on the test set, staying below the baseline; a convolutional network with 23 parameters reads 0.8000 and 0.8150. - The difference the second run opens is 0.0250 for the fully connected network and 0.0150 for the convolutional network. #### Next Step The first layer of the convolutional network produced a response at 144 positions, but the network used only **the largest** one and threw away the remaining 143 numbers. This is the most extreme example of discarding: all of the position information is gone. The next lesson turns this into an **adjustable step**; the responses of neighboring positions are combined in small windows. Three things will be measured: how far the output size drops, how many of the discarded cells can be rebuilt, and in how many examples the decision changes when the grid is shifted by one cell. ### Pooling Source: https://academia.sh/en/courses/deep-learning/pooling Counting what pooling discards and what it gives in return: 2x2 max pooling reduces 144 cells to 36, 96.9 cells can be recovered on average and an uncertainty of 46.0 bits remains. When the raw grid is pooled the line information is lost entirely and agreement falls from 0.9380 to 0.4220; when the same pooling is applied to the feature map, shift invariance rises from 0.0% to 38.1%, and to 88.7% with global pooling. With a baseline of 0.5850, the network without pooling gives 0.5150 and 0.5350 with 309 parameters, the 2x2 pooled network gives 0.6100 and 0.6150 with 93 parameters, and the globally pooled network gives 0.8050 and 0.8050 with 23 parameters; the number of decisions that change on the shifted test set is 84, 49 and 12 respectively. The convolutional network of the previous lesson produced a response at 144 positions in its first layer and used only the largest of them. The remaining 143 numbers were discarded; the network no longer knew **where** the line was, only **whether** there was one. This is the most extreme form of discarding. The name of this discarding operation is **pooling**, and it is not limited to the extreme form: the window size is a hyperparameter and can take every value from 1 up to the whole grid. The measure of this lesson has two sides — what pooling discards in a way that cannot be recovered, and what it gives in return. - **AR10. The set, the split and the baseline do not change.** The fictional coverage grid of the previous lesson is generated with the same seed: 12x12 cells, 9% activity, a line of three adjacent cells as the label. The split is 200/100/200, and the most frequent class on the test set is **0.5850**. - **AR11. Max pooling** reduces each of the $p \times p$ disjoint windows to the largest value in that window. **Average pooling** reduces the same window to its mean. Neither carries a parameter to be learned. - **AR12. Pooling is a reduction whose stride equals the window size**; the output side is $12/p$. When the window size reaches 12 a single number remains; this extreme case is called **global pooling**. - **AR13. Recoverability is counted.** In a binary grid, max pooling determines all the cells of a window that gives zero and none of the cells of a window that gives one. The remaining uncertainty is the binary logarithm of the number of original grids that could give the same pooled output. - **AR14. Where pooling is applied matters.** Pooling applied to the raw grid and pooling applied to the output of a convolution — the **feature map** — are the same operation with different results. - **AR15. Shift invariance** is the fraction of examples whose representation is preserved exactly when the grid is shifted by one cell. Grids whose map is entirely zero are left out of this measure, because shifting changes nothing in them anyway. - **AR16. The shift is one cell down**; a row of zeros is added at the top and the bottom row falls off. In grids where the line is on the bottom edge, information is genuinely lost; this is part of the measure. - **AR17. No library is called;** pooling, convolution and the training loop are written inside the lesson and are **a model**. - **AR18. Every network is run twice;** in the second run only the initial values change. #### What Pooling Discards ```python # pooling.py — MODEL: the FICTIONAL coverage grid of 01 is generated with the same seed. import math SEED, M32, SIDE = 20260218, 0xFFFFFFFF, 12 DENSITY, TRACE_RATE, RUN_LENGTH = 0.09, 0.42, 3 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def grid(subscriber_id): r = rng(SEED + 300000 + subscriber_id) g = [[1 if r() < DENSITY else 0 for _ in range(SIDE)] for _ in range(SIDE)] trace = 1 if r() < TRACE_RATE else 0 if trace: horizontal, a, b = r() < 0.5, int(r() * SIDE), int(r() * (SIDE - RUN_LENGTH + 1)) for k in range(RUN_LENGTH): if horizontal: g[a][b + k] = 1 else: g[b + k][a] = 1 return g, trace def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s RAW = [grid(i) for i in range(500)] DATA = [RAW[i] for i in shuffle(500, SEED + 90000)] TRAIN, TEST = DATA[:200], DATA[300:] BASELINE = max(sum(y for _, y in TEST), len(TEST) - sum(y for _, y in TEST)) / len(TEST) def pool(m, p, average=False): n = len(m) // p return [[(sum(m[i * p + a][j * p + b] for a in range(p) for b in range(p)) / (p * p) if average else max(m[i * p + a][j * p + b] for a in range(p) for b in range(p))) for j in range(n)] for i in range(n)] def has_line(m, k): n = len(m) for i in range(n): for j in range(n - k + 1): if all(m[i][j + t] > 0 for t in range(k)) or \ all(m[j + t][i] > 0 for t in range(k)): return True return False print(f"{'window':>8}{'output':>8}{'cells':>7}{'discarded':>10}" f"{'recovered':>14}{'possible grids (log2)':>23}") for p in (1, 2, 3, 4): n = SIDE // p known, uncertain = 0, 0 for g, y in DATA: s = sum(sum(r) for r in pool(g, p)) known += (n * n - s) * p * p + (s * p * p if p == 1 else 0) uncertain += s known, uncertain = known / len(DATA), uncertain / len(DATA) print(f"{f'{p}x{p}':>8}{f'{n}x{n}':>8}{n * n:>7}{SIDE * SIDE - n * n:>10}" f"{known:>13.1f} {uncertain * math.log2(2 ** (p * p) - 1):>22.1f}") print(f"\n{'pooling':>10}{'line sought':>12}{'agreement with label':>22}") for p, k in ((1, 3), (2, 2), (3, 2)): agree = sum(1 for g, y in DATA if has_line(pool(g, p), k) == (y == 1)) print(f"{('none' if p == 1 else f'{p}x{p}'):>10}{k:>12}{agree / len(DATA):>22.4f}") print(f"baseline (most frequent class on the test set) {BASELINE:.4f}") actual = sum(sum(sum(r) for r in g) for g, _ in DATA) / len(DATA) largest = sum(sum(sum(r) for r in pool(g, 2)) for g, _ in DATA) / len(DATA) mean = sum(sum(sum(r) for r in pool(g, 2, True)) * 4 for g, _ in DATA) / len(DATA) print(f"active cell count {actual:.2f}; lower bound read from 2x2 max pooling " f"{largest:.2f}, read from average pooling {mean:.2f}") ``` ``` window output cells discarded recovered possible grids (log2) 1x1 12x12 144 0 144.0 0.0 2x2 6x6 36 108 96.9 46.0 3x3 4x4 16 128 60.0 84.0 4x4 3x3 9 135 30.8 113.2 pooling line sought agreement with label none 3 0.9380 2x2 2 0.4220 3x3 2 0.4060 baseline (most frequent class on the test set) 0.5850 active cell count 13.96; lower bound read from 2x2 max pooling 11.78, read from average pooling 13.96 ``` The first table gives the size of the discarding. A 2x2 window reduces 144 cells to 36; 108 cells are gone. Not all of the gone cells are unrecoverable: a window that gives zero says that all four of its cells are zero, so the values of **96.9** cells on average are determined. The share that remains uncertain is **46.0 bits** — there are original grids of the order of $2^{46}$ that give the same pooled output. When the window rises to 4x4 the uncertainty climbs to 113.2 bits and the recovered cells fall to 30.8. The second table is the real warning. When a line of three cells is sought in the raw grid, agreement is 0.9380. When the same grid is pooled 2x2 and a line of two cells is sought, agreement falls to **0.4220**, that is, **below** the baseline of 0.5850. When pooling is applied to the raw input, noise cells move close together and produce false lines, and a real line can no longer be told apart from chance. Here pooling does not merely **discard** information; it discards exactly the information that tells the cases apart. The last line gives the difference between the two kinds of pooling. The number of active cells is read back exactly from average pooling: 13.96. What is read from max pooling is only a **lower bound**, 11.78. Average pooling preserves the count, max pooling preserves presence; which one does the job depends on the question being asked. #### Pooling on the Feature Map ```python # feature_map.py — pooling is applied not to the raw grid but to the FEATURE # MAP produced by the convolution. The kernels are the hand-written kernels of 01. PADDING = 1 HORIZONTAL = [[0, 0, 0], [1, 1, 1], [0, 0, 0]] VERTICAL = [[0, 1, 0], [0, 1, 0], [0, 1, 0]] def active_cells(g): return [(i, j) for i in range(SIDE) for j in range(SIDE) if g[i][j]] def convolve(active, K, bias): n = SIDE + 2 * PADDING - 3 + 1 output = [[bias] * n for _ in range(n)] for (i, j) in active: for di in range(3): for dj in range(3): p, q = i + PADDING - di, j + PADDING - dj if 0 <= p < n and 0 <= q < n: output[p][q] += K[di][dj] return output def representation(g, p): active, rep = active_cells(g), [] for K in (HORIZONTAL, VERTICAL): m = [[max(0.0, v) for v in s] for s in convolve(active, K, -2.5)] rep += [v for s in (pool(m, p) if p > 1 else m) for v in s] return rep def shift(g): # shift one cell down return [[0] * SIDE] + [s[:] for s in g[:-1]] NONZERO = [g for g, y in TEST if any(v > 0 for v in representation(g, 1))] print(f"test grids whose map is not entirely zero {len(NONZERO)}/{len(TEST)}") print(f"{'pooling':>10}{'map':>9}{'head input':>12}{'head params':>13}" f"{'total params':>14}{'same after shift':>17}") for p in (1, 2, 3, 12): n = SIDE // p head = 2 * n * n same = sum(1 for g in NONZERO if all(abs(x - y) < 1e-9 for x, y in zip(representation(g, p), representation(shift(g), p)))) print(f"{('none' if p == 1 else f'{p}x{p}'):>10}{f'{n}x{n}':>9}{head:>12}" f"{head + 1:>13}{20 + head + 1:>14}{same / len(NONZERO):>16.1%}") ``` ``` test grids whose map is not entirely zero 97/200 pooling map head input head params total params same after shift none 12x12 288 289 309 0.0% 2x2 6x6 72 73 93 38.1% 3x3 4x4 32 33 53 55.7% 12x12 1x1 2 3 23 88.7% ``` The same operation in a different place gives the opposite result. When pooling is applied to the output of the convolution the line information is not lost, because the kernel has **already found** the line; what remains on the map is the position and strength of the line, and pooling only makes the position coarser. The gain is in the last column. In the representation without pooling, a one-cell shift changes the map in **every** grid: invariance is 0.0%. With 2x2 pooling the representation is preserved exactly in **38.1%** of the grids, with 3x3 in 55.7%, and with global pooling in **88.7%**. The reason is something that can be counted: as long as the largest value moves around inside the window, the output of the window does not change, that is, pooling absorbs a shift as large as the window size. The remaining 11.3% is the share where the shift genuinely carries information away: if the line is in the bottom row, the shift throws it out of the grid. The three middle columns write the price — the input of the head falls from 288 to 2, and the total parameter count from 309 to 23. #### Three Poolings, Three Networks Until now the kernels were written by hand. Below the kernels are learned, and the only thing that changes is the pooling window. ```python # train.py — MODEL: kernels are learned, the pooling window changes. # Output sigmoid, loss cross-entropy, update per example. TRAIN_ACTIVE = [(active_cells(g), y) for g, y in TRAIN] TEST_ACTIVE = [(active_cells(g), y) for g, y in TEST] SHIFTED_ACTIVE = [(active_cells(shift(g)), y) for g, y in TEST] def sigmoid(z): return 1 / (1 + math.exp(-z)) if z > -30 else 0.0 def build(count, p, seed): r, n = rng(seed), SIDE // p return [[[[(r() * 2 - 1) / 3 for _ in range(3)] for _ in range(3)] for _ in range(count)], [0.0] * count, [(r() * 2 - 1) / (count * n * n) ** 0.5 for _ in range(count * n * n)], [0.0], p] def forward(P, active): Ks, b, v, c, p = P n, z, record, idx = SIDE // p, c[0], [], 0 for k, K in enumerate(Ks): m = convolve(active, K, b[k]) for i in range(n): for j in range(n): best, bp, bq = m[i * p][j * p], i * p, j * p for a in range(p): for t in range(p): if m[i * p + a][j * p + t] > best: best, bp, bq = m[i * p + a][j * p + t], i * p + a, j * p + t h = max(0.0, best) record.append((k, bp, bq, h)) z += v[idx] * h idx += 1 return record, sigmoid(z) def train_model(P, data, epochs, lr, seed): r, n = rng(seed), len(data) Ks, b, v, c, p = P for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: active, y = data[idx] record, out = forward(P, active) d = out - y D = [[[0.0] * SIDE for _ in range(SIDE)] for _ in Ks] for t, (k, bp, bq, h) in enumerate(record): if h > 0.0: D[k][bp][bq] += d * v[t] v[t] -= lr * d * h c[0] -= lr * d for k in range(len(Ks)): b[k] -= lr * sum(sum(s2) for s2 in D[k]) for di in range(3): for dj in range(3): Ks[k][di][dj] -= lr * sum( D[k][i + PADDING - di][j + PADDING - dj] for (i, j) in active if 0 <= i + PADDING - di < SIDE and 0 <= j + PADDING - dj < SIDE) return P def accuracy(P, data): return sum(1 for active, y in data if (forward(P, active)[1] >= 0.5) == (y == 1)) / len(data) print(f"{'pooling':>10}{'param':>7}{'run':>7}{'train':>8}{'test':>8}" f"{'decisions changed on shift':>28}") print(f"{'baseline':>10}{0:>7}{'-':>7}{'-':>8}{BASELINE:>8.4f}{'-':>28}") for p in (1, 2, 12): for k, seed in enumerate((SEED + 11, SEED + 12)): P = train_model(build(2, p, seed), TRAIN_ACTIVE, 25, 0.05, seed + 500) changed = sum(1 for (a1, y), (a2, _) in zip(TEST_ACTIVE, SHIFTED_ACTIVE) if (forward(P, a1)[1] >= 0.5) != (forward(P, a2)[1] >= 0.5)) print(f"{('none' if p == 1 else f'{p}x{p}') if k == 0 else '':>10}" f"{(20 + 2 * (SIDE // p) ** 2 + 1) if k == 0 else '':>7}{k + 1:>7}" f"{accuracy(P, TRAIN_ACTIVE):>8.4f}{accuracy(P, TEST_ACTIVE):>8.4f}{changed:>28d}") ``` ``` pooling param run train test decisions changed on shift baseline 0 - - 0.5850 - none 309 1 1.0000 0.5150 84 2 1.0000 0.5350 88 2x2 93 1 0.8950 0.6100 49 2 0.7900 0.6150 31 12x12 23 1 0.8250 0.8050 12 2 0.8200 0.8050 11 ``` Three numbers stand inside the table. The baseline is **0.5850**. The network without pooling memorizes the training set completely with 309 parameters and reads **0.5150** on the test set — below the baseline. The 2x2 pooled network reads **0.6100** with 93 parameters, and the globally pooled network reads **0.8050** with 23 parameters. As pooling grows the network shrinks and the held-out number rises. The last column confirms the same order with an independent measure. When the 200 test grids are shifted one cell down, the decision of the network without pooling changes in **84** examples, in the 2x2 pooled one in 49, and in the globally pooled one in **12**. The shift does not change the label. **The second run** is written in all three rows, and it speaks loudest in 2x2 pooling: the test numbers of the two runs are 0.6100 and 0.6150, that is, close to each other, but their training numbers are 0.8950 and 0.7900 and the decisions changed on shift are 49 and 31. Two runs of the same capacity give a similar number on the held-out set while opening a **difference of eighteen examples** in how the decision behaves against the shift. The held-out number reports this difference in no way. #### Summary - 2x2 max pooling reduces 144 cells to 36; 96.9 cells can be recovered on average, an uncertainty of 46.0 bits remains, and with a 4x4 window the uncertainty rises to 113.2 bits. - Pooling applied to the raw grid destroys the line information: agreement falls from 0.9380 to 0.4220, that is, below the baseline. - Average pooling preserves the active cell count exactly (13.96), while max pooling gives only a lower bound (11.78). - When the same pooling is applied to the feature map, shift invariance rises from 0.0% to 38.1%, and to 88.7% with global pooling; the price is the head input falling from 288 to 2. - With a baseline of 0.5850, the network without pooling reads 0.5150 and 0.5350 with 309 parameters, the 2x2 pooled network 0.6100 and 0.6150 with 93 parameters, and the globally pooled network 0.8050 and 0.8050 with 23 parameters. - The number of decisions that change on the shifted test set is 84, 49 and 12; the two runs of 2x2 pooling give the same test number while they diverge between 49 and 31 on this count. #### Next Step The network up to here is **two-layered**: one convolution, one pooling, one linear head. Global pooling gave the best number but threw away the position of the line entirely — this network cannot answer the question "are there two lines". The way to do that is, instead of reducing the map to zero in one go, to **stack convolution and pooling in sequence**: each layer sees a larger area of the output of the one before it. The next lesson measures the price of this stacking: as the number of layers grows, it is counted whether the gradient coming from the output reaches the first layer, how much a **skip connection** fixes this is measured, and a point where a deeper network cannot beat a shallower one is shown. ### Convolutional Network Architectures Source: https://academia.sh/en/courses/deep-learning/convolutional-network-architectures Counting what depth adds and whether the gradient reaches the first layer: in the plain stack the magnitude of the gradient reaching the first layer falls from 6.405e-01 at depth 0 to 9.269e-07 at depth 8, the skip connection holds 6.779e-01 at the same depth and the ratio is 731,349 times. With a baseline of 0.5850, the shallow network with 23 parameters gives 0.7850 and 0.8050, and the plain depth 2 network with 63 parameters gives 0.5850 in both runs, that is, the deeper network cannot beat the shallower one. The skip connection gives 0.7950 and 0.8150 at the same depth and 0.8150 and 0.8900 at depth 5; the mean difference of 0.0575 that depth adds is smaller than the difference of 0.0750 between two runs of the same network. The previous two lessons used a single convolution layer. That layer looks through a 3x3 window and sees only that much of the grid; to answer a question like "are two lines close to each other" it has to look at a wider area. The way to do this is not to enlarge the kernel but to **stack layers**: the second layer looks at 3x3 on the output of the first and, indirectly, sees a much larger area of the input. Stacking has a price, and that is the measure of this lesson. The gradient coming from the loss starts at the output and is carried back layer by layer; each layer multiplies it once more. As the stack deepens, the **gradient reaching the first layer** shrinks, and after a point the first layer is never updated. Three things will be measured: the receptive field that grows with depth, the magnitude of the gradient reaching the first layer, and the difference depth adds to the held-out number. - **AR19. The set, the split and the baseline do not change.** The fictional coverage grid is generated with the same seed; the split is 200/100/200, and the most frequent class on the test set is **0.5850**. - **AR20. The stack is fixed, and the only thing that changes is the number of blocks.** Layer 0: two 3x3 kernels, padding 1, a rectified linear unit, then 4x4 pooling — a 3x3 map per channel remains. Then $D$ **blocks**, then global pooling and a linear head. - **AR21. A block** is a 3x3 convolution applied to the map of one channel followed by a rectified linear unit; the channels do not mix. - **AR22. A skip connection** means **adding the input of the block to its output**. The block no longer changes the input, it learns a **correction** to the input; the gradient goes backward by two paths, and one of them is not shrunk by any factor. - **AR23. The receptive field** is the side length of the square formed by the input cells that can affect the value of an output unit. Since the pooling window is 4 and the growth per block is 2 windows, the side is $4(2D+1)+2$. - **AR24. The gradient measurement is taken at initialization** and averaged over 80 training grids; the measured number is the sum of the absolute values of the kernel gradients of the first layer. The question asked is **whether training can start**. - **AR25. Initialization is fixed** and the gain is not scaled: all kernel weights are drawn from the same range, so that what happens with depth can be seen. - **AR26. No library is called;** the stack, the forward pass, backpropagation and the training loop are written inside the lesson and are **a model**. No architecture is named after a person or an institution; it is named by its structure. - **AR27. Every network is run twice;** in the second run only the initial values change. #### The Stack and the Receptive Field ```python # architecture.py — MODEL: the FICTIONAL coverage grid of 01 is generated with the same seed. # Stack: convolution (3x3, padding 1, two kernels) -> rectified -> pooling 4x4 # -> 3x3 map; then D blocks (3x3 convolution + rectified on each channel, # in the skip version # the input of the block is added to the output); then global pooling and linear head. import math SEED, M32, SIDE = 20260218, 0xFFFFFFFF, 12 DENSITY, TRACE_RATE, RUN_LENGTH = 0.09, 0.42, 3 PADDING, POOL, MAP_SIDE, CHANNELS = 1, 4, 3, 2 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def grid(subscriber_id): r = rng(SEED + 300000 + subscriber_id) g = [[1 if r() < DENSITY else 0 for _ in range(SIDE)] for _ in range(SIDE)] trace = 1 if r() < TRACE_RATE else 0 if trace: horizontal, a, b = r() < 0.5, int(r() * SIDE), int(r() * (SIDE - RUN_LENGTH + 1)) for k in range(RUN_LENGTH): if horizontal: g[a][b + k] = 1 else: g[b + k][a] = 1 return g, trace def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s RAW = [grid(i) for i in range(500)] DATA = [RAW[i] for i in shuffle(500, SEED + 90000)] TRAIN_ACTIVE = [([(i, j) for i in range(SIDE) for j in range(SIDE) if g[i][j]], y) for g, y in DATA[:200]] TEST_ACTIVE = [([(i, j) for i in range(SIDE) for j in range(SIDE) if g[i][j]], y) for g, y in DATA[300:]] BASELINE = max(sum(y for _, y in TEST_ACTIVE), len(TEST_ACTIVE) - sum(y for _, y in TEST_ACTIVE)) / len(TEST_ACTIVE) def build(D, seed, a=1 / 3): r = rng(seed) return [[[[[(r() * 2 - 1) * a for _ in range(3)] for _ in range(3)] for _ in range(CHANNELS)] for _ in range(D + 1)], [[0.0] * CHANNELS for _ in range(D + 1)], [(r() * 2 - 1) * 0.5 for _ in range(CHANNELS)], [0.0], D] def forward(P, active, skip): Ks, bs, v, c, D = P x, pooled = [], [] for k in range(CHANNELS): # layer 0: convolution on sparse input m = [[bs[0][k]] * SIDE for _ in range(SIDE)] for (i, j) in active: for di in range(3): for dj in range(3): p, q = i + PADDING - di, j + PADDING - dj if 0 <= p < SIDE and 0 <= q < SIDE: m[p][q] += Ks[0][k][di][dj] ch_rec, ch_map = [], [[0.0] * MAP_SIDE for _ in range(MAP_SIDE)] for i in range(MAP_SIDE): for j in range(MAP_SIDE): best, bp, bq = None, 0, 0 for a in range(POOL): for b in range(POOL): u = m[i * POOL + a][j * POOL + b] if best is None or u > best: best, bp, bq = u, i * POOL + a, j * POOL + b ch_map[i][j] = max(0.0, best) ch_rec.append((bp, bq, best)) x.append(ch_map) pooled.append(ch_rec) stages, pre_acts = [x], [] for t in range(1, D + 1): # D blocks prev, new, pre = stages[-1], [], [] for k in range(CHANNELS): z = [[bs[t][k]] * MAP_SIDE for _ in range(MAP_SIDE)] for p in range(MAP_SIDE): for q in range(MAP_SIDE): s = bs[t][k] for di in range(3): i2 = p + di - PADDING if 0 <= i2 < MAP_SIDE: xi = prev[k][i2] for dj in range(3): j2 = q + dj - PADDING if 0 <= j2 < MAP_SIDE: s += Ks[t][k][di][dj] * xi[j2] z[p][q] = s pre.append(z) new.append([[max(0.0, z[i][j]) + (prev[k][i][j] if skip else 0.0) for j in range(MAP_SIDE)] for i in range(MAP_SIDE)]) pre_acts.append(pre) stages.append(new) last, h, arg = stages[-1], [], [] for k in range(CHANNELS): # global pooling best, bp, bq = last[k][0][0], 0, 0 for i in range(MAP_SIDE): for j in range(MAP_SIDE): if last[k][i][j] > best: best, bp, bq = last[k][i][j], i, j h.append(best) arg.append((bp, bq)) z = c[0] + sum(v[k] * h[k] for k in range(CHANNELS)) return 1 / (1 + math.exp(-max(-30, min(30, z)))), (pooled, stages, pre_acts, h, arg) print(f"most frequent class on the test set {BASELINE:.4f}; training {len(TRAIN_ACTIVE)} grids") print(f"{'depth':>9}{'block':>6}{'parameters':>11}{'receptive field':>17}" f"{'covered grid':>17}") for D in (0, 1, 2, 3, 5, 8): area = POOL * (2 * D + 1) + 2 print(f"{D:>9}{D:>6}{20 * (D + 1) + 3:>11}{f'{area}x{area}':>17}" f"{min(area, SIDE) ** 2 / (SIDE * SIDE):>16.1%}") ``` ``` most frequent class on the test set 0.5850; training 200 grids depth block parameters receptive field covered grid 0 0 23 6x6 25.0% 1 1 43 14x14 100.0% 2 2 63 22x22 100.0% 3 3 83 30x30 100.0% 5 5 123 46x46 100.0% 8 8 183 70x70 100.0% ``` The receptive field grows linearly with depth and **saturates in one block**: at depth 0 an output unit sees 25% of the grid, while at depth 1 the side rises to 14 cells and exceeds the 12-cell grid. The following blocks **bring no new input cell**, they only add parameters. The gain expected from depth cannot come from the receptive field here. #### Does the Gradient Reach the First Layer ```python # gradient.py — backpropagation is written by hand. What is measured: how much of the gradient # from the output reaches the kernels of the FIRST layer. Measured at INITIALIZATION. def backward(P, active, cache, d, skip): Ks, bs, v, c, D = P pooled, stages, pre_acts, h, arg = cache dv = [d * h[k] for k in range(CHANNELS)] delta = [[[0.0] * MAP_SIDE for _ in range(MAP_SIDE)] for _ in range(CHANNELS)] for k in range(CHANNELS): bp, bq = arg[k] delta[k][bp][bq] = d * v[k] dK = [[[[0.0] * 3 for _ in range(3)] for _ in range(CHANNELS)] for _ in range(D + 1)] db = [[0.0] * CHANNELS for _ in range(D + 1)] for t in range(D, 0, -1): prev, pre2 = stages[t - 1], pre_acts[t - 1] new = [[[delta[k][i][j] if skip else 0.0 for j in range(MAP_SIDE)] for i in range(MAP_SIDE)] for k in range(CHANNELS)] for k in range(CHANNELS): z = pre2[k] for p in range(MAP_SIDE): for q in range(MAP_SIDE): g = delta[k][p][q] if z[p][q] > 0 else 0.0 if g == 0.0: continue db[t][k] += g for di in range(3): i2 = p + di - PADDING if not 0 <= i2 < MAP_SIDE: continue for dj in range(3): j2 = q + dj - PADDING if 0 <= j2 < MAP_SIDE: dK[t][k][di][dj] += g * prev[k][i2][j2] new[k][i2][j2] += g * Ks[t][k][di][dj] delta = new for k in range(CHANNELS): d0 = [[0.0] * SIDE for _ in range(SIDE)] t2 = 0 for i in range(MAP_SIDE): for j in range(MAP_SIDE): bp, bq, best = pooled[k][t2] t2 += 1 if best > 0 and delta[k][i][j] != 0.0: d0[bp][bq] += delta[k][i][j] for di in range(3): for dj in range(3): dK[0][k][di][dj] = sum( d0[i + PADDING - di][j + PADDING - dj] for (i, j) in active if 0 <= i + PADDING - di < SIDE and 0 <= j + PADDING - dj < SIDE) db[0][k] = sum(sum(s) for s in d0) return dK, db, dv print(f"{'depth':>9}{'plain first layer':>19}{'skip first layer':>21}" f"{'ratio':>12}") for D in (0, 1, 2, 3, 5, 8): measured = [] for skip in (False, True): P, total = build(D, SEED + 11), 0.0 for active, y in TRAIN_ACTIVE[:80]: out, cache = forward(P, active, skip) dK = backward(P, active, cache, out - y, skip)[0] total += sum(abs(u) for k in range(CHANNELS) for s in dK[0][k] for u in s) measured.append(total / 80) print(f"{D:>9}{measured[0]:>19.3e}{measured[1]:>21.3e}{measured[1] / measured[0]:>12.1f}") ``` ``` depth plain first layer skip first layer ratio 0 6.405e-01 6.405e-01 1.0 1 2.393e-02 8.261e-01 34.5 2 2.729e-03 1.925e-01 70.6 3 2.457e-03 7.907e-01 321.8 5 1.984e-04 1.043e+00 5256.1 8 9.269e-07 6.779e-01 731349.2 ``` In the plain stack the gradient reaching the first layer is $6.405 \times 10^{-1}$ at depth 0. One block reduces it to $2.393 \times 10^{-2}$, five blocks to $1.984 \times 10^{-4}$, eight blocks to $9.269 \times 10^{-7}$ — about **seven hundred thousand times**. This is not a training defect but the product itself: each block multiplies the gradient going backward by its weights once more, and if the factors are less than one the product shrinks exponentially with depth. The skip connection column stays between $1.9 \times 10^{-1}$ and $1.0$ at the same depths; it does **not decrease** with depth. The reason is written in AR22: since the output of the block contains its input, one copy of the gradient going backward is not multiplied by any kernel and is carried as it is. The ratio column is the size of the correction — 70.6 times at depth 2, **731,349 times** at depth 8. #### What Depth Adds ```python # train.py — MODEL: same data, same epoch budget, same learning rate. # Only the depth and the presence of the skip connection change. def train_model(P, data, epochs, lr, seed, skip): r, n = rng(seed), len(data) Ks, bs, v, c, D = P for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: active, y = data[idx] out, cache = forward(P, active, skip) d = out - y dK, db, dv = backward(P, active, cache, d, skip) for t in range(D + 1): for k in range(CHANNELS): bs[t][k] -= lr * db[t][k] for di in range(3): for dj in range(3): Ks[t][k][di][dj] -= lr * dK[t][k][di][dj] for k in range(CHANNELS): v[k] -= lr * dv[k] c[0] -= lr * d return P def accuracy(P, data, skip): return sum(1 for active, y in data if (forward(P, active, skip)[0] >= 0.5) == (y == 1)) / len(data) print(f"{'depth':>9}{'connection':>12}{'param':>7}{'run':>7}{'train':>8}" f"{'test':>8}{'vs baseline':>13}") print(f"{'baseline':>9}{'-':>12}{0:>7}{'-':>7}{'-':>8}{BASELINE:>8.4f}{'0.0000':>13}") for D, skip in ((0, False), (2, False), (2, True), (5, False), (5, True)): for k, seed in enumerate((SEED + 11, SEED + 12)): P = train_model(build(D, seed), TRAIN_ACTIVE, 25, 0.05, seed + 500, skip) acc = accuracy(P, TEST_ACTIVE, skip) print(f"{D if k == 0 else '':>9}" f"{('skip' if skip else 'plain') if k == 0 else '':>12}" f"{(20 * (D + 1) + 3) if k == 0 else '':>7}{k + 1:>7}" f"{accuracy(P, TRAIN_ACTIVE, skip):>8.4f}{acc:>8.4f}{acc - BASELINE:>+13.4f}") ``` ``` depth connection param run train test vs baseline baseline - 0 - - 0.5850 0.0000 0 plain 23 1 0.8100 0.7850 +0.2000 2 0.8200 0.8050 +0.2200 2 plain 63 1 0.5950 0.5850 +0.0000 2 0.5950 0.5850 +0.0000 2 skip 63 1 0.8350 0.7950 +0.2100 2 0.8150 0.8150 +0.2300 5 plain 123 1 0.5950 0.5850 +0.0000 2 0.5950 0.5850 +0.0000 5 skip 123 1 0.8400 0.8150 +0.2300 2 0.9000 0.8900 +0.3050 ``` The two rows in the middle of the table are the form the second claim of this course takes in this lesson, and it is not hidden: **a deeper network could not beat a shallower one**. The shallow network with 23 parameters reads 0.7850 and 0.8050; the plain network with 63 parameters, three times deeper, reads **0.5850** in both runs, that is, exactly the baseline. The plain network with five blocks is in the same place with 123 parameters. These networks stay at 0.5950 on the training set too; they have not learned, nor have they memorized. The previous table writes the reason: the gradients reaching the first layer are $2.7 \times 10^{-3}$ and $2.0 \times 10^{-4}$, that is, the first layer has stayed where it was initialized and everything after it is built on a random map. The skip connection makes the same depth work: 0.7950 and 0.8150 at depth 2, 0.8150 and **0.8900** at depth 5. The structure did not change, the parameter count did not change, the only difference is that the input of the block is added to its output. The last comparison must be read carefully. The mean of the two runs of the shallow network is 0.7950, that of the five-block skip network 0.8525; the apparent difference depth adds is **0.0575**. But the two runs of the five-block network lie between 0.8150 and 0.8900, a range of **0.0750**. The difference depth adds is **smaller** than the difference the same network gives with only its initial values changing; a comparison made with a single run could have shown this network as above the shallow network by both 0.0850 and 0.0100. #### Summary - The receptive field grows linearly with depth and saturates in one block on this grid: 25% at depth 0, 100% at depth 1. The following blocks bring no new input cell, they only add parameters. - In the plain stack the gradient reaching the first layer is $6.405 \times 10^{-1}$ at depth 0 and $9.269 \times 10^{-7}$ at depth 8; the shrinking comes from the product itself. - The skip connection holds the gradient between $1.9 \times 10^{-1}$ and $1.0$ at the same depths; the correction ratio is 70.6 times at depth 2 and 731,349 times at depth 8. - With a baseline of 0.5850, the shallow network with 23 parameters reads 0.7850 and 0.8050; the plain deep networks with 63 and 123 parameters stay at 0.5850 in both runs — **a deeper network cannot beat a shallower one**. - The skip connection gives 0.7950 and 0.8150 at depth 2 and 0.8150 and 0.8900 at depth 5, with the parameter count unchanged. - The mean difference depth adds is 0.0575, and the difference between two runs of the same network is 0.0750; as long as the second is large, the first cannot be measured with a single run. #### Next Step For three lessons the structure of the input was **spatial** and the whole grid was given to the network at once; there was no question of which cell came first. The meter readings of a subscriber are not like that: the readings form a **sequence**, the place in the sequence is meaningful, and the number of readings is not the same for every subscriber. A kernel looking through a fixed-size window cannot see the past that falls outside the window. The next lesson builds a structure that reads the input **step by step** from start to end and accumulates what it sees in a **hidden state**. What will be measured is what that state carries: how much of the information at the start of the sequence remains in the state as the sequence grows longer. ### Recurrent Networks Source: https://academia.sh/en/courses/deep-learning/recurrent-networks Counting how much information the hidden state carries as the sequence grows longer: when only the first step of the same sequence is changed, the difference in the last hidden state is 0.92844 at length 2 and falls to 0.00913 at length 12, a factor of 0.62 per step. The accuracy of reading the marker at the first step back from the last state falls from 1.0000 to 0.5300, that is, to chance level. The same network with 61 parameters beats the baseline by 0.2300 and 0.2800 at length 2, by 0.2100 and 0.1700 at length 8 and by 0.1850 and 0.1850 at length 10, while at length 12 it beats it by only 0.0250 and 0.0050: the dependency is carried over ten steps and is not carried over twelve steps. For three lessons the input was a grid and was given to the network all at once. The meter readings of a subscriber, however, form a **sequence**: the readings arrive period by period, the place in the sequence is meaningful, and the number of readings per subscriber does not have to be the same. A kernel looking through a fixed-size window does not fit this data — the past that falls outside the window is invisible. A **recurrent network** reads the sequence from start to end step by step and accumulates what it has seen up to that point in a **hidden state**. The same weights are used at every step; this is the counterpart on the time axis of the weight sharing that convolution does in space, and it makes the parameter count independent of the sequence length. The measure of this lesson is this: how much of a piece of information at the **start** of the sequence remains in the hidden state as the sequence grows longer. - **AR28. The reading sequence is fictional.** Twelve periods of meter readings are generated for each subscriber; seed 20260218. The length **does not exceed 12**, and the sequence is a separate setup, independent of the grid in the first three lessons of this topic. - **AR29. The marker is at the start of the sequence.** In about half of the subscribers **the reading of the first period is zero** — a meter replacement. A zero reading is seen only in the first period, so the marker is unique and its place is known. - **AR30. The label is the joint satisfaction of two conditions:** a zero reading in the first period **and** the reading of the last period above the tariff tier of 25 m³. For the network to decide at the last step, it has to carry the information at the start of the sequence all the way to the end. - **AR31. The input of each step is two numbers:** the reading divided by 40 m³ and the zero-reading indicator. - **AR32. The label depends on the sequence length,** because the "last period" changes with the length. For this reason **the baseline is computed separately for each length** and comparisons are made as a difference from the baseline. - **AR33. The network is fixed:** a hidden state of 6 units, hyperbolic tangent, and a sigmoid output read from the last hidden state. The parameter count is **61** and does not change whatever the sequence length. - **AR34. The information measurement is made on the untrained network.** Only the first step of the same sequence is changed and the difference between the two last hidden states is measured. What is asked is what the structure **can carry**, not what training has achieved. - **AR35. No library is called;** the forward pass, backpropagation through time and the training loop are written inside the lesson and are **a model**. - **AR36. Every network is run twice;** in the second run only the initial values change. #### The Sequence and the Hidden State ```python # sequence.py — MODEL. The reading sequence is FICTIONAL: for each subscriber twelve periods of # meter readings are generated. In some subscribers the reading of the first period is ZERO # (meter replacement); this is the MARKER placed at the start of the sequence. The label is the # presence of the marker AND the last period exceeding the tariff tier TOGETHER. import math SEED, M32 = 20260218, 0xFFFFFFFF LENGTH, TIER, SCALE, HIDDEN = 12, 25.0, 40.0, 6 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def reading_sequence(subscriber_id): r = rng(SEED + 400000 + subscriber_id) base = 23.0 + 8.0 * r() d = [math.floor(base * math.exp((r() + r() + r() - 1.5) * 0.62) * 100 + 0.5) / 100 for _ in range(LENGTH)] if r() < 0.5: d[0] = 0.0 # marker return d def cut(d, T): """First T periods; two numbers per step input, the label from both conditions together.""" x = [[v / SCALE, 1.0 if v == 0.0 else 0.0] for v in d[:T]] return x, 1 if (d[0] == 0.0 and d[T - 1] > TIER) else 0 def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s RAW = [reading_sequence(i) for i in range(500)] SEQS = [RAW[i] for i in shuffle(500, SEED + 95000)] TRAIN, TEST = SEQS[:200], SEQS[300:] def baseline_for(T): p = sum(cut(d, T)[1] for d in TEST) return max(p, len(TEST) - p) / len(TEST) def build(seed, H=HIDDEN): r = rng(seed) a1, a2 = (1 / 2) ** 0.5, (1 / H) ** 0.5 return [[[(r() * 2 - 1) * a1 for _ in range(2)] for _ in range(H)], [[(r() * 2 - 1) * a2 for _ in range(H)] for _ in range(H)], [0.0] * H, [(r() * 2 - 1) * a2 for _ in range(H)], [0.0]] def forward(P, x): """h_t = tanh(Wx x_t + Wh h_(t-1) + b); the output is read from the last hidden state.""" Wx, Wh, b, v, c = P H = len(b) h, states = [0.0] * H, [] for t in range(len(x)): h = [math.tanh(b[k] + Wx[k][0] * x[t][0] + Wx[k][1] * x[t][1] + sum(Wh[k][j] * h[j] for j in range(H))) for k in range(H)] states.append(h) z = c[0] + sum(v[k] * h[k] for k in range(H)) return 1 / (1 + math.exp(-max(-30, min(30, z)))), states example = next(d for d in SEQS if d[0] == 0.0 and d[LENGTH - 1] > TIER) print("example sequence (m3):", " ".join(f"{v:.2f}" for v in example)) print(f"marker rate {sum(1 for d in SEQS if d[0] == 0.0) / len(SEQS):.3f}; " f"split {len(TRAIN)}/100/{len(TEST)}; hidden state {HIDDEN} units; " f"parameters {HIDDEN * 2 + HIDDEN * HIDDEN + HIDDEN + HIDDEN + 1}") print(f"{'T':>3}{'positive':>9}{'baseline':>15}") for T in (2, 4, 6, 8, 10, 12): print(f"{T:>3}{sum(cut(d, T)[1] for d in TEST):>9}{baseline_for(T):>15.4f}") ``` ``` example sequence (m3): 0.00 26.80 37.98 20.95 35.18 49.78 24.15 22.38 47.64 35.14 26.95 25.18 marker rate 0.514; split 200/100/200; hidden state 6 units; parameters 61 T positive baseline 2 56 0.7200 4 56 0.7200 6 63 0.6850 8 51 0.7450 10 60 0.7000 12 58 0.7100 ``` In the example sequence the first reading is zero and the last reading, 25.18 m³, is above the tariff tier; the label is 1. The ten periods in between say nothing for the decision — the only information the network has to carry is the zero at the start. The parameter count is **61** and is the same for all six lengths in the table. The same weights are reused at every step; however long a sequence grows, no new weight is needed. This is the counterpart on the time axis of what convolution does in space, and it is something a fully connected layer cannot do: a fully connected layer has to know the length of the input beforehand. The baselines wander between 0.6850 and 0.7450 with the length, because the second condition of the label is defined by the "last period" and the last period changes with the length. For this reason the comparison across lengths is made not with raw accuracy but with the **difference from the baseline**. #### What the Hidden State Carries ```python # trace.py — the information the hidden state carries. ONLY the first step of the same sequence # is changed (marked / unmarked) and the difference of the last hidden states is measured. # The measurement is made on the UNTRAINED network: what is asked is what the structure can carry. def pairs(d, T): x, _ = cut(d, T) a = [s[:] for s in x] a[0] = [0.0, 1.0] b = [s[:] for s in x] b[0] = [d[1] / SCALE, 0.0] return a, b def readback(P, T): """Nearest-center reader: can the marker be read back from h_T.""" hs = [(forward(P, cut(d, T)[0])[1][-1], 1 if d[0] == 0.0 else 0) for d in SEQS] train, test = hs[:200], hs[300:] centers, count = [[0.0] * HIDDEN, [0.0] * HIDDEN], [0, 0] for h, s in train: count[s] += 1 for k in range(HIDDEN): centers[s][k] += h[k] for s in (0, 1): centers[s] = [u / count[s] for u in centers[s]] mean = [(centers[0][k] * count[0] + centers[1][k] * count[1]) / len(train) for k in range(HIDDEN)] sd = [max(1e-9, (sum((h[k] - mean[k]) ** 2 for h, _ in train) / len(train)) ** 0.5) for k in range(HIDDEN)] def distance(h, s): return sum(((h[k] - centers[s][k]) / sd[k]) ** 2 for k in range(HIDDEN)) return sum(1 for h, s in test if (distance(h, 1) < distance(h, 0)) == (s == 1)) / len(test) print(f"{'T':>3}{'h diff k1':>12}{'h diff k2':>12}{'per step k1':>16}" f"{'readback k1':>15}{'readback k2':>15}") previous = [None, None] for T in (2, 4, 6, 8, 10, 12): row = [] for P in (build(SEED + 11), build(SEED + 12)): f = 0.0 for d in SEQS: a, b = pairs(d, T) ia, ib = forward(P, a)[1][-1], forward(P, b)[1][-1] f += sum(abs(ia[k] - ib[k]) for k in range(HIDDEN)) row.append((f / len(SEQS), readback(P, T))) ratio = "-" if previous[0] is None else f"{(row[0][0] / previous[0]) ** 0.5:.3f}" previous[0] = row[0][0] print(f"{T:>3}{row[0][0]:>12.5f}{row[1][0]:>12.5f}{ratio:>16}" f"{row[0][1]:>15.4f}{row[1][1]:>15.4f}") ``` ``` T h diff k1 h diff k2 per step k1 readback k1 readback k2 2 0.92844 1.39970 - 1.0000 0.9750 4 0.34417 0.24972 0.609 0.7850 0.9350 6 0.15338 0.06435 0.668 0.8000 0.6350 8 0.05986 0.01560 0.625 0.5900 0.5500 10 0.02362 0.00333 0.628 0.4600 0.4600 12 0.00913 0.00067 0.622 0.5300 0.4750 ``` The first column is the lesson's number. When only the first step of the same sequence is changed, the difference remaining in the last hidden state is **0.92844** at length 2 and **0.00913** at length 12 — less than 1% of it. The fourth column gives the form of the shrinking: the per-step factor lies between 0.609 and 0.668, that is, an exponential decrease with **a constant factor**. The hidden state does not decide to forget the information; at every step it is multiplied by the same matrix, and since the magnitude of the factor is less than one, the trace is erased by itself. The second run shows the same form at a different speed: at length 12 the difference is 0.00067, one fourteenth of the first run's. Only the initial values changed; the forgetting speed is not a constant of the architecture but a result of initialization. The last two columns write what this means. At length 2 the marker is read back from the last hidden state with an accuracy of **1.0000**. At length 8 it falls to 0.5900, at length 10 to 0.4600 — since the marker rate is 0.514, this is chance level. The 0.5300 at length 12 is not a recovery, but measurement wobble at the same level. The information is **not** in the hidden state; no reader can bring it back. #### How Far Back It Is Carried ```python # train_model.py — MODEL: backpropagation through time is written by hand. Same network, same # epoch budget, the only thing that changes is the sequence length. def train_model(P, data, epochs, lr, seed): Wx, Wh, b, v, c = P H, r, n = len(b), rng(seed), len(data) for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: x, y = data[idx] p, states = forward(P, x) d = p - y dh = [d * v[k] for k in range(H)] for k in range(H): v[k] -= lr * d * states[-1][k] c[0] -= lr * d for t in range(len(x) - 1, -1, -1): previous = states[t - 1] if t > 0 else [0.0] * H new = [0.0] * H for k in range(H): g = dh[k] * (1 - states[t][k] ** 2) b[k] -= lr * g Wx[k][0] -= lr * g * x[t][0] Wx[k][1] -= lr * g * x[t][1] for j in range(H): new[j] += g * Wh[k][j] Wh[k][j] -= lr * g * previous[j] dh = new return P def accuracy(P, data): return sum(1 for x, y in data if (forward(P, x)[0] >= 0.5) == (y == 1)) / len(data) print(f"{'T':>3}{'baseline':>10}{'run 1':>9}{'run 2':>9}{'diff 1':>9}{'diff 2':>9}" f"{'run range':>15}") for T in (2, 4, 6, 8, 10, 12): tr = [cut(d, T) for d in TRAIN] te = [cut(d, T) for d in TEST] tb, s = baseline_for(T), [] for seed in (SEED + 11, SEED + 12): P = build(seed) train_model(P, tr, 30, 0.05, seed + 500) s.append(accuracy(P, te)) print(f"{T:>3}{tb:>10.4f}{s[0]:>9.4f}{s[1]:>9.4f}{s[0] - tb:>+9.4f}" f"{s[1] - tb:>+9.4f}{abs(s[0] - s[1]):>15.4f}") ``` ``` T baseline run 1 run 2 diff 1 diff 2 run range 2 0.7200 0.9500 1.0000 +0.2300 +0.2800 0.0500 4 0.7200 0.9550 0.9550 +0.2350 +0.2350 0.0000 6 0.6850 0.9350 0.9550 +0.2500 +0.2700 0.0200 8 0.7450 0.9550 0.9150 +0.2100 +0.1700 0.0400 10 0.7000 0.8850 0.8850 +0.1850 +0.1850 0.0000 12 0.7100 0.7350 0.7150 +0.0250 +0.0050 0.0200 ``` Training confirms what the measurement said, but not at exactly the same place. From length 2 to 10 the network beats the baseline by between 0.1700 and 0.2800. At length 12 the difference drops to **0.0250** and **0.0050**: the network now says the same thing as the baseline, that is, it does not use the marker at the start at all and has turned into a rule looking only at the reading of the last period. The untrained measurement had said that the marker could not be read back at length 10 already; the trained network, however, still produces a difference of 0.1850 at length 10. This is not a contradiction: training moves the weights away from their initial values and tunes the recurrent matrix to carry as much information as it allows. What is gained is two steps. At twelve even those two steps are not enough, because the problem is not of the kind to be solved by tuning the weights: carrying is done **by multiplication**, and enlarging the factor a little leads to an explosion on a long sequence, shrinking it a little leads to fading. The **run range** column writes its own warning into the table. At length 8 the two runs differ by 0.0400; this is above the difference between length 8 and length 10 (0.0250). The difference added by the length and the difference added by the initial values are of the same order of magnitude, and an answer to "up to how many steps does it work" measured with a single run is therefore not reliable. #### Summary - The reading sequence is fictional: twelve periods of readings, the zero reading of the first period as the marker, and the label as the joint satisfaction of the marker and the last period exceeding the tariff tier. - The recurrent network uses the same weights at every step; the parameter count is 61 and is independent of the sequence length. - The difference that the marker at the first step leaves in the last hidden state is 0.92844 at length 2 and 0.00913 at length 12; the per-step factor is constant between 0.609 and 0.668. - The accuracy of reading the marker back from the last state falls from 1.0000 to 0.4600 at length 10, that is, to chance level; in the second run the forgetting is even faster (at length 12 the difference is 0.00067). - The trained network beats the baseline by 0.1700–0.2800 up to length 10 and by only 0.0250 and 0.0050 at length 12. - The run range at length 8, 0.0400, is larger than the difference between length 8 and 10; the question "up to how many steps" cannot be answered with a single run. #### Next Step The hidden state does not **choose** to hold the information; at every step it is multiplied by the same matrix, and the magnitude of the factor determines what remains. The same factor affects the new reading and the marker at the start equally, yet the lifetimes of these two pieces of information should not be the same: the reading is unnecessary one step later, the marker is needed until the end of the sequence. The next lesson adds two decisions to the state — what to **erase** and what to **write** — and builds units that make these decisions by looking at the input. The measure stays the same and is compared directly: the plain recurrent network and the network with gated units are placed side by side on the same task, at the same lengths, each with two runs, and how many steps back the dependency can be carried is counted. ### Gated Units Source: https://academia.sh/en/courses/deep-learning/gated-units Counting how many steps back a long dependency can be carried: in the untrained network the share of the marker at the first step that remains in the last hidden state is 0.98% and 0.05% in the plain network with 6 units, 0.71% and 0.05% in the plain network with 9 units, and 9.34% and 5.63% in the network with gated units. With the same epoch budget all three structures beat the baseline by 0.1700–0.2800 up to length 8; at length 12 the plain network with 61 parameters drops to 0.0250 and 0.0050, while the gated network with 115 parameters gives 0.2650 and 0.2050 and the wide plain network with 118 parameters gives 0.2700 and 0.1700. The advantage of the gated structure over the equal-budget plain network cannot be measured at twelve steps, because the difference of 0.1000 between the two runs of the wide plain network is larger than the difference between the two structures. The hidden state of the previous lesson did not **choose** to hold the information. At every step it was multiplied by the same matrix, the magnitude of the factor determined what would remain, and the measured factor was 0.62 per step. The same factor was applied to both the new reading and the marker at the start, though the lifetimes of the two should differ: the reading is unnecessary one step later, the marker is needed until the end of the sequence. The **gated unit** adds two separate computations to the state: how much of it is to be **retained** and what is to be **written** in its place. Both are computed from the input of that step and the state so far, that is, the unit decides separately at every step. The measure here: how many steps back does this decision carry the dependency, and does the gain come from the gate. - **AR37. The set and the split do not change.** The fictional reading sequence of the previous lesson is generated with the same seed: twelve periods, the zero reading of the first period as the marker, and the label as the joint satisfaction of the marker and the last period exceeding the tariff tier. The split is 200/100/200, and the baseline is separate for each length. - **AR38. The gated unit is built from two blocks.** The **update gate** $u_t$ has a sigmoid output and gives a number between 0 and 1 for each unit. The **candidate value** $a_t$ has a hyperbolic tangent output. The new state is a mixture of the two: $h_t = u_t h_{t-1} + (1 - u_t) a_t$. - **AR39. The path the gate opens is selective, not multiplicative.** If $u_t$ is close to one, the old state passes almost as it is; if it is close to zero, the candidate value is written. In the plain unit there is no such option, every step applies the same transformation. - **AR40. The bias of the update gate is initialized positive** (value 1). The gate is open at the start and training learns to close it; initialization in the opposite direction requires waiting for the gate to open before learning can begin. - **AR41. The parameter counts are compared.** Because the gated unit carries two blocks, it holds about twice the parameters of the plain unit at the same number of hidden units. For this reason an **equal-budget** plain network is also put in the table: the plain network with 9 units carries about the same parameters as the gated network with 6 units. - **AR42. The measure has two layers.** The first is the share of the marker remaining in the last hidden state of the **untrained** network — what the structure can carry. The second is the difference of the trained network from the baseline — what it carries. - **AR43. The three structures are trained with the same budget:** 30 epochs, learning rate 0.05, the same split, the same order generator. The numbers of the plain network with 6 units are identical to the previous lesson. - **AR44. No library is called;** the gated unit, backpropagation through time and the training loop are written inside the lesson and are **a model**. The structure is referred to by its name, never abbreviated. - **AR45. Every network is run twice;** in the second run only the initial values change. #### The Gated Unit ```python # gated.py — MODEL. The FICTIONAL reading sequence of 04 is generated with the same seed. # Plain: h_t = tanh(...). Gated: h_t = u_t * h_(t-1) + (1 - u_t) * a_t. import math SEED, M32 = 20260218, 0xFFFFFFFF LENGTH, TIER, SCALE, HIDDEN = 12, 25.0, 40.0, 6 def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def reading_sequence(subscriber_id): r = rng(SEED + 400000 + subscriber_id) base = 23.0 + 8.0 * r() d = [math.floor(base * math.exp((r() + r() + r() - 1.5) * 0.62) * 100 + 0.5) / 100 for _ in range(LENGTH)] if r() < 0.5: d[0] = 0.0 return d def cut(d, T): return ([[v / SCALE, 1.0 if v == 0.0 else 0.0] for v in d[:T]], 1 if (d[0] == 0.0 and d[T - 1] > TIER) else 0) def shuffle(n, seed): r, s = rng(seed), list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s SEQS = [reading_sequence(i) for i in shuffle(500, SEED + 95000)] TRAIN, TEST = SEQS[:200], SEQS[300:] def baseline_for(T): p = sum(cut(d, T)[1] for d in TEST) return max(p, len(TEST) - p) / len(TEST) def sigmoid(z): return 1 / (1 + math.exp(-max(-30, min(30, z)))) def build(seed, H=HIDDEN): r = rng(seed) return [[[(r() * 2 - 1) * (1 / 2) ** 0.5 for _ in range(2)] for _ in range(H)], [[(r() * 2 - 1) * (1 / H) ** 0.5 for _ in range(H)] for _ in range(H)], [0.0] * H, [(r() * 2 - 1) * (1 / H) ** 0.5 for _ in range(H)], [0.0]] def forward(P, x): Wx, Wh, b, v, c = P H = len(b) h, states = [0.0] * H, [] for t in range(len(x)): h = [math.tanh(b[k] + Wx[k][0] * x[t][0] + Wx[k][1] * x[t][1] + sum(Wh[k][j] * h[j] for j in range(H))) for k in range(H)] states.append(h) return sigmoid(c[0] + sum(v[k] * h[k] for k in range(H))), states def build_gated(seed, H=HIDDEN, gate_bias=1.0): r = rng(seed) def block(b0): return [[[(r() * 2 - 1) * (1 / 2) ** 0.5 for _ in range(2)] for _ in range(H)], [[(r() * 2 - 1) * (1 / H) ** 0.5 for _ in range(H)] for _ in range(H)], [b0] * H] return [block(gate_bias), block(0.0), [(r() * 2 - 1) * (1 / H) ** 0.5 for _ in range(H)], [0.0], H] def forward_gated(P, x): U, A, v, c, H = P h, states = [0.0] * H, [] for t in range(len(x)): u = [sigmoid(U[2][k] + U[0][k][0] * x[t][0] + U[0][k][1] * x[t][1] + sum(U[1][k][j] * h[j] for j in range(H))) for k in range(H)] a = [math.tanh(A[2][k] + A[0][k][0] * x[t][0] + A[0][k][1] * x[t][1] + sum(A[1][k][j] * h[j] for j in range(H))) for k in range(H)] new = [u[k] * h[k] + (1 - u[k]) * a[k] for k in range(H)] states.append((h, u, a, new)) h = new return sigmoid(c[0] + sum(v[k] * h[k] for k in range(H))), states def params_plain(H): return 2 * H + H * H + H + H + 1 def params_gated(H): return 2 * (2 * H + H * H + H) + H + 1 print(f"{'structure':<22}{'hidden units':>12}{'parameters':>11}") for name, H, f in (("plain recurrent", HIDDEN, params_plain), ("plain recurrent", 9, params_plain), ("gated unit", HIDDEN, params_gated)): print(f"{name:<22}{H:>12}{f(H):>11}") print(f"\n{'T':>3}{'baseline':>15}") for T in (2, 4, 8, 12): print(f"{T:>3}{baseline_for(T):>15.4f}") ``` ``` structure hidden units parameters plain recurrent 6 61 plain recurrent 9 118 gated unit 6 115 T baseline 2 0.7200 4 0.7200 8 0.7450 12 0.7100 ``` The mixing relation of the gated unit is a single line, and what matters is the **form** of that line: the old state is multiplied by a coefficient computed from the input of that step, and a candidate is added on top. In the plain unit the remaining share was determined by the magnitude of the weights and was the same for all steps. The parameter table sets the rule of the comparison. With the same 6 units the gated network holds **115** parameters and the plain network **61**; comparing these two directly would credit the gate with a difference that is not its own. For this reason a plain network with 9 units and **118** parameters is also tabulated. #### The Trace of the Marker in Three Structures ```python # trace.py — the measure of 04 repeated in three structures on the UNTRAINED network: only the # first step of the sequence is changed, the difference in the last hidden state is measured. def pairs(d, T): x = cut(d, T)[0] a = [s[:] for s in x] a[0] = [0.0, 1.0] b = [s[:] for s in x] b[0] = [d[1] / SCALE, 0.0] return a, b def trace(P, T, gated): f = 0.0 for d in SEQS: a, b = pairs(d, T) if gated: ia, ib = forward_gated(P, a)[1][-1][3], forward_gated(P, b)[1][-1][3] else: ia, ib = forward(P, a)[1][-1], forward(P, b)[1][-1] f += sum(abs(ia[k] - ib[k]) for k in range(len(ia))) return f / len(SEQS) STRUCTURES = [("plain 6", lambda t: build(t, 6), False), ("plain 9", lambda t: build(t, 9), False), ("gated 6", lambda t: build_gated(t, 6), True)] print(f"{'T':>3}" + "".join(f"{name + ' k' + str(k + 1):>12}" for name, _, _ in STRUCTURES for k in range(2))) first = {} for T in (2, 4, 8, 12): row = [] for name, f, gated in STRUCTURES: for k, seed in enumerate((SEED + 11, SEED + 12)): u = trace(f(seed), T, gated) row.append(u) if T == 2: first[(name, k)] = u print(f"{T:>3}" + "".join(f"{u:>12.5f}" for u in row)) print("share remaining at length 12 relative to the trace at length 2:") for i, (name, f, gated) in enumerate(STRUCTURES): for k, seed in enumerate((SEED + 11, SEED + 12)): print(f" {name} k{k + 1}: {trace(f(seed), 12, gated) / first[(name, k)]:.4%}") ``` ``` T plain 6 k1 plain 6 k2 plain 9 k1 plain 9 k2 gated 6 k1 gated 6 k2 2 0.92844 1.39970 2.23946 1.75462 0.32959 0.46329 4 0.34417 0.24972 0.65803 0.40122 0.17443 0.22063 8 0.05986 0.01560 0.10734 0.01796 0.07712 0.06693 12 0.00913 0.00067 0.01600 0.00083 0.03077 0.02610 share remaining at length 12 relative to the trace at length 2: plain 6 k1: 0.9837% plain 6 k2: 0.0478% plain 9 k1: 0.7143% plain 9 k2: 0.0473% gated 6 k1: 9.3350% gated 6 k2: 5.6329% ``` Raw numbers are misleading, the ratio lines correct them. The gated network carries **less** trace than the plain network at length 2 (0.32959 against 0.92844), because the update gate slows the new information too. What is to be measured is not the trace at the start but **the share remaining after ten steps**. That share is **0.98%** and **0.05%** in the plain network with 6 units; in the gated network it is **9.34%** and **5.63%**. The gate enlarges the remaining share of the marker ten to a hundred times. The two middle columns write what the gate is not. Widening the plain network from 6 units to 9 **lowers** the remaining share from 0.98% to 0.71%: the forgetting speed results not from the number of units but from the multiplication applied at every step. Only the form of the mixing relation changes the carrying capacity. #### How Many Steps Back ```python # train_model.py — MODEL: three structures on the same data, 30 epochs, learning rate 0.05. # Backpropagation through time is written by hand. def train_model(P, data, epochs, lr, seed): Wx, Wh, b, v, c = P H, r, n = len(b), rng(seed), len(data) for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: x, y = data[idx] p, states = forward(P, x) d = p - y dh = [d * v[k] for k in range(H)] for k in range(H): v[k] -= lr * d * states[-1][k] c[0] -= lr * d for t in range(len(x) - 1, -1, -1): previous, new = (states[t - 1] if t > 0 else [0.0] * H), [0.0] * H for k in range(H): g = dh[k] * (1 - states[t][k] ** 2) b[k] -= lr * g Wx[k][0] -= lr * g * x[t][0] Wx[k][1] -= lr * g * x[t][1] for j in range(H): new[j] += g * Wh[k][j] Wh[k][j] -= lr * g * previous[j] dh = new return P def train_gated(P, data, epochs, lr, seed): U, A, v, c, H = P r, n = rng(seed), len(data) for _ in range(epochs): s = list(range(n)) for i in range(n - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] for idx in s: x, y = data[idx] p, states = forward_gated(P, x) d = p - y dh = [d * v[k] for k in range(H)] for k in range(H): v[k] -= lr * d * states[-1][3][k] c[0] -= lr * d for t in range(len(x) - 1, -1, -1): previous, u, a, _ = states[t] new = [0.0] * H for k in range(H): du = dh[k] * (previous[k] - a[k]) * u[k] * (1 - u[k]) da = dh[k] * (1 - u[k]) * (1 - a[k] ** 2) U[2][k] -= lr * du A[2][k] -= lr * da for i2 in (0, 1): U[0][k][i2] -= lr * du * x[t][i2] A[0][k][i2] -= lr * da * x[t][i2] new[k] += u[k] * dh[k] for j in range(H): new[j] += du * U[1][k][j] + da * A[1][k][j] U[1][k][j] -= lr * du * previous[j] A[1][k][j] -= lr * da * previous[j] dh = new return P def accuracy(P, data, gated): f = forward_gated if gated else forward return sum(1 for x, y in data if (f(P, x)[0] >= 0.5) == (y == 1)) / len(data) print(f"{'T':>3}{'baseline':>10}" + "".join(f"{name + ' k' + str(k + 1):>12}" for name, _, _ in STRUCTURES for k in range(2))) for T in (2, 4, 8, 12): tr = [cut(d, T) for d in TRAIN] te = [cut(d, T) for d in TEST] tb, row = baseline_for(T), [] for name, builder, gated in STRUCTURES: for seed in (SEED + 11, SEED + 12): P = builder(seed) (train_gated if gated else train_model)(P, tr, 30, 0.05, seed + 500) row.append(accuracy(P, te, gated) - tb) print(f"{T:>3}{tb:>10.4f}" + "".join(f"{u:>+12.4f}" for u in row)) ``` ``` T baseline plain 6 k1 plain 6 k2 plain 9 k1 plain 9 k2 gated 6 k1 gated 6 k2 2 0.7200 +0.2300 +0.2800 +0.2350 +0.2800 +0.2250 +0.2650 4 0.7200 +0.2350 +0.2350 +0.2300 +0.2450 +0.2300 +0.2550 8 0.7450 +0.2100 +0.1700 +0.2000 +0.1800 +0.1750 +0.1800 12 0.7100 +0.0250 +0.0050 +0.2700 +0.1700 +0.2650 +0.2050 ``` The numbers are differences from the baseline: since the label is defined by the last period, each length has its own baseline. In the first three rows the three structures cannot be told apart — up to length 8 all of them beat the baseline by between 0.1700 and 0.2800, and a plain matrix can carry the dependency too. The last row opens the distinction. The plain network with 61 parameters drops to **0.0250** and **0.0050**, that is, to what the baseline says; the gated network gives **0.2650** and **0.2050**. The dependency is carried between eight and ten steps in the plain network with 6 units, and twelve in the gated network — the whole sequence. The third pair is not hidden and is the most important row of the lesson. **The wide plain network at an equal parameter budget also carries twelve steps:** 0.2700 and 0.1700. Compared with 0.2650 and 0.2050 of the gated network, there is no measurable difference between the two structures. Moreover the difference between the two runs of the wide plain network is **0.1000**; it is larger than any reading of the difference between the two structures. The untrained measurement had said that the gate enlarged the remaining share ten to a hundred times; the number of the trained network **neither confirms nor refutes** this at this length — because training can build a slowly forgetting matrix from the weights of the plain network by moving them from their initial values. The rule: the difference a structure adds is measurable only if the task is hard enough to force it. Twelve steps are not enough, and twelve steps is the limit of this setup; a distinction is expected to open on a longer sequence, but **an unmeasured expectation does not count**. #### Summary - The gated unit updates the state with $h_t = u_t h_{t-1} + (1 - u_t) a_t$; the coefficient is not constant, it is computed from the input of that step. - With the same 6 units the gated network holds 115 parameters and the plain network 61; to equalize the budget, the plain network with 9 units and 118 parameters is also measured. - In the untrained network the share remaining at length 12 is 0.98% and 0.05% in the plain network with 6 units, 0.71% and 0.05% in the plain network with 9 units, and 9.34% and 5.63% in the gated network; width does not change the forgetting speed. - Up to length 8 all three structures beat the baseline by 0.1700–0.2800 and cannot be told apart. - At length 12 the plain network with 6 units drops to 0.0250 and 0.0050; the gated network gives 0.2650 and 0.2050, and the wide plain network gives 0.2700 and 0.1700. - The advantage of the gate over the equal-budget plain network cannot be measured at this length: the difference of 0.1000 between the two runs of the wide plain network is larger than the difference between the two structures. #### Next Step The gated unit carried the long dependency, but the form of carrying did not change: the information still had to pass through **a single hidden state**, and that state was rewritten at every step. A distant step arrives by being transferred over all the steps in between; it is not read directly. The next lesson asks this: what would the network gain if it could choose **which step** of the sequence to look at at the moment of decision. What will be measured is where the weights fall: how many of the selections go to the correct step, and does a high weight mean that that step really determines the decision. ### Attention Mechanism Source: https://academia.sh/en/courses/deep-learning/attention-mechanism Measuring what looking directly at a distant step of the sequence gains and how much of the weights that are read amounts to an explanation: while the most-frequent-class baseline is 0.6333, two runs of the same attention network give 0.8889 and 0.9944 on the held-out set, and the 0.1056 between them comes from the initialization seed alone; the first run sends only 0.2721 of the weight to the marker step and puts the largest weight on the correct step in 0.3444 of the examples while producing 0.8889 accuracy, whereas the second run gives 0.9691 and 1.0000. In the step-removal test the agreement between weight and effect stays at 0.5889 in the first run; in the second run, in 43 of the 180 examples whose weight exceeds 0.5, removing that step moves the prediction by less than 0.05. Gated units solved one question. In a recurrent network, information from a distant step was kept from being erased on the way, and the dependency length that can be carried was measured. The form of the solution did not change, though: the information still moved forward step by step, was written into a single hidden state at every step, and was read from that single state at the end of the sequence. The gate regulated what would enter the state; it did not multiply the state itself. The question of this lesson is one step back. What does **looking directly** at a distant step of the sequence gain — which number would move if the output could see all the steps at once and choose for itself how much weight to give to which. The second question follows the first and is the real measure of this lesson: when the chosen weights are read, can one understand why the network gave that answer. - **AR46.** The sequence setup is **rebuilt** in this lesson and is **fictional**: a counter-reading sequence of twelve steps per subscriber, seed 20260218, 720 sequences, split 360/180/180. In the sequence of `architectures/04` the marker was always **in the first period**; whether the weights fall on the correct step can be measured only when the marker position varies between sequences, so the setup is built for this measure. - **AR47.** The readings are written as the deviation that remains after the subscriber's own level and seasonal component are removed. Step 0 is the same calibration reading for every subscriber and its value is 0.0. - **AR48.** Each sequence has a **marker step**; it lies between the second and the eleventh step. The label depends only on whether the reading of that step exceeds 0.35. The other eleven steps never enter the label. - **AR49.** The input is two channels per step: the reading and the marker indicator. The network sees which step is marked; it does not see the label. - **AR50.** The baseline is **the most frequent class**: giving every sequence the majority label. The table baseline of 0.7579 in the Introduction to Machine Learning through Supervised Learning courses is the number of a different task and is not carried here. - **AR51.** The attention layer is written inside the lesson and is **a model**. The query is single and learned; the key and the value are produced by linear projection; the weights are normalized with softmax. - **AR52.** Training is full-batch gradient descent: 200 epochs, learning rate 0.6. The gradient descent procedure was built in the `linear-models` topic of the Supervised Learning course and is not repeated. - **AR53.** Every number is printed with **two runs**, and the runs differ only in the initialization seed. This is the third number of this course. - **AR54.** The causality measure is **step removal**: a step is dropped from the attention distribution, the remaining weights are renormalized, and how much the output probability moved is printed. #### The Sequence Setup and the Baseline The task is deliberately narrow. There is a single step that determines the label, and that step is marked; the only thing the network has to do is find the marked step and compare its reading with the threshold. In such a task the question "did it look at the correct step" has an exact answer, because the correct step is written in the setup that generates the data. ```python # common.py — MODEL. FICTIONAL sequence set: a 12-step counter-reading sequence per subscriber. import math SEED, M32, STEPS = 20260218, 0xFFFFFFFF, 12 REGIONS = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("center", 0.18, 23)] def generator(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def discrete(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def build_sequences(n, seed): D = [] for i in range(n): r = generator(seed + i) b = REGIONS[discrete(r(), [x[1] for x in REGIONS])] reading = [0.0] + [round((r() + r() + r() - 1.5) * 1.9, 3) for _ in range(STEPS - 1)] m = 2 + int(r() * (STEPS - 2)) D.append({"reading": reading, "marker": m, "y": 1 if reading[m] > 0.35 else 0, "region": b[0]}) return D DATA = build_sequences(720, SEED) TRAIN, HELD, TEST = DATA[:360], DATA[360:540], DATA[540:] rate = sum(x["y"] for x in TEST) / len(TEST) BASELINE = max(rate, 1 - rate) print(f"sequences {len(DATA)}, length {STEPS}, split {len(TRAIN)}/{len(HELD)}/{len(TEST)}") print(f"positive rate train {sum(x['y'] for x in TRAIN) / len(TRAIN):.4f}, " f"held-out {rate:.4f}") print(f"baseline (most frequent class) {BASELINE:.4f}") print(f"share of sequences with a step above the threshold " f"{sum(any(x['reading'][t] > 0.35 for t in range(2, STEPS)) for x in DATA) / len(DATA):.4f}") print(f"does the largest reading fall on the marker step " f"{sum(max(range(2, STEPS), key=lambda t: x['reading'][t]) == x['marker'] for x in TEST) / len(TEST):.4f}") ``` ``` sequences 720, length 12, split 360/180/180 positive rate train 0.3806, held-out 0.3667 baseline (most frequent class) 0.6333 share of sequences with a step above the threshold 0.9903 does the largest reading fall on the marker step 0.0611 ``` The last two lines show that the task really requires looking inside the sequence. In 0.9903 of the sequences there is at least one step above the threshold, so the question "is there a high reading anywhere" answers yes for almost every sequence and separates nothing. Looking at the largest reading does not work either: the highest step of the sequence and the marker step coincide in only 0.0611 of the held-out set. The step the network has to choose is neither the largest nor at a fixed place. The baseline is 0.6333. Since the positive rate in the held-out set is 0.3667, giving every sequence the negative label yields this number. It is the plainest procedure that does not learn, and every number from here on is read next to it. #### Query, Key and Value Attention is defined with three projections. Each step produces a **key** and a **value** from its own input; the side that will produce the output carries a **query**. The inner product of the query and the key is a score that says how relevant that step is, the scores are passed through softmax and turned into weights that sum to one, and the output is the average of the values with these weights. The inner product is divided by the square root of the key dimension; dividing by $\sqrt{d}$ keeps the scores from growing as the dimension grows and saturating the softmax. In classification the query does not have to come from a step; a single learned query is enough. The network asks "which step is important for this decision" once and takes one summary of the whole sequence. There is no hidden state, no recurrence, no order dependency between steps. ```python DK = DV = 3 SCALE = 1.0 / math.sqrt(DK) INP = [([[x["reading"][t], 1.0 if t == x["marker"] else 0.0] for t in range(STEPS)], x["y"], x["marker"]) for x in DATA] TR_IN, TE_IN = INP[:360], INP[540:] def forward(P, X, skip=None): K = [[P["Wk"][i][0] * x[0] + P["Wk"][i][1] * x[1] for i in range(DK)] for x in X] V = [[P["Wv"][i][0] * x[0] + P["Wv"][i][1] * x[1] for i in range(DV)] for x in X] s = [sum(P["q"][i] * k[i] for i in range(DK)) * SCALE for k in K] d = [t for t in range(STEPS) if t != skip] mx = max(s[t] for t in d) e = [math.exp(s[t] - mx) if t in d else 0.0 for t in range(STEPS)] z = sum(e) a = [u / z for u in e] c = [sum(a[t] * V[t][i] for t in d) for i in range(DV)] return K, V, a, c, sum(P["w"][i] * c[i] for i in range(DV)) + P["b"] def train_model(seed, epochs=200, lr=0.6): r = generator(seed) g = lambda: (r() + r() + r() - 1.5) * 0.9 P = {"Wk": [[g(), g()] for _ in range(DK)], "Wv": [[g(), g()] for _ in range(DV)], "q": [g() for _ in range(DK)], "w": [g() for _ in range(DV)], "b": 0.0} for _ in range(epochs): gWk = [[0.0, 0.0] for _ in range(DK)] gWv = [[0.0, 0.0] for _ in range(DV)] gq, gw, gb = [0.0] * DK, [0.0] * DV, 0.0 for X, y, _m in TR_IN: K, V, a, c, o = forward(P, X) d = 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - y for i in range(DV): gw[i] += d * c[i] gb += d da = [sum(d * P["w"][i] * V[t][i] for i in range(DV)) for t in range(STEPS)] mean = sum(a[t] * da[t] for t in range(STEPS)) for t in range(STEPS): ds = a[t] * (da[t] - mean) * SCALE for i in range(DK): gq[i] += ds * K[t][i] gWk[i][0] += ds * P["q"][i] * X[t][0] gWk[i][1] += ds * P["q"][i] * X[t][1] for i in range(DV): gWv[i][0] += d * a[t] * P["w"][i] * X[t][0] gWv[i][1] += d * a[t] * P["w"][i] * X[t][1] n = len(TR_IN) for i in range(DK): P["q"][i] -= lr * gq[i] / n P["Wk"][i][0] -= lr * gWk[i][0] / n P["Wk"][i][1] -= lr * gWk[i][1] / n for i in range(DV): P["w"][i] -= lr * gw[i] / n P["Wv"][i][0] -= lr * gWv[i][0] / n P["Wv"][i][1] -= lr * gWv[i][1] / n P["b"] -= lr * gb / n return P def accuracy(P, G): return sum((forward(P, X)[4] > 0) == (y == 1) for X, y, _ in G) / len(G) NETS = [train_model(SEED + 1), train_model(SEED + 2)] print(f"parameter count {DK * 2 + DV * 2 + DK + DV + 1}, epochs 200, " f"training examples {len(TR_IN)}") print(f"baseline (most frequent class) {BASELINE:.4f}") for i, P in enumerate(NETS): print(f"attention net run {i + 1} train {accuracy(P, TR_IN):.4f} " f"held-out {accuracy(P, TE_IN):.4f}") print(f"difference between the runs {abs(accuracy(NETS[0], TE_IN) - accuracy(NETS[1], TE_IN)):.4f}") ``` ``` parameter count 19, epochs 200, training examples 360 baseline (most frequent class) 0.6333 attention net run 1 train 0.9028 held-out 0.8889 attention net run 2 train 0.9889 held-out 0.9944 difference between the runs 0.1056 ``` Three numbers are in the table. The baseline is 0.6333; the attention network gives 0.8889 and 0.9944 on the held-out set. Nineteen parameters learn to find the correct step in a sequence of length twelve and rise at least 0.2556 above the baseline. A recurrent network could do the same job only by carrying state for twelve steps. The third number says more than the second. The two runs share capacity, data and epoch budget; only the initialization seeds differ. The difference between them is 0.1056 — about one ninth of this network's held-out figure — and it comes not from what the network is but from where it started in that run. The rule of the course can be read directly here: the number of a network may come from its training rather than its capacity. #### Where the Weights Fall The weight vector that attention produces is the only readable intermediate product inside the network: it gives one number for each step, and they sum to one. This readability is the reason the weights are so often presented as "what the model looked at". The setup suits testing this claim, because the correct step is known. ```python def attention_measure(P, G): mass, hit = 0.0, 0 for X, _y, m in G: a = forward(P, X)[2] mass += a[m] hit += max(range(STEPS), key=lambda t: a[t]) == m return mass / len(G), hit / len(G) print(f"weight of one step in a uniform distribution {1 / STEPS:.4f}") print(f"{'':16}{'weight on marker step':>21}{'largest weight':>18}{'held-out':>10}") for i, P in enumerate(NETS): k, s = attention_measure(P, TE_IN) print(f"run {i + 1:<12}{k:>21.4f}{s:>18.4f}{accuracy(P, TE_IN):>10.4f}") X0, _y0, m0 = TE_IN[0] a0 = forward(NETS[0], X0)[2] a1 = forward(NETS[1], X0)[2] print(f"single example, marker step {m0}") print("run 1 weights " + " ".join(f"{u:.2f}" for u in a0)) print("run 2 weights " + " ".join(f"{u:.2f}" for u in a1)) ``` ``` weight of one step in a uniform distribution 0.0833 weight on marker step largest weight held-out run 1 0.2721 0.3444 0.8889 run 2 0.9691 1.0000 0.9944 single example, marker step 11 run 1 weights 0.00 0.00 0.10 0.00 0.01 0.01 0.00 0.29 0.31 0.00 0.10 0.18 run 2 weights 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 ``` The second run shows the expected behavior: 0.9691 of the weight falls on the marker step, and the largest weight is on the correct step in the whole held-out set. Since the share of one step in a uniform distribution is 0.0833, this is more than eleven times the baseline. The first run solves the same task with 0.8889 accuracy, but it sends only 0.2721 of the weight to the marker step and puts the largest weight on the correct step in 0.3444 of the examples. The single-example row shows how this is possible: the marker step is the eleventh step, run 1 gives it 0.18, and distributes the rest to the seventh and eighth steps. The network has found a compromise that uses the marker channel partially and spreads the remaining weight over steps with high readings, and this compromise produces the correct answer in most cases. While the accuracy is 0.8889, the hit rate of the weights on the correct step is 0.3444; there is no necessary link between the two numbers. #### A High Weight Is Not Causation A large weight does not mean that the step determines the output. Whether the two coincide can be tested only with an intervention: the step is dropped from the attention distribution, the remaining weights are renormalized, and how much the output probability moved is examined. This is a measure independent of the size of the weight. ```python def probability(P, X, skip=None): o = forward(P, X, skip)[4] return 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) def causality(P, G, threshold=0.05): agree, high, high_inert, mean_effect = 0, 0, 0, 0.0 for X, _y, _m in G: a = forward(P, X)[2] p0 = probability(P, X) effect = [abs(probability(P, X, t) - p0) for t in range(STEPS)] ta = max(range(STEPS), key=lambda t: a[t]) agree += ta == max(range(STEPS), key=lambda t: effect[t]) mean_effect += effect[ta] if a[ta] > 0.5: high += 1 high_inert += effect[ta] < threshold n = len(G) return agree / n, mean_effect / n, high, high_inert print(f"{'':10}{'weight-effect agreement':>25}{'effect of heaviest step':>25}" f"{'weight>0.5':>13}{'but effect<0.05':>17}") for i, P in enumerate(NETS): u, e, y, ye = causality(P, TE_IN) print(f"run {i + 1:<6}{u:>25.4f}{e:>25.4f}{y:>13d}{ye:>17d}") ``` ``` weight-effect agreement effect of heaviest step weight>0.5 but effect<0.05 run 1 0.5889 0.3318 81 27 run 2 1.0000 0.4081 180 43 ``` In the first run, the heaviest step and the most influential step are the same in only 0.5889 of the examples. That is, someone who reads the weight ranking and says "the model looked here" misses, in four examples out of ten, the step that actually determines the output. In the second run the agreement is 1.0000. Same architecture, same data, same budget; the explanatory value of the weights changes from run to run. The weights alone do not reveal which run one is in — both produce concentrated, orderly looking distributions. The last column shows that the second run has a limit within itself as well. In the whole held-out set the largest weight exceeds 0.5, but in 43 of these 180 examples, removing that step moves the prediction probability by less than 0.05. The reason is clear: if the remaining steps also carry low readings, dropping the marker step leaves the average on the same side and the decision does not change. Here a high weight does not mean "this step went into the decision"; it means "this step was read". The rule that follows: attention weights are an intermediate product useful in debugging, not a justification document on their own. A weight reported without a removal test or the run-to-run variability beside it is an unmeasured number. #### Summary - The sequence setup was built in this lesson and is fictional: 720 sequences, length 12, split 360/180/180, seed 20260218; the label depends only on the reading of the marker step. - The baseline is the most frequent class and is 0.6333; the attention network with nineteen parameters gives 0.8889 and 0.9944 on the held-out set, and the difference between the two runs is 0.1056. - Attention scales the inner product of the query and the key by $\sqrt{d}$, passes it through softmax, and produces the weighted average of the values; there is no hidden state and no recurrence. - Where the weights fall is not the same thing as accuracy: at 0.8889 accuracy, the first run sends 0.2721 of the weight to the marker step and places the largest weight with a hit rate of 0.3444. - In the step-removal test, the agreement between weight and effect is 0.5889 in the first run, and in the second run, in 43 of the 180 examples whose weight exceeds 0.5, removing that step moves the prediction by less than 0.05. #### Next Step In this lesson a single query produced a single weight distribution and the sequence was reduced to a single summary. The task suited this: the label came from a single step and step order never entered into it. Two questions remain open. The first: if there are several patterns of interest in the same sequence — one looking for the marker step, the other for the general level — can a single distribution carry both. The second, and the sharper one: since attention looks at all the steps at the same time, how will it know the order of the steps. The next lesson multiplies the query, measures the difference the number of heads adds, and prints against the same baseline how much of the order information remains when the positional encoding is removed. ### Multi-Head Attention and Transformers Source: https://academia.sh/en/courses/deep-learning/multi-head-attention-and-transformers Measuring what multiplying the query and the positional encoding add under the same parameter budget: with 33 parameters held fixed, the held-out mean is 0.8722, 0.8806 and 0.8972 for 1, 2 and 4 heads, so the number of heads adds 0.0250 while the difference between the two runs of a single setup rises to 0.0500. The second run of the two-head network separates the heads to the two ends of the sequence (weight centers 3.30 and 8.55, separation 0.7264) but the first run does not (5.57 and 6.30) and its accuracy does not drop. When the positional encoding is removed, both runs fall exactly to the baseline, 0.6611, and when the readings are shuffled the output moves by 0.0000. The attention layer carried a single query and reduced the whole sequence to a single weight distribution. The task was suited to this: the label was read from a single step and the order in which the steps arrived changed nothing. In real sequences two things change at once. A decision often comes out of the combination of information read from several places in the sequence, and where that information stands matters as much as the information itself. This lesson measures two additions. The first is the multiplication of the query: the same inner dimension is divided among several **heads**, each head produces its own weight distribution, and the outputs are concatenated. The second is the **positional encoding**: the place of the step in the sequence is added next to the input as a number. These two, together with a per-step projection and skip connections, are the load-bearing parts of the **transformer architecture** — a structure without recurrence that looks at all the steps at once. Here only the two load-bearing parts are written and measured. - **AR55.** The sequence setup of the Architectures topic was built in `architectures/06` and is **fictional**: 720 sequences, length 12, seed 20260218. The same sequences are regenerated with the same seed. - **AR56.** The question of this lesson is different and **depends on order**: the label is whether the average of the seventh through eleventh steps exceeds the average of the first through fifth steps by more than 0.30. The sixth step is left out of both regions. - **AR57.** The baseline is again **the most frequent class**. Because the label is read from the difference of two regions, the general level of the sequence is not enough to pass the baseline. - **AR58.** The positional encoding is a sine channel and a cosine channel: $\sin(\pi t / 11)$ and $\cos(\pi t / 11)$ for step $t$. A single frequency was chosen to keep the scale small; real setups use several frequencies. - **AR59.** The comparison is **under an equal parameter budget**. The total inner dimension is fixed at 4 and the size per head shrinks as the number of heads grows. The parameter count is the same in all three setups. - **AR60.** The layer is written inside the lesson and is **a model**. No library is called. - **AR61.** Training is **descent with momentum**: 150 epochs, learning rate 0.15, momentum 0.8. Momentum was built in the `neural-network-fundamentals` topic of this course and is not repeated here. - **AR62.** Each setup is printed with **two runs**; the runs differ only in the initialization seed. - **AR63.** The order test is this: the **readings** of a sequence are shuffled among themselves, the position channels are left in place, and how much the output probability moved is measured. #### The Same Budget, a Different Number of Heads The sequences of the previous lesson are used as they are; the question changes. Because the new label is read from the difference between the averages of two regions, it is hard to produce with a single weighted average: a sequence-wide average cannot separate two regions. ```python # common.py — MODEL. The FICTIONAL sequence set built in `architectures/06` is regenerated with the same seed. import math SEED, M32, STEPS = 20260218, 0xFFFFFFFF, 12 def generator(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_sequences(n, seed): D = [] for i in range(n): r = generator(seed + i) r() # region draw: not used in this lesson, consumed so the sequence stays the same reading = [0.0] + [round((r() + r() + r() - 1.5) * 1.9, 3) for _ in range(STEPS - 1)] r() D.append({"reading": reading}) return D DATA = build_sequences(720, SEED) FIRST, LAST = range(1, 6), range(7, 12) for x in DATA: a = sum(x["reading"][t] for t in FIRST) / 5 b = sum(x["reading"][t] for t in LAST) / 5 x["y"] = 1 if b - a > 0.30 else 0 TRAIN, TEST = DATA[:200], DATA[540:] rate = sum(x["y"] for x in TEST) / len(TEST) BASELINE = max(rate, 1 - rate) print(f"sequences {len(DATA)}, length {STEPS}, training {len(TRAIN)}, held-out {len(TEST)}") print(f"rise label rate train {sum(x['y'] for x in TRAIN) / len(TRAIN):.4f}, " f"held-out {rate:.4f}") print(f"baseline (most frequent class) {BASELINE:.4f}") print(f"example sequence {DATA[0]['reading']}") ``` ``` sequences 720, length 12, training 200, held-out 180 rise label rate train 0.3150, held-out 0.3389 baseline (most frequent class) 0.6611 example sequence [0.0, 0.893, -0.222, -0.542, -0.404, -1.114, 0.234, -1.03, -1.493, 0.534, -1.807, 0.05] ``` In multi-head attention every head has its own key, value and query projections. Because the total inner dimension is held fixed, when the number of heads doubles the size per head is halved; the parameter count does not change. The meaning of the comparison lies here — if there is a gain, it comes not from capacity but from the **way the same capacity is divided**. ```python DTOP, DIN = 4, 3 def inputs(x, encode=True): K = [[math.sin(math.pi * t / (STEPS - 1)), math.cos(math.pi * t / (STEPS - 1))] for t in range(STEPS)] return [[x["reading"][t]] + (K[t] if encode else [0.0, 0.0]) for t in range(STEPS)] def init_params(seed, H): d = DTOP // H r = generator(seed) g = lambda: (r() + r() + r() - 1.5) * 0.9 return {"H": H, "d": d, "scale": 1.0 / math.sqrt(d), "Wk": [[[g() for _ in range(DIN)] for _ in range(d)] for _ in range(H)], "Wv": [[[g() for _ in range(DIN)] for _ in range(d)] for _ in range(H)], "q": [[g() for _ in range(d)] for _ in range(H)], "w": [g() for _ in range(DTOP)], "b": [0.0]} def weights(P, X, h): # the query and the key projection fold into a single direction: u = q^T Wk d, wk, q = P["d"], P["Wk"][h], P["q"][h] u = [sum(q[i] * wk[i][j] for i in range(d)) for j in range(DIN)] s = [(u[0] * x[0] + u[1] * x[1] + u[2] * x[2]) * P["scale"] for x in X] mx = max(s) e = [math.exp(v - mx) for v in s] z = sum(e) return [v / z for v in e] def head(P, X, h): a = weights(P, X, h) Xa = [sum(a[t] * X[t][j] for t in range(STEPS)) for j in range(DIN)] return a, Xa, [sum(P["Wv"][h][i][j] * Xa[j] for j in range(DIN)) for i in range(P["d"])] def forward(P, X): d, o = P["d"], P["b"][0] for h in range(P["H"]): c = head(P, X, h)[2] o += sum(P["w"][h * d + i] * c[i] for i in range(d)) return o def train_model(seed, H, TR, epochs=150, lr=0.15, mom=0.8): P = init_params(seed, H) d, scale, n, vel = P["d"], P["scale"], len(TR), None for _ in range(epochs): gWk = [[[0.0] * DIN for _ in range(d)] for _ in range(H)] gWv = [[[0.0] * DIN for _ in range(d)] for _ in range(H)] gq = [[0.0] * d for _ in range(H)] gw, gb = [0.0] * DTOP, [0.0] for X, y in TR: B = [head(P, X, h) for h in range(H)] o = P["b"][0] + sum(P["w"][h * d + i] * B[h][2][i] for h in range(H) for i in range(d)) df = 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - y gb[0] += df for h in range(H): a, Xa, c = B[h] dc = [df * P["w"][h * d + i] for i in range(d)] mv = [sum(dc[i] * P["Wv"][h][i][j] for i in range(d)) for j in range(DIN)] da = [mv[0] * x[0] + mv[1] * x[1] + mv[2] * x[2] for x in X] mean = sum(a[t] * da[t] for t in range(STEPS)) S = [0.0, 0.0, 0.0] for t in range(STEPS): ds, x = a[t] * (da[t] - mean) * scale, X[t] S[0] += ds * x[0] S[1] += ds * x[1] S[2] += ds * x[2] for i in range(d): gw[h * d + i] += df * c[i] gq[h][i] += sum(P["Wk"][h][i][j] * S[j] for j in range(DIN)) for j in range(DIN): gWk[h][i][j] += P["q"][h][i] * S[j] gWv[h][i][j] += dc[i] * Xa[j] params = [P["Wk"][h][i] for h in range(H) for i in range(d)] \ + [P["Wv"][h][i] for h in range(H) for i in range(d)] \ + P["q"] + [P["w"], P["b"]] grads = [gWk[h][i] for h in range(H) for i in range(d)] \ + [gWv[h][i] for h in range(H) for i in range(d)] + gq + [gw, gb] if vel is None: vel = [[0.0] * len(u) for u in params] for u, v, g in zip(params, grads, vel): # gradient descent with momentum for i in range(len(u)): g[i] = mom * g[i] - lr * v[i] / n u[i] += g[i] return P def accuracy(P, G): return sum((forward(P, X) > 0) == (y == 1) for X, y in G) / len(G) ALL_IN = [(inputs(x), x["y"]) for x in DATA] TR_IN, TE_IN = ALL_IN[:200], ALL_IN[540:] NETS = {} print(f"total inner dimension {DTOP}, epochs 150, training examples {len(TR_IN)}, " f"baseline {BASELINE:.4f}") print(f"{'heads':>6}{'head size':>12}{'parameters':>11}{'run 1':>10}{'run 2':>10}" f"{'mean':>10}{'run difference':>16}") for H in (1, 2, 4): NETS[H] = [train_model(SEED + k, H, TR_IN) for k in (1, 2)] a, b = (accuracy(P, TE_IN) for P in NETS[H]) print(f"{H:>6}{DTOP // H:>12}{2 * DTOP * DIN + 2 * DTOP + 1:>11}{a:>10.4f}" f"{b:>10.4f}{(a + b) / 2:>10.4f}{abs(a - b):>16.4f}") ``` ``` total inner dimension 4, epochs 150, training examples 200, baseline 0.6611 heads head size parameters run 1 run 2 mean run difference 1 4 33 0.8889 0.8556 0.8722 0.0333 2 2 33 0.8778 0.8833 0.8806 0.0056 4 1 33 0.8722 0.9222 0.8972 0.0500 ``` The means move in one direction: 0.8722, 0.8806, 0.8972. Raising the number of heads from one to four adds 0.0250 to the held-out figure and carries it 0.2361 above the baseline. The column on the right puts the gain in context: the difference between the two runs of the four-head setup is 0.0500, which is twice what the number of heads adds. The difference that dividing the same capacity among heads brings is smaller than the difference that rerunning that division itself brings. At this scale the number of heads is a hyperparameter, and if it is chosen with a single run it is chosen wrongly. #### What the Heads Look At Why the number of heads adds anything can be asked by looking at where the weights fall. The task wants the average of two regions; if the heads split into these two regions, the structure has found the form of the task. The measure is two numbers: the weight center of each head and the separation between heads — half the difference between two distributions, from zero to one. ```python def center(P, G, h): return sum(sum(t * weights(P, X, h)[t] for t in range(STEPS)) for X, _ in G) / len(G) def separation(P, G): H, s = P["H"], 0.0 if H < 2: return 0.0 for X, _ in G: A = [weights(P, X, h) for h in range(H)] for i in range(H): for j in range(i + 1, H): s += sum(abs(A[i][t] - A[j][t]) for t in range(STEPS)) / 2 return s / (len(G) * H * (H - 1) / 2) print(f"the two regions the task wants: steps {min(FIRST)}-{max(FIRST)} and {min(LAST)}-{max(LAST)}") print(f"{'':18}{'weight centers':>34}{'separation':>12}{'held-out':>10}") for H in (1, 2, 4): for k in (0, 1): P = NETS[H][k] m = ", ".join(f"{center(P, TE_IN, h):.2f}" for h in range(H)) name = f"{H} head" + ("s" if H > 1 else "") + f", run {k + 1}" print(f"{name:<18}{m:>34}{separation(P, TE_IN):>12.4f}" f"{accuracy(P, TE_IN):>10.4f}") ``` ``` the two regions the task wants: steps 1-5 and 7-11 weight centers separation held-out 1 head, run 1 5.51 0.0000 0.8889 1 head, run 2 6.02 0.0000 0.8556 2 heads, run 1 5.57, 6.30 0.4096 0.8778 2 heads, run 2 3.30, 8.55 0.7264 0.8833 4 heads, run 1 5.84, 5.48, 5.79, 5.61 0.4782 0.8722 4 heads, run 2 3.24, 5.81, 8.37, 4.59 0.4291 0.9222 ``` The second run of the two-head setup makes exactly the expected distinction: one head stands at 3.30, the other at 8.55, and these two points fall in the middle of the two regions the task wants. The separation is 0.7264, that is, the two distributions largely look at different steps. The first run of the same setup does not do this. The heads are at 5.57 and 6.30, both standing in the middle of the sequence, and the separation stays at 0.4096. Yet the held-out figure is 0.8778 — only 0.0055 below the run that separates. The same observation is even clearer in the four-head setup: while the four heads of the first run are squeezed between 5.48 and 5.84, its accuracy is 0.8722. The structure "finding the form of the task" comes not from the architecture but from the run, and a run that has not found it can also produce a close number. #### Positional Encoding Is a Prerequisite, Not an Improvement Attention looks at all the steps at once and carries no order link between steps. If there is no number about its place in the input of a step, then for the layer the sequence is a **multiset**: the same readings arriving in another order give the same output. This is not a loss of performance but an absence of information, and it can be tested directly. ```python def shuffle_readings(X, seed): r, s = generator(seed), list(range(STEPS)) for i in range(STEPS - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [[X[s[t]][0], X[t][1], X[t][2]] for t in range(STEPS)] def probability(P, X): return 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, forward(P, X))))) def order_test(P, G): e, mx = 0.0, 0.0 for i, (X, _y) in enumerate(G): d = abs(probability(P, shuffle_readings(X, SEED + 4000 + i)) - probability(P, X)) e, mx = e + d, max(mx, d) return e / len(G), mx NO_POS = [(inputs(x, False), x["y"]) for x in DATA] NO_ENC = [train_model(SEED + k, 2, NO_POS[:200]) for k in (1, 2)] print(f"baseline (most frequent class) {BASELINE:.4f}") print(f"{'':28}{'run 1':>9}{'run 2':>9}{'mean |change|':>15}{'largest':>10}") for name, M, G in (("positional encoding present", NETS[2], TE_IN), ("positional encoding absent", NO_ENC, NO_POS[540:])): e, mx = order_test(M[1], G) print(f"{name:<28}{accuracy(M[0], G):>9.4f}{accuracy(M[1], G):>9.4f}" f"{e:>15.4f}{mx:>10.4f}") ``` ``` baseline (most frequent class) 0.6611 run 1 run 2 mean |change| largest positional encoding present 0.8778 0.8833 0.3173 0.9753 positional encoding absent 0.6611 0.6611 0.0000 0.0000 ``` The bottom row is sharp in two places at once. The setup without the encoding gives exactly 0.6611 in both runs — the baseline itself. There is no difference even in the decimal places, because the network cannot learn anything other than giving every sequence the same class. The order test says why: when the readings are shuffled, the output probability moves 0.0000 on average and 0.0000 at most. This is not rounding; because the layer sees the input as an unordered set, the output is the same bit for bit. In the setup with the encoding, the same shuffling moves the output 0.3173 on average and 0.9753 at most. The difference between them is not that the encoding "helps"; it is that the order enters the reach of the network only through the encoding. In a recurrent network the order comes from the structure itself and does not need to be encoded separately; because attention does not give this assurance, writing the order into the input is not a choice but a necessity. #### Summary - When the same inner dimension is divided among different numbers of heads, the parameter count stays fixed at 33; the held-out mean is 0.8722, 0.8806 and 0.8972, and what the number of heads adds is 0.0250. - The difference between the two runs of the four-head setup is 0.0500, twice what the number of heads adds; if the number of heads is chosen with a single run, it is chosen wrongly. - The heads separate into the two regions the task wants only in some runs: the second run of the two-head setup gives a separation of 0.7264 with centers 3.30 and 8.55, its first run stays at 0.4096 with 5.57 and 6.30, and its accuracy is only 0.0055 lower. - Without the positional encoding both runs fall exactly to the baseline, 0.6611; when the readings are shuffled the output moves 0.0000, because the layer sees the sequence as an unordered set. - In the setup with the positional encoding, the same shuffling moves the output 0.3173 on average and 0.9753 at most. #### Next Step In these two lessons every step of the input was a ready-made number: the reading a floating-point value, the marker a zero or a one, the position a sine. Most sequences do not arrive like this. A counter type, a tariff tier, a region code or an event label is a **discrete** symbol and has no natural numeric value. The encoding lessons of the Data Preparation and Feature Engineering course had turned these symbols into one-hot columns; as the number of categories grows, this grows the input dimension in the same proportion and carries no closeness between the symbols. The next lesson builds the embedding layer that maps a discrete input to a learned vector, counts the parameter count against one-hot encoding, and compares this learned representation, on the same measure, with the **non-learned** projection of the Unsupervised Learning course. ### Embedding Layers Source: https://academia.sh/en/courses/deep-learning/embedding-layers Measuring what mapping a discrete event code to a continuous vector adds: while the most-frequent-class baseline is 0.6444, the learned embedding with 103 parameters gives 0.9722 and 0.9833 at 150 epochs, the one-hot encoding with 157 parameters stays at 0.7167 and 0.7000 in the same epochs and reaches the same figure only at 600 epochs (0.9611 and 0.9556). The frozen table of the same size stays at 0.6944 and 0.7278, and the ordinal numeric code stands exactly at the baseline in all four epoch budgets. In the learned table the distance between the suspect codes and the other codes is 3.9153 times the within-group distance; in the frozen table it is 1.0728 times. In the previous two lessons every step of the input was a ready-made number. Sequences do not arrive like this. In a counter record, next to the reading there is an **event code** — calibration, communication outage, cover opening, manual reading — and this code is a symbol. It has no numeric value; that the seventeenth code is larger than the sixteenth says nothing. The Data Preparation and Feature Engineering course had turned these symbols into one-hot columns: one indicator column for each code. The procedure works, but it has two costs. The input dimension grows with the number of codes, and no closeness between the symbols is carried — two codes that can substitute for each other are as far apart as two unrelated codes. The question of this lesson is this: which number moves if the representation of the code is **learned** instead of given by hand, and what is the difference between the learned representation and a representation fixed without ever looking at the task. - **AR64.** The sequence setup of the Architectures topic was built in `architectures/06` and is **fictional**. In this lesson a **fictional** event code is added to each step: 24 codes, one per step, seed 20260218. The readings do not change. - **AR65.** Eight codes are marked as **suspect** and the numbers of these eight in the series are not adjacent. The label is the presence of a suspect code in at least five of the twelve steps. - **AR66.** The baseline is **the most frequent class**. Because the order of the code numbers is unrelated to the label, an ordinal numeric encoding is not expected to pass the baseline, and this is measured. - **AR67.** The model is written inside the lesson and is **a model**: the representations of the steps are averaged, passed through a hidden layer (six units, rectified linear unit) and connected to a single output. No library is called. - **AR68.** The four setups share the same body and differ only in the **representation of the code**: one-hot, learned embedding, frozen embedding, ordinal numeric code. - **AR69.** In the one-hot setup the table is **the identity matrix** and is not stored; the code realizes it with counting. In the parameter count the identity matrix is not counted as parameters, because it has no free value. - **AR70.** The frozen embedding is a table of the same size that is initialized and **never updated**. The projection of the `dimensionality-reduction` topic of the Unsupervised Learning course was also fixed without looking at the task; the measure there was the reconstruction error, here it is the held-out figure. - **AR71.** Training is full-batch gradient descent with learning rate 0.9 and is measured at four epoch budgets: 150, 300, 450, 600. - **AR72.** Each setup is printed with **two runs**; the runs differ in the initialization seed. #### The Same Body, Four Representations The task depends on the count of the codes, not on their order. For this reason the model averages the step representations and gives the average to a small hidden layer. Because the label is determined exactly by the codes, a representation that can separate the suspect codes from the others can solve the task perfectly; a representation that cannot separate them cannot solve it in any epoch budget. ```python # common.py — MODEL. The FICTIONAL sequence set of `architectures/06` is regenerated with the same seed; # a FICTIONAL event code is added to every step. import math SEED, M32, STEPS = 20260218, 0xFFFFFFFF, 12 CODES, SUSPECT, MIN_SUSPECT = 24, (1, 4, 5, 9, 13, 16, 20, 22), 5 def generator(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_sequences(n, seed): D = [] for i in range(n): r = generator(seed + i) r() reading = [0.0] + [round((r() + r() + r() - 1.5) * 1.9, 3) for _ in range(STEPS - 1)] r() o = generator(seed + 8000 + i) codes = [int(o() * CODES) for _ in range(STEPS)] D.append({"reading": reading, "codes": codes, "y": 1 if sum(c in SUSPECT for c in codes) >= MIN_SUSPECT else 0}) return D DATA = build_sequences(720, SEED) TRAIN, TEST = DATA[:200], DATA[540:] rate = sum(x["y"] for x in TEST) / len(TEST) BASELINE = max(rate, 1 - rate) print(f"sequences {len(DATA)}, length {STEPS}, codes {CODES}, suspect codes {len(SUSPECT)}") print(f"label rate train {sum(x['y'] for x in TRAIN) / len(TRAIN):.4f}, held-out {rate:.4f}") print(f"baseline (most frequent class) {BASELINE:.4f}") print(f"example code sequence {DATA[0]['codes']}, suspect count " f"{sum(c in SUSPECT for c in DATA[0]['codes'])}, label {DATA[0]['y']}") ``` ``` sequences 720, length 12, codes 24, suspect codes 8 label rate train 0.3450, held-out 0.3556 baseline (most frequent class) 0.6444 example code sequence [0, 8, 0, 16, 22, 10, 17, 7, 14, 19, 7, 18], suspect count 2, label 0 ``` The embedding layer is a table: as many rows as the number of codes, as many columns as the embedding dimension. The representation of a code is the row read from that table, and the row is a parameter — the gradient updates it too. In the one-hot setup there is no table, because multiplying the indicator vector by a linear layer is already reading a column of that layer. The difference between the two setups is not a difference of principle but of **where** the row is written and how many there are. ```python HIDDEN, STOPS = 6, (150, 300, 450, 600) def represent(T, codes): if T is None: # one-hot: the table is the identity matrix, counting is enough h = [0.0] * CODES for c in codes: h[c] += 1.0 / STEPS return h h = [0.0] * len(T[0]) for c in codes: for j in range(len(h)): h[j] += T[c][j] / STEPS return h def make_table(kind, seed, d=3): if kind == "one-hot": return None, CODES if kind == "numeric": return [[c / (CODES - 1)] for c in range(CODES)], 1 r = generator(seed) return [[(r() + r() + r() - 1.5) * 1.1 for _ in range(d)] for _ in range(CODES)], d def train_model(seed, T, dm, learn, TR, lr=0.9): r = generator(seed + 500) g = lambda: (r() + r() + r() - 1.5) * 0.9 W1 = [[g() for _ in range(dm)] for _ in range(HIDDEN)] b1, w, b, n, scores = [0.0] * HIDDEN, [g() for _ in range(HIDDEN)], [0.0], len(TR), {} for epoch in range(1, max(STOPS) + 1): gW1 = [[0.0] * dm for _ in range(HIDDEN)] gb1, gw, gb = [0.0] * HIDDEN, [0.0] * HIDDEN, 0.0 gT = [[0.0] * dm for _ in range(CODES)] if learn else None for h, codes, y in TR: z = [sum(W1[i][j] * h[j] for j in range(dm)) + b1[i] for i in range(HIDDEN)] o = sum(w[i] * z[i] for i in range(HIDDEN) if z[i] > 0) + b[0] df = 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - y gb += df da = [df * w[i] if z[i] > 0 else 0.0 for i in range(HIDDEN)] for i in range(HIDDEN): gw[i] += df * max(0.0, z[i]) gb1[i] += da[i] for j in range(dm): gW1[i][j] += da[i] * h[j] if learn: dh = [sum(da[i] * W1[i][j] for i in range(HIDDEN)) for j in range(dm)] for c in codes: for j in range(dm): gT[c][j] += dh[j] / STEPS for i in range(HIDDEN): w[i] -= lr * gw[i] / n b1[i] -= lr * gb1[i] / n for j in range(dm): W1[i][j] -= lr * gW1[i][j] / n b[0] -= lr * gb / n if learn: # the embedding table is a parameter too for c in range(CODES): for j in range(dm): T[c][j] -= lr * gT[c][j] / n for k, (_h, codes, y) in enumerate(TR): TR[k] = (represent(T, codes), codes, y) if epoch in STOPS: P = {"T": T, "dm": dm, "W1": [u[:] for u in W1], "b1": b1[:], "w": w[:], "b": b[0]} scores[epoch] = accuracy(P, TEST) return P, scores def output(P, codes): h = represent(P["T"], codes) z = [sum(P["W1"][i][j] * h[j] for j in range(P["dm"])) + P["b1"][i] for i in range(HIDDEN)] return sum(P["w"][i] * max(0.0, z[i]) for i in range(HIDDEN)) + P["b"] def accuracy(P, D): return sum((output(P, x["codes"]) > 0) == (x["y"] == 1) for x in D) / len(D) NETS, SETUPS = {}, (("one-hot", "one-hot", False), ("embedding, learned", "embedding", True), ("embedding, frozen", "embedding", False), ("numeric code", "numeric", False)) print(f"hidden units {HIDDEN}, training examples {len(TRAIN)}, " f"baseline {BASELINE:.4f}") print(f"{'setup':<21}{'parameters':>10}{'learned':>10}{'run':>7}" + "".join(f"{'epoch ' + str(t):>10}" for t in STOPS)) for name, kind, learn in SETUPS: NETS[name] = [] for k in (1, 2): T, dm = make_table(kind, SEED + 900 + k) TR = [(represent(T, x["codes"]), x["codes"], x["y"]) for x in TRAIN] P, scores = train_model(SEED + k, T, dm, learn, TR) NETS[name].append(P) emb = CODES * dm if kind == "embedding" else 0 params = HIDDEN * dm + 2 * HIDDEN + 1 + emb learned = params - (0 if learn else emb) print(f"{name if k == 1 else '':<21}{params:>10}{learned:>10}{k:>7}" + "".join(f"{scores[t]:>10.4f}" for t in STOPS)) ``` ``` hidden units 6, training examples 200, baseline 0.6444 setup parameters learned run epoch 150 epoch 300 epoch 450 epoch 600 one-hot 157 157 1 0.7167 0.8000 0.9222 0.9611 157 157 2 0.7000 0.8111 0.8611 0.9556 embedding, learned 103 103 1 0.9722 1.0000 1.0000 1.0000 103 103 2 0.9833 1.0000 1.0000 1.0000 embedding, frozen 103 31 1 0.6944 0.6778 0.6778 0.6833 103 31 2 0.7278 0.7389 0.7444 0.7444 numeric code 19 19 1 0.6444 0.6444 0.6444 0.6444 19 19 2 0.6444 0.6444 0.6444 0.6444 ``` Four rows give four separate results. The learned embedding rises to 0.9722 and 0.9833 at 150 epochs and settles at 1.0000 in both runs at 300 epochs — because the task is determined exactly by the codes, this is the expected ceiling. The one-hot setup gives 0.7167 and 0.7000 in the same 150 epochs, that is, a little above the baseline. The difference does not come from expressive power: at 600 epochs it rises to 0.9611 and 0.9556. It arrives at the same place with four times the epoch budget, and it does so with 157 parameters, against 103 for the embedding. The frozen table has the same size and is connected to the same body; its only difference is that the 72-row table is never updated. It stays between 0.68 and 0.75 in all four epoch budgets. Adding epochs does not help, because what is missing is not budget: three directions chosen without looking at the task do not separate the eight suspect codes from the other sixteen, and the body cannot produce a distinction from a representation that is not separated. The `dimensionality-reduction` topic of the Unsupervised Learning course had measured the same position with the reconstruction error — there, the two-component projection had given 2.7762 against the 3.3411 of reconstructing with the column means — and the directions there were also chosen from the data, not from the label. That a representation preserves the data well does not mean it carries the distinction the task wants. The last row is the harshest. The ordinal numeric code gives exactly 0.6444 in all four budgets — the baseline itself. Because the order among the code numbers is unrelated to the label, a single channel carries nothing. The warning of the encoding lesson of the Data Preparation and Feature Engineering course turns into a number here. #### How the Parameter Count Grows with the Number of Codes At twenty-four codes the parameter gain of the embedding is small: 103 against 157. Where the gain comes from becomes visible only as the number of codes and the hidden layer grow. The calculation is a single line and requires no run. ```python def count_params(V, hidden, d=None): if d is None: return V * hidden + 2 * hidden + 1 # one-hot vector straight into the layer return V * d + d * hidden + 2 * hidden + 1 # embedding table, then the layer print(f"{'codes':>11}{'hidden units':>13}{'one-hot':>11}{'embedding d=3':>15}" f"{'ratio':>8}{'multiplications per step':>26}") for V in (CODES, 240, 2400): for hidden in (HIDDEN, 64): a, b = count_params(V, hidden), count_params(V, hidden, 3) print(f"{V:>11}{hidden:>13}{a:>11}{b:>15}{b / a:>8.3f}" f"{f'{V * hidden} / {3 * hidden}':>26}") ``` ``` codes hidden units one-hot embedding d=3 ratio multiplications per step 24 6 157 103 0.656 144 / 18 24 64 1665 393 0.236 1536 / 192 240 6 1453 751 0.517 1440 / 18 240 64 15489 1041 0.067 15360 / 192 2400 6 14413 7231 0.502 14400 / 18 2400 64 153729 7521 0.049 153600 / 192 ``` In the one-hot setup the parameter count grows as the **product** of the number of codes and the number of hidden units. In the embedding the table grows in direct proportion to the number of codes, but the hidden layer depends only on the embedding dimension, so the second factor stays fixed. At 2,400 codes and 64 hidden units the ratio drops to 0.049. The last column reads the number of multiplications in the same way: multiplying by a one-hot vector requires 153,600 multiplications, and 153,536 of them are done with zero; reading a row from the table and doing 192 multiplications gives the same result. Only the first row was measured; the rest are values of the same formula. #### What the Learned Table Encoded The embedding raising the number is a result; which structure it found is a separate question. The task divides the codes into two groups, and if this distinction is visible in the table, the suspect codes should stand closer to one another than to the others. The measure is scale independent: the ratio of the mean distance between groups to the mean distance within groups. ```python def distance(u, v): return sum((u[i] - v[i]) ** 2 for i in range(len(u))) ** 0.5 def mean_pairwise(T, A, B): c = [distance(T[i], T[j]) for i in A for j in B if i != j] return sum(c) / len(c) SUSP = [c for c in range(CODES) if c in SUSPECT] OTHER = [c for c in range(CODES) if c not in SUSPECT] print(f"suspect codes {len(SUSP)}, others {len(OTHER)}") print(f"{'table':<26}{'run':>7}{'within suspect':>16}{'within other':>14}" f"{'between groups':>16}{'separation ratio':>18}") for name in ("embedding, learned", "embedding, frozen"): for k in (0, 1): T = NETS[name][k]["T"] i, o, a = (mean_pairwise(T, SUSP, SUSP), mean_pairwise(T, OTHER, OTHER), mean_pairwise(T, SUSP, OTHER)) print(f"{name if k == 0 else '':<26}{k + 1:>7}{i:>16.4f}{o:>14.4f}" f"{a:>16.4f}{a / ((i + o) / 2):>18.4f}") ``` ``` suspect codes 8, others 16 table run within suspect within other between groups separation ratio embedding, learned 1 0.7629 0.8020 3.0634 3.9153 2 0.8673 0.9887 3.0602 3.2977 embedding, frozen 1 0.9898 1.0181 1.0771 1.0728 2 1.0155 1.1029 1.0766 1.0164 ``` In the learned table the two groups are separated from each other: the mean distance between groups is 3.9153 and 3.2977 times the within-group mean. In the frozen table the same ratio is 1.0728 and 1.0164 — that is, the within-group and between-group distances are almost equal, because three randomly chosen directions have scattered the codes in an order the task does not recognize. There is a difference of 0.6176 between the separation ratios of the two runs, and both runs produce 1.0000 accuracy; where exactly the table is placed depends on the run, but whether it makes the distinction does not. #### Summary - Giving a one-hot vector to a linear layer is the same operation as reading a row from an embedding table; the difference is in the parameter count and the multiplication count. - On the same body, the learned embedding gives 0.9722 and 0.9833 at 150 epochs with 103 parameters; the one-hot setup with 157 parameters stays at 0.7167 and 0.7000 in the same epochs and reaches a similar figure only at 600 epochs. - The table of the same size frozen without looking at the task stays between 0.68 and 0.75 and adding epochs does not fix this; the projection of the Unsupervised Learning course was in the same position and was measured there with the reconstruction error. - The ordinal numeric encoding gives exactly the baseline, 0.6444, in all four epoch budgets, because the order of the code numbers is unrelated to the label. - In the learned table the distance between groups is 3.9153 and 3.2977 times the within-group distance; in the frozen table it is 1.0728 and 1.0164 times, that is, there is no distinction. #### Next Step In this lesson the embedding table was learned with the labels of the task itself, and two hundred training examples were enough for it. Where the label is expensive, this assumption fails: if there are twenty or forty labeled sequences, learning all hundred and three parameters from those forty examples ends in overfitting. There is another way, and the structure built in this lesson uses it directly — the table can be learned on another task and carried over to this one. The next lesson produces a pretrained model by hand from a fictional source task, transfers the weights either by freezing or by fine-tuning, and prints the difference the transfer adds together with **how many labeled examples** it adds it with, against the same baseline. ### Transfer Learning Source: https://academia.sh/en/courses/deep-learning/transfer-learning Measuring what carrying a pretrained model, produced by our own hands from a fictional source task, over to a target task adds: while the most-frequent-class baseline is 0.6444, training from scratch with twenty labels gives 0.7194, transfer with freezing 0.8417 and transfer with fine-tuning 0.8083. Freezing stays flat at around 0.83 in all four measurements and training from scratch passes it between forty and eighty examples; fine-tuning stays above training from scratch at three of the four points, but at one hundred sixty examples the difference falls to 0.0028. The table from the source task carries a maintenance separation of 3.6373 and a suspect separation of 2.4767; after fine-tuning these two numbers become 2.4685 and 3.7991. In the previous lesson the embedding table was learned with the labels of the task itself, and two hundred examples were enough for it. Where the label is expensive, this assumption fails. A subscriber record marked as suspect is the equivalent of an hour an inspector spent in the field; there may be twenty or forty of them at hand. Learning all hundred and three parameters from twenty labels is a case where capacity exceeds the data, and `generalization` of the Introduction to Machine Learning course had measured this. There is one more way. The same codes also answer another question, and for that question labels are plentiful. The representation can be learned there and carried over here. The question of this lesson is twofold: which number does carrying it move, and **with how many labels**. The second question matters more than the first, because the value of transfer lies not in the final number but in how many labels it takes to reach that number. - **AR73.** The sequence setup of the Architectures topic was built in `architectures/06` and the event codes in `architectures/08`; they are **fictional**. In this lesson a second **fictional** label is added. Seed 20260218. - **AR74.** **The source task is fictional and its label is plentiful**: the presence of a **maintenance code** in at least six of the twelve steps. There are ten maintenance codes. - **AR75.** The **target task** is the task of `architectures/08`: a **suspect code** in at least five steps. There are eight suspect codes and all eight are among the maintenance codes; the maintenance set has two more codes besides them. The two tasks are related but not the same. - **AR76.** **The pretrained model is produced in this lesson.** No ready-made model, weight repository or external source is used; the source task is trained here with three hundred examples and is **a model**. - **AR77.** The source and the target work on **separate sequences**: the source is the first three hundred sequences, the target pool is the next two hundred forty, and the held-out set is the last one hundred eighty. The pretrained model never sees a single label of the target task. - **AR78.** Three setups are compared: **from scratch** (the table is randomly initialized), **transfer + freezing** (the table is taken from the source and never updated), **transfer + fine-tuning** (the table is taken from the source and updated). - **AR79.** The number of target labels is measured at four values: 20, 40, 80, 160. The held-out set is the same in all four. - **AR80.** The baseline is **the most frequent class** and is read from the held-out set; the baseline does not change when the label count changes. - **AR81.** Every number is printed with **two runs**; the runs differ in the initialization seed. #### The Source Task and the Pretrained Model The subject of transfer is not a file but a **representation**. What is carried here is the embedding table: the three-dimensional vectors of the twenty-four codes. The source task places this table according to its own question, and the target task inherits it. ```python # common.py — MODEL. The FICTIONAL sequence set of `architectures/06` + the FICTIONAL event codes of `08`. import math SEED, M32, STEPS = 20260218, 0xFFFFFFFF, 12 CODES = 24 MAINT = (1, 2, 4, 5, 7, 9, 13, 16, 20, 22) # FICTIONAL source task: maintenance codes SUSPECT = (1, 4, 5, 9, 13, 16, 20, 22) # FICTIONAL target task: suspect codes def generator(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_sequences(n, seed): D = [] for i in range(n): r = generator(seed + i) r() [round((r() + r() + r() - 1.5) * 1.9, 3) for _ in range(STEPS - 1)] r() o = generator(seed + 8000 + i) codes = [int(o() * CODES) for _ in range(STEPS)] D.append({"codes": codes, "source": 1 if sum(c in MAINT for c in codes) >= 6 else 0, "y": 1 if sum(c in SUSPECT for c in codes) >= 5 else 0}) return D DATA = build_sequences(720, SEED) SOURCE, POOL, TEST = DATA[:300], DATA[300:540], DATA[540:] rate = sum(x["y"] for x in TEST) / len(TEST) BASELINE = max(rate, 1 - rate) SHARED = [c for c in SUSPECT if c in MAINT] print(f"codes {CODES}, maintenance {len(MAINT)}, suspect {len(SUSPECT)}, shared {len(SHARED)}") print(f"source task rate {sum(x['source'] for x in SOURCE) / len(SOURCE):.4f}, " f"source examples {len(SOURCE)}") print(f"target task rate held-out {rate:.4f}, baseline {BASELINE:.4f}, " f"pool {len(POOL)}") ``` ``` codes 24, maintenance 10, suspect 8, shared 8 source task rate 0.3533, source examples 300 target task rate held-out 0.3556, baseline 0.6444, pool 240 ``` The relation between the two tasks is in a single line: all eight suspect codes are in the maintenance set, but the maintenance set has two extra codes. A representation that solves the source task carries the separation the target task wants approximately; it does not carry it exactly. ```python HIDDEN, DIM = 6, 3 def represent(T, codes): h = [0.0] * DIM for c in codes: for j in range(DIM): h[j] += T[c][j] / STEPS return h def new_table(seed): r = generator(seed) return [[(r() + r() + r() - 1.5) * 1.1 for _ in range(DIM)] for _ in range(CODES)] def train_model(seed, T, D, field, epochs, learn, lr=0.9): r = generator(seed + 500) g = lambda: (r() + r() + r() - 1.5) * 0.9 W1 = [[g() for _ in range(DIM)] for _ in range(HIDDEN)] b1, w, b, n = [0.0] * HIDDEN, [g() for _ in range(HIDDEN)], [0.0], len(D) for _ in range(epochs): gW1 = [[0.0] * DIM for _ in range(HIDDEN)] gb1, gw, gb = [0.0] * HIDDEN, [0.0] * HIDDEN, 0.0 gT = [[0.0] * DIM for _ in range(CODES)] if learn else None for x in D: h = represent(T, x["codes"]) z = [sum(W1[i][j] * h[j] for j in range(DIM)) + b1[i] for i in range(HIDDEN)] o = sum(w[i] * z[i] for i in range(HIDDEN) if z[i] > 0) + b[0] df = 1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - x[field] gb += df da = [df * w[i] if z[i] > 0 else 0.0 for i in range(HIDDEN)] for i in range(HIDDEN): gw[i] += df * max(0.0, z[i]) gb1[i] += da[i] for j in range(DIM): gW1[i][j] += da[i] * h[j] if learn: # with freezing the gradient of the table is never computed dh = [sum(da[i] * W1[i][j] for i in range(HIDDEN)) for j in range(DIM)] for c in x["codes"]: for j in range(DIM): gT[c][j] += dh[j] / STEPS for i in range(HIDDEN): w[i] -= lr * gw[i] / n b1[i] -= lr * gb1[i] / n for j in range(DIM): W1[i][j] -= lr * gW1[i][j] / n b[0] -= lr * gb / n if learn: for c in range(CODES): for j in range(DIM): T[c][j] -= lr * gT[c][j] / n return {"T": T, "W1": W1, "b1": b1, "w": w, "b": b[0]} def accuracy(P, D, field="y"): s = 0 for x in D: h = represent(P["T"], x["codes"]) z = [sum(P["W1"][i][j] * h[j] for j in range(DIM)) + P["b1"][i] for i in range(HIDDEN)] o = sum(P["w"][i] * max(0.0, z[i]) for i in range(HIDDEN)) + P["b"] s += (o > 0) == (x[field] == 1) return s / len(D) def copy_table(T): return [u[:] for u in T] PRETRAINED = [] print(f"source task examples {len(SOURCE)}, epochs 400, embedding dimension {DIM}, " f"hidden units {HIDDEN}") for k in (1, 2): P = train_model(SEED + k, new_table(SEED + 900 + k), SOURCE, "source", 400, True) PRETRAINED.append(P) print(f"pretrained model run {k}: training on the source task " f"{accuracy(P, SOURCE, 'source'):.4f}, held-out " f"{accuracy(P, TEST, 'source'):.4f}") print(f"these two models never saw any label of the target task") ``` ``` source task examples 300, epochs 400, embedding dimension 3, hidden units 6 pretrained model run 1: training on the source task 1.0000, held-out 1.0000 pretrained model run 2: training on the source task 1.0000, held-out 1.0000 these two models never saw any label of the target task ``` #### Freezing, Fine-Tuning and How Many Labels Transfer has two forms. In **freezing** the carried table is fixed and only the body on top of it is learned: thirty-one parameters. In **fine-tuning** the table is updated as well: one hundred three parameters. The choice between the two depends on how many labels are at hand, and this dependence can itself be measured. ```python SIZES, SETUPS = (20, 40, 80, 160), (("from scratch", False, True), ("transfer + freezing", True, False), ("transfer + fine-tuning", True, True)) RESULT = {} print(f"target epochs 250, baseline {BASELINE:.4f}, held-out {len(TEST)}") print(f"{'setup':<24}{'learned':>10}{'run':>7}" + "".join(f"{'n=' + str(n):>9}" for n in SIZES)) for name, from_source, learn in SETUPS: RESULT[name] = {} for k in (1, 2): row = [] for n in SIZES: T = copy_table(PRETRAINED[k - 1]["T"]) if from_source else new_table(SEED + 900 + k) P = train_model(SEED + 30 + k, T, POOL[:n], "y", 250, learn) RESULT[name].setdefault(n, []).append(accuracy(P, TEST)) row.append(RESULT[name][n][-1]) if n == SIZES[-1] and k == 1: RESULT[name]["T"] = P["T"] print(f"{name if k == 1 else '':<24}" f"{HIDDEN * DIM + 2 * HIDDEN + 1 + (0 if not learn else CODES * DIM):>10}" f"{k:>7}" + "".join(f"{v:>9.4f}" for v in row)) ``` ``` target epochs 250, baseline 0.6444, held-out 180 setup learned run n=20 n=40 n=80 n=160 from scratch 103 1 0.7222 0.8056 0.8167 0.9556 103 2 0.7167 0.8389 0.8611 0.9722 transfer + freezing 31 1 0.8333 0.8278 0.8389 0.8333 31 2 0.8500 0.8444 0.8278 0.8278 transfer + fine-tuning 103 1 0.8111 0.8556 0.8556 0.9556 103 2 0.8056 0.8500 0.8556 0.9667 ``` The freezing row is a horizontal line: between 0.8278 and 0.8500, it hardly moves with the label count. Thirty-one parameters can be learned from twenty labels, but one hundred sixty labels do not find a new place to go in the same thirty-one parameters. The ceiling of transfer is in the carried table. ```python def mean_acc(name, n): return sum(RESULT[name][n]) / 2 print(f"{'examples':>9}{'from scratch':>14}{'freezing':>10}{'fine-tuning':>13}" f"{'freezing difference':>21}{'fine-tuning difference':>24}") for n in SIZES: a, b, c = (mean_acc(k, n) for k, *_ in SETUPS) print(f"{n:>9}{a:>14.4f}{b:>10.4f}{c:>13.4f}{b - a:>+21.4f}{c - a:>+24.4f}") target = mean_acc("transfer + freezing", 20) enough = [n for n in SIZES if mean_acc("from scratch", n) >= target] print(f"freezing with 20 examples {target:.4f}; from scratch reaches the same " f"figure with {enough[0] if enough else '>160'} examples") def pairwise(T, P, Q): d = [sum((T[i][j] - T[k][j]) ** 2 for j in range(DIM)) ** 0.5 for i in P for k in Q if i != k] return sum(d) / len(d) def separation(T, G): A = [c for c in range(CODES) if c in G] B = [c for c in range(CODES) if c not in G] return pairwise(T, A, B) / ((pairwise(T, A, A) + pairwise(T, B, B)) / 2) print(f"{'table':<28}{'maintenance separation':>24}{'suspect separation':>20}") for name, T in (("from the source task", PRETRAINED[0]["T"]), ("after fine-tuning, n=160", RESULT["transfer + fine-tuning"]["T"]), ("from scratch, n=160", RESULT["from scratch"]["T"])): print(f"{name:<28}{separation(T, MAINT):>24.4f}{separation(T, SUSPECT):>20.4f}") ``` ``` examples from scratch freezing fine-tuning freezing difference fine-tuning difference 20 0.7194 0.8417 0.8083 +0.1222 +0.0889 40 0.8222 0.8361 0.8528 +0.0139 +0.0306 80 0.8389 0.8333 0.8556 -0.0056 +0.0167 160 0.9639 0.8306 0.9611 -0.1333 -0.0028 freezing with 20 examples 0.8417; from scratch reaches the same figure with 160 examples table maintenance separation suspect separation from the source task 3.6373 2.4767 after fine-tuning, n=160 2.4685 3.7991 from scratch, n=160 2.1087 3.3228 ``` The first table reads the difference that transfer adds together with the number of examples and does not give a single number. With twenty labels freezing stands 0.1222 above training from scratch; with forty the difference drops to 0.0139; with eighty the sign turns, and with one hundred sixty freezing falls **below** training from scratch by 0.1333. Fine-tuning is above at three of the four points, but at one hundred sixty its difference falls to 0.0028, that is, within the measurement noise. The second line is the real number of this lesson. Freezing gives 0.8417 with twenty labels; training from scratch reaches the same figure only with one hundred sixty labels — eight times as many. What transfer buys is not the final number but the **labels that are not collected** to reach it. If one field visit is assumed per label, the unit of the measure is not accuracy but one hundred forty visits. The minus sign in the one hundred sixty row deserves a pause. There, transfer with freezing is beaten by the setup that transfers nothing; the carried representation has stopped being a starting point and has become a constraint. Transfer does not always pay off, and whether it pays off is known only when the setup without transfer is run on the same held-out set. The baseline of this measure is not the most frequent class but **training from scratch itself**. Run-to-run variability determines the reading of this lesson in two places. With twenty labels the difference between the runs of the three setups is at most 0.0167, while what transfer adds is 0.1222 — that is, the difference is seven times the run-to-run variability and counts as measured. With one hundred sixty, however, the difference of fine-tuning from training from scratch is 0.0028, and the two runs of the same setup differ by 0.0111; there the difference is lost inside the run, and saying "fine-tuning is a little better" is an unmeasured sentence. The third table says why transfer stops somewhere. The table from the source carries a maintenance separation of 3.6373 but a suspect separation of only 2.4767; the two extra maintenance codes have settled next to the suspects in the table, and a frozen table cannot correct this. Fine-tuning swaps the two numbers: the maintenance separation drops to 2.4685 while the suspect separation rises to 3.7991. The table trained from scratch also brings the suspect separation to 3.3228, but it spends eight times the labels to do so. #### Summary - The pretrained model was produced in this lesson from a fictional source task; on the source task both runs give 1.0000 and the model never saw a single label of the target task. - With twenty labels the baseline is 0.6444, training from scratch 0.7194, freezing 0.8417 and fine-tuning 0.8083; at this point transfer adds a difference of up to 0.1222. - Freezing learns thirty-one parameters and does not move with the label count: it stays flat at around 0.83 in all four measurements and at one hundred sixty labels falls 0.1333 below training from scratch. - The measure of transfer is not accuracy but the number of labels: training from scratch reaches the 0.8417 that freezing gives with twenty labels only with one hundred sixty labels. - The cause of the ceiling is the carried representation: the source table carries a maintenance separation of 3.6373 and a suspect separation of 2.4767; after fine-tuning these two numbers become 2.4685 and 3.7991. #### Next Step In nine lessons every architecture predicted a **label**. The input was given, the output was reduced to a single number, and the measure was always of the same form: how many correct on the held-out set. A network can have another job besides predicting a label — reproducing the input itself, or building new examples that resemble the input. What is the measure of such a network, because there is no label to compare against? The next lesson takes up two generative arrangements that work over a latent space: the autoencoder, measured by the reconstruction error (with reference to the measure of the Unsupervised Learning course), and the adversarial arrangement in which two networks are trained against each other; in the latter, what is measured is not performance but how far apart two runs of the same arrangement end up. ### Generative Models Source: https://academia.sh/en/courses/deep-learning/generative-models Measuring two generative arrangements that work without labels on the same fictional sequence set: while the baseline of reconstruction with the mean sequence is 2.9007, the linear autoencoder gives 1.2817 and 1.2815 with one latent dimension and 0.7346 and 0.7314 with four; the error of the two runs differs by at most 0.0311, while the directions the same runs found differ between 1.6330 and 3.4005. In the adversarial arrangement the accuracy of the discriminator on the real sequences falls from 0.6133 to 0.3467 and returns to 0.4600, and the step means produced by the two runs move 1.2707 away from each other — more than the deviation of each from the real set, 0.5321 and 0.8877. In nine lessons every architecture predicted a label, and the measure was always of the same form: how many correct on the held-out set. The job of a network may not be to predict a label. Reconstructing the input itself from a few numbers, or producing new sequences that resemble the input, is also a task, and in neither is there a label to compare against. The question of this lesson is about the measure: which number is written when there is no label. Two arrangements are taken up. The first reconstructs the input from a **latent space**, and its error can be measured directly. In the second two networks are trained against each other and there is not even a single error; what is measured there is not performance but how far apart two runs of the same arrangement end up. - **AR82.** The sequence setup of the Architectures topic was built in `architectures/06` and is **fictional**. In this lesson the sequence is used in its **raw** form: subscriber level, periodic component and the residual of the earlier lessons. The earlier lessons took the sequence scaled as a deviation; the compressible structure lies exactly in the components extracted there. - **AR83.** The columns are scaled **from the training set only** and the error is in units of standard deviation. This is the same unit as `dimensionality-reduction` in the Unsupervised Learning course. - **AR84.** The baseline is **reconstruction with the mean sequence**, which is the case where the latent dimension is zero. This is the plainest reconstruction that does not learn. - **AR85.** **The reconstruction error procedure was built in the Unsupervised Learning course and is not repeated here.** There the directions were computed from the covariance matrix, and against the 3.3411 of reconstructing with the column means, two components had given 2.7762. Here the directions are not computed; they are **learned with gradient descent**. - **AR86.** The autoencoder is a linear encoder and a linear decoder; because the data is centered, no bias term is written. It is written inside the lesson and is **a model**. - **AR87.** The **adversarial generative network** is an arrangement built from two networks: the **generator** builds a sequence from a latent vector, the **discriminator** reduces a sequence to a single number. The two are updated in turn. - **AR88.** The measure of the adversarial arrangement is not accuracy. The step means and spreads of the generated sequences are compared with those of the real set, and how far apart **the two runs** end up is printed. - **AR89.** This lesson takes up only **the structure and the measure**. No production application, forgery or detection-evasion procedure is written; these topics are left to the defensive framework of the Cybersecurity and Application Security and DevSecOps curricula. - **AR90.** Every number is printed with **two runs**; the runs differ in the initialization seed. #### Autoencoder and Reconstruction Error The autoencoder is two parts. The encoder reduces the twelve-number sequence to k numbers, the decoder reconstructs twelve from those k numbers, and training adjusts the two together to make the reconstruction error smaller. There is no label; the target is the input itself. ```python # common.py — MODEL. The FICTIONAL sequence of `architectures/06` is built in its RAW form: # subscriber level + periodic component + the residual of the earlier lessons. import math SEED, M32, STEPS = 20260218, 0xFFFFFFFF, 12 def generator(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def build_sequences(n, seed): D = [] for i in range(n): r = generator(seed + i) r() residual = [0.0] + [round((r() + r() + r() - 1.5) * 1.9, 3) for _ in range(STEPS - 1)] r() s = generator(seed + 6000 + i) level = 21.0 + (s() + s() + s() - 1.5) * 6.0 a, b = (s() + s() - 1.0) * 2.6, (s() + s() - 1.0) * 2.6 angle = [2 * math.pi * t / STEPS for t in range(STEPS)] D.append([round(level + a * math.sin(angle[t]) + b * math.cos(angle[t]) + residual[t], 3) for t in range(STEPS)]) return D DATA = build_sequences(720, SEED) TRAIN, TEST = DATA[:150], DATA[540:] MEAN = [sum(x[t] for x in TRAIN) / len(TRAIN) for t in range(STEPS)] SPREAD = [max(1e-9, (sum((x[t] - MEAN[t]) ** 2 for x in TRAIN) / len(TRAIN)) ** 0.5) for t in range(STEPS)] ZTRAIN = [[(x[t] - MEAN[t]) / SPREAD[t] for t in range(STEPS)] for x in TRAIN] ZTEST = [[(x[t] - MEAN[t]) / SPREAD[t] for t in range(STEPS)] for x in TEST] def distance(u, v): return sum((u[t] - v[t]) ** 2 for t in range(STEPS)) ** 0.5 ZERO = [0.0] * STEPS BASELINE = sum(distance(z, ZERO) for z in ZTEST) / len(ZTEST) print(f"sequences {len(DATA)}, length {STEPS}, training {len(TRAIN)}, held-out {len(TEST)}") print(f"example raw sequence {DATA[0]}") print(f"baseline (reconstruction with the mean sequence) {BASELINE:.4f}") ``` ``` sequences 720, length 12, training 150, held-out 180 example raw sequence [21.188, 21.247, 19.977, 20.223, 21.497, 22.188, 24.826, 24.397, 24.089, 25.549, 22.073, 22.529] baseline (reconstruction with the mean sequence) 2.9007 ``` Under the setup there are three components: the subscriber level and the two coefficients of the periodic component. Even if twelve numbers are stored, what the sequence carries adds only noise on top of these three. This is what the autoencoder is expected to find, and at which k it finds it can be measured. ```python def autoencoder(seed, k, epochs=300, lr=0.3): # the data is centered with the column means, so no bias term is written; # the k=0 case is exactly the baseline. r = generator(seed) g = lambda: (r() + r() + r() - 1.5) * 0.7 W1 = [[g() for _ in range(STEPS)] for _ in range(k)] W2 = [[g() for _ in range(k)] for _ in range(STEPS)] n = len(ZTRAIN) for _ in range(epochs): gW1 = [[0.0] * STEPS for _ in range(k)] gW2 = [[0.0] * k for _ in range(STEPS)] for z in ZTRAIN: h = [sum(W1[i][t] * z[t] for t in range(STEPS)) for i in range(k)] y = [sum(W2[t][i] * h[i] for i in range(k)) for t in range(STEPS)] d = [2.0 * (y[t] - z[t]) / STEPS for t in range(STEPS)] for t in range(STEPS): for i in range(k): gW2[t][i] += d[t] * h[i] for i in range(k): dh = sum(d[t] * W2[t][i] for t in range(STEPS)) for t in range(STEPS): gW1[i][t] += dh * z[t] for i in range(k): for t in range(STEPS): W1[i][t] -= lr * gW1[i][t] / n for t in range(STEPS): for i in range(k): W2[t][i] -= lr * gW2[t][i] / n return {"k": k, "W1": W1, "W2": W2} def reconstruct(P, z): h = [sum(P["W1"][i][t] * z[t] for t in range(STEPS)) for i in range(P["k"])] return [sum(P["W2"][t][i] * h[i] for i in range(P["k"])) for t in range(STEPS)] def error(P, Z): return sum(distance(z, reconstruct(P, z)) for z in Z) / len(Z) def direction_difference(A, B): # distance between the normalized encoder directions of the two runs f = 0.0 for i in range(A["k"]): u, v = A["W1"][i], B["W1"][i] nu = sum(q * q for q in u) ** 0.5 nv = sum(q * q for q in v) ** 0.5 f += min(sum(abs(u[t] / nu - v[t] / nv) for t in range(STEPS)), sum(abs(u[t] / nu + v[t] / nv) for t in range(STEPS))) return f / A["k"] print(f"baseline (k=0) {BASELINE:.4f}, epochs 300, training {len(ZTRAIN)}") print(f"{'k':>3}{'parameters':>11}{'training 1':>12}{'training 2':>12}{'held-out 1':>12}" f"{'held-out 2':>12}{'run difference':>16}{'direction difference':>22}") for k in (1, 2, 3, 4): A, B = autoencoder(SEED + 1, k), autoencoder(SEED + 2, k) a, b = error(A, ZTEST), error(B, ZTEST) print(f"{k:>3}{2 * STEPS * k:>11}{error(A, ZTRAIN):>12.4f}{error(B, ZTRAIN):>12.4f}" f"{a:>12.4f}{b:>12.4f}{abs(a - b):>16.4f}{direction_difference(A, B):>22.4f}") ``` ``` baseline (k=0) 2.9007, epochs 300, training 150 k parameters training 1 training 2 held-out 1 held-out 2 run difference direction difference 1 24 1.3001 1.2999 1.2817 1.2815 0.0003 1.6330 2 48 1.0322 1.0278 1.0597 1.0576 0.0021 2.8634 3 72 0.7936 0.7502 0.7949 0.7638 0.0311 3.4005 4 96 0.7219 0.7123 0.7346 0.7314 0.0031 3.3412 ``` The baseline is 2.9007. A single latent number brings the error down to 1.2817 and three numbers to 0.7949; going from the third to the fourth the drop is about 0.06, because the structure the setup carries ends at three components and the fourth direction sits largely on noise. The Unsupervised Learning course had taken the same measure with computed directions; the autoencoder here does not compute the directions, it searches for them with gradient descent, and it goes below the baseline down to 0.7346. The last two columns are the reading of this course. The error of the two runs is nearly the same in all four rows: the largest difference is 0.0311 at k=3. By contrast, the directions the same two runs found differ between 1.6330 and 3.4005 — for normalized directions of twelve components this is a large distance. The autoencoder finds a **subspace**, and that subspace is repeated from run to run; the **axes** inside the subspace are not repeated. Anyone who wants to name a latent dimension must know that what they name changes with the seed. #### The Adversarial Generative Network and Instability In the adversarial generative network there is no input to reconstruct. The generator builds a sequence from a latent vector, the discriminator tries to tell the real from the produced, and the loss of the generator is the discriminator being wrong. The goals of the two networks are the reverse of each other; for this reason the loss of neither side improves on its own, and the sentence "the loss went down" is not a sign of progress here. ```python LATENT, GEN_HIDDEN, DISC_HIDDEN, STOPS = 3, 6, 6, (30, 60, 90, 120) def adversarial(seed, epochs=120, lr=0.6): r = generator(seed) g = lambda: (r() + r() + r() - 1.5) * 0.8 GEN = {"W1": [[g() for _ in range(LATENT)] for _ in range(GEN_HIDDEN)], "b1": [[0.0] for _ in range(GEN_HIDDEN)], "W2": [[g() for _ in range(GEN_HIDDEN)] for _ in range(STEPS)], "b2": [[0.0] for _ in range(STEPS)]} DISC = {"W1": [[g() for _ in range(STEPS)] for _ in range(DISC_HIDDEN)], "b1": [[0.0] for _ in range(DISC_HIDDEN)], "w": [[g()] for _ in range(DISC_HIDDEN)], "b": [[0.0]]} z, N, trace = generator(seed + 77), len(ZTRAIN), [] def generate(v): # generator: from latent vector to sequence h = [math.tanh(sum(GEN["W1"][i][j] * v[j] for j in range(LATENT)) + GEN["b1"][i][0]) for i in range(GEN_HIDDEN)] return h, [sum(GEN["W2"][t][i] * h[i] for i in range(GEN_HIDDEN)) + GEN["b2"][t][0] for t in range(STEPS)] def discriminate(x): # discriminator: from sequence to a single number u = [sum(DISC["W1"][i][t] * x[t] for t in range(STEPS)) + DISC["b1"][i][0] for i in range(DISC_HIDDEN)] return u, sum(DISC["w"][i][0] * u[i] for i in range(DISC_HIDDEN) if u[i] > 0) + DISC["b"][0][0] def descend(P, grad): for key in P: for i in range(len(P[key])): for j in range(len(P[key][i])): P[key][i][j] -= lr * grad[key][i][j] def zeros(P): return {key: [[0.0] * len(u) for u in P[key]] for key in P} for t in range(1, epochs + 1): Z = [[(z() + z() + z() - 1.5) * 1.6 for _ in range(LATENT)] for _ in range(N)] FAKE = [generate(v)[1] for v in Z] gD = zeros(DISC) for x, y in [(u, 1) for u in ZTRAIN] + [(u, 0) for u in FAKE]: u, o = discriminate(x) d = (1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - y) / (2 * N) gD["b"][0][0] += d for i in range(DISC_HIDDEN): gD["w"][i][0] += d * max(0.0, u[i]) du = d * DISC["w"][i][0] if u[i] > 0 else 0.0 gD["b1"][i][0] += du for j in range(STEPS): gD["W1"][i][j] += du * x[j] gG = zeros(GEN) for v in Z: h, x = generate(v) u, o = discriminate(x) d = (1.0 / (1.0 + math.exp(-max(-30.0, min(30.0, o)))) - 1) / N dx = [sum(d * DISC["w"][i][0] * DISC["W1"][i][j] for i in range(DISC_HIDDEN) if u[i] > 0) for j in range(STEPS)] for j in range(STEPS): gG["b2"][j][0] += dx[j] for i in range(GEN_HIDDEN): gG["W2"][j][i] += dx[j] * h[i] for i in range(GEN_HIDDEN): dh = sum(dx[j] * GEN["W2"][j][i] for j in range(STEPS)) * (1 - h[i] * h[i]) gG["b1"][i][0] += dh for j in range(LATENT): gG["W1"][i][j] += dh * v[j] descend(DISC, gD) descend(GEN, gG) if t in STOPS: trace.append((t, sum(discriminate(x)[1] > 0 for x in ZTRAIN) / N, sum(discriminate(x)[1] <= 0 for x in FAKE) / N)) mean = [sum(x[t] for x in FAKE) / N for t in range(STEPS)] return trace, mean, [(sum((x[t] - mean[t]) ** 2 for x in FAKE) / N) ** 0.5 for t in range(STEPS)] ``` ```python REAL_MEAN = [sum(x[t] for x in ZTRAIN) / len(ZTRAIN) for t in range(STEPS)] REAL_SPREAD = [(sum((x[t] - REAL_MEAN[t]) ** 2 for x in ZTRAIN) / len(ZTRAIN)) ** 0.5 for t in range(STEPS)] RUNS = [adversarial(SEED + j) for j in (1, 2)] print(f"{'epoch':>5}" + "".join(f"{'r' + str(j + 1) + ' real':>12}{'r' + str(j + 1) + ' fake':>11}" for j in (0, 1))) for i, (t, _real, _fake) in enumerate(RUNS[0][0]): print(f"{t:>5}" + "".join(f"{RUNS[j][0][i][1]:>12.4f}{RUNS[j][0][i][2]:>11.4f}" for j in (0, 1))) def deviation(a, b): return sum(abs(a[t] - b[t]) for t in range(STEPS)) / STEPS print(f"{'measure':<30}{'run 1':>10}{'run 2':>10}{'difference between runs':>25}") print(f"{'deviation of mean from real':<30}{deviation(RUNS[0][1], REAL_MEAN):>10.4f}" f"{deviation(RUNS[1][1], REAL_MEAN):>10.4f}{deviation(RUNS[0][1], RUNS[1][1]):>25.4f}") print(f"{'deviation of spread from real':<30}{deviation(RUNS[0][2], REAL_SPREAD):>10.4f}" f"{deviation(RUNS[1][2], REAL_SPREAD):>10.4f}{deviation(RUNS[0][2], RUNS[1][2]):>25.4f}") ``` ``` epoch r1 real r1 fake r2 real r2 fake 30 0.6133 0.8067 0.5200 1.0000 60 0.3933 0.7800 0.6267 1.0000 90 0.3467 0.6600 0.5800 1.0000 120 0.4600 0.8533 0.6400 0.9333 measure run 1 run 2 difference between runs deviation of mean from real 0.5321 0.8877 1.2707 deviation of spread from real 0.3669 0.5434 0.3418 ``` The first table shows the instability of the arrangement directly. In the first run the accuracy of the discriminator on the real sequences falls from 0.6133 to 0.3467 and then returns to 0.4600; meanwhile the rate of recognizing the generated ones falls from 0.8067 to 0.6600 and rises to 0.8533. In the second run the discriminator catches all of the generated ones in three of the measurements — there the generator made no progress at all. At which epoch one stops changes the result, and there is no one-directional curve that would tell the stopping point. The second table is the closing number of this lesson. The step means produced by the two runs deviate from the real set by 0.5321 and 0.8877, but they deviate from **each other** by 1.2707. The distance between the two runs is larger than the distance of each to the target; that is, the two runs have gone not toward the same place but to two separate places. In the autoencoder the same comparison did not exceed 0.0311. The difference in stability comes not from the capacity of the arrangement but from the form of its target: minimizing a fixed target and fooling a moving opponent are not the same problem. #### Summary - There is a measure even without labels: the baseline of the autoencoder is reconstruction with the mean sequence and is 2.9007; this baseline is the case where the latent dimension is zero. - One latent dimension brings the error down to 1.2817 and three dimensions to 0.7949; the fourth dimension adds only about 0.06, because the structure of the setup ends at three components. - While the reconstruction error of the two runs differs by at most 0.0311, the directions they find differ between 1.6330 and 3.4005: the subspace is repeated, the axes are not. - In the adversarial arrangement the accuracy of the discriminator on the real sequences moves between 0.6133 and 0.3467 and gives no one-directional curve; in the second run the generator makes no progress at all. - The step means produced by the two runs differ from each other by 1.2707 — more than the deviation of each from the real set, 0.5321 and 0.8877. #### Next Step In ten lessons the architecture was chosen. Convolution used the grid structure of the input, recurrence and gated units the sequence structure, attention and multi-head attention direct access to distant steps, embedding the discrete symbols, transfer a representation from another task, and the generative arrangements an unlabeled target. The difference each added was measured against its baseline and the second run was written next to every number. All ten of these lessons have one thing in common that was never asked: all of them ran in a small training loop written by hand. The gradient was derived on this page every time, the data fit in memory, the batch size was equal to the number of samples, the seed stood in a single place and no record of any run was kept. The next topic takes up not the architecture but the **setup** of training: deriving the gradient from a computation graph instead of by hand, producing the data in batches, the relation of memory to batch size, the reproducible record of runs, and how the search for settings goes wrong with run-to-run variability. ## Training Practice Measuring how much the training setup moves the number: the gradient difference between the computation graph and automatic differentiation and the hand-written backpropagation and the code each one takes, the difference data augmentation adds and the case where augmentation produces leakage, the batch generation order moving the result, the relation of batch size to the number of stored values for parameters, activations and gradients and the need for it to change together with the learning rate, the items to pin that are specific to training and how much the result moves when one of them is released, what the search selects when run-to-run variability is larger than the difference between candidates. ### Computation Graph and Automatic Differentiation Source: https://academia.sh/en/courses/deep-learning/computation-graph-and-automatic-differentiation Writing, within the lesson, the abstraction that separates gradient computation from the architecture and comparing it with a hand-written backpropagation: a thirty-two-line machinery with four operations gives the gradient of eighty-one parameters with a difference of 2.78e-17 from the eighteen-line hand-written backpropagation and builds 6,622 nodes for a batch of thirty examples. The course's three numbers are set up here: the baseline 0.7579, the network's number 0.8095 and the second run, which differs only in the initial values, 0.7937. The range of ten initializations is between 0.7857 and 0.8095, and the numbers of the Supervised Learning course's decision tree (0.7857) and linear model (0.8016) fall within this range. In the previous ten lessons the architecture was chosen: convolution, pooling, skip connection, recurrent state, gated unit, attention, embedding, transfer, autoencoder. In each one the structure of the network changed and the gradient computation was **re-derived and rewritten**; all of them ran in a small hand-written training loop. The **machinery** of real training was never addressed. This topic addresses that machinery, and its first piece is the abstraction that separates gradient computation from the architecture. In the catalog this lesson mentions deep learning libraries; what is described here is **not libraries but the abstraction libraries sell** — the computation graph and automatic differentiation — and the abstraction is written within the lesson. The measure is three numbers: the **difference** between the gradient given by the hand-written backpropagation and by automatic differentiation, **how many lines** the two routes take, and how many **nodes** the graph builds. - **TP1.** The dataset is the **fictional** subscriber table of the Introduction to Machine Learning, Data Preparation and Feature Engineering, Supervised Learning and Unsupervised Learning courses, seed 20260218, 1,260 subscribers; the label is suspected illicit use or a fault and is fictional. The baseline is **saying the most frequent class**: 0.7579 on the test set, inherited from the Introduction to Machine Learning, Data Preparation and Feature Engineering and Supervised Learning courses. - **TP2.** The training set in this topic is **300 rows instead of 756**; it is a scale decision, and every block of the topic runs in a few seconds. The test set is 252 rows and one subscriber is worth 0.0040. - **TP3.** The network is fixed throughout this topic: eight features, a single hidden layer of eight units (rectified linear unit), a single output; the loss is **cross-entropy**. The constant term of the layer is called the **bias** and is distinct from the **bias** in statistics. Scaling is learned from the **training set** only (the Data Preparation and Feature Engineering course, pipeline). - **TP4.** The third number of the course is the **second run**: the same network, the same data, the same example order; only the **initial values** change. The measure is not raw time: the cost of the graph is measured in **nodes**, the cost of the code in **lines**. - **TP5.** The loss node is **fused**: sigmoid and cross-entropy are held in a single node, because held separately they overflow on large outputs. Its local derivative is the same on both routes. The machinery is **a model**, and no framework, library or vendor name is mentioned. #### The Dataset, Scaling and the Baseline ```python # COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through # Unsupervised Learning courses is generated with the same seed (20260218); # the label is suspected illicit use or a fault. The training set is REDUCED in this topic # from 756 to 300 rows: every block of the topic runs in a few seconds. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "mean"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def mix(items, seed): r, s = rng(seed), list(items) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue subscriber_id = 10001 + i r, v = rng(SEED + 7000 + subscriber_id), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean": round(mean, 2), "suspicious": int(z > 1.35)} for q in REGION[1:]: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) K = mix(DATA, SEED + 90000) TRAIN, TEST = K[:300], K[1008:] def table(rows, stats=None): X = [[float(s[a]) for a in FIELDS] for s in rows] if stats is None: # scaling is learned from the training set only stats = [] for j in range(4, 8): m = sum(v[j] for v in X) / len(X) stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X)))) for v in X: for j, m, sd in stats: v[j] = (v[j] - m) / sd return X, [s["suspicious"] for s in rows], stats X_train, y_train, STATS = table(TRAIN) X_test, y_test, _ = table(TEST, STATS) BASELINE = sum(v == 0 for v in y_test) / len(y_test) print(f"fictional subscribers {len(DATA)}; train {len(X_train)} (reduced from 756), test " f"{len(X_test)}, features {len(FIELDS)}, suspicious share in train " f"{sum(y_train) / len(y_train):.4f}") print(f"baseline (predicting the most frequent class, on the test set) {BASELINE:.4f}") ``` ``` fictional subscribers 1260; train 300 (reduced from 756), test 252, features 8, suspicious share in train 0.2200 baseline (predicting the most frequent class, on the test set) 0.7579 ``` #### A Graph Is a Data Structure The trouble with deriving by hand is not that the computation is hard but that it is **tied to the structure**: when a layer is added, the derivative is rewritten. The computation graph breaks that tie. Every intermediate value is kept as a **node**, and a node stores its own value, its **parents** and its **local derivative with respect to its parents**; the backward pass walks the graph in reverse and distributes the gradient of each node to its parents. The chain rule is this distribution, and it is written only once. ```python # Computation graph machinery — MODEL. Four operations: addition, multiplication, rectification # and fused loss. Each node holds its value, its gradient, its parents and its LOCAL DERIVATIVE # WITH RESPECT TO ITS PARENTS; the backward step uses only these. import inspect class Value: __slots__ = ("v", "grad", "parents", "local") def __init__(self, v, parents=(), local=()): self.v, self.grad, self.parents, self.local = float(v), 0.0, parents, local def __add__(self, o): o = o if isinstance(o, Value) else Value(o) return Value(self.v + o.v, (self, o), (1.0, 1.0)) def __mul__(self, o): o = o if isinstance(o, Value) else Value(o) return Value(self.v * o.v, (self, o), (o.v, self.v)) def rectify(self): return Value(self.v if self.v > 0 else 0.0, (self,), (1.0 if self.v > 0 else 0.0,)) def loss(self, y): # cross-entropy, directly on the output p = 1.0 / (1.0 + math.exp(-self.v)) if self.v > -30 else 0.0 k = max(0.0, self.v) + math.log(1.0 + math.exp(-abs(self.v))) - self.v * y return Value(k, (self,), (p - y,)) def backward(root): order, seen = [], set() def visit(d): if id(d) in seen: return seen.add(id(d)) for a in d.parents: visit(a) order.append(d) visit(root) root.grad = 1.0 for d in reversed(order): for a, y in zip(d.parents, d.local): a.grad += d.grad * y return len(order) LINES = sum(len([s for s in inspect.getsource(o).splitlines() if s.strip()]) for o in (Value, backward)) print(f"machinery {LINES} lines, 4 operations; L = 5 * rectify(a*b + c)") for b_value in (3.0, -3.0): a, b, c = Value(2.0), Value(b_value), Value(4.0) L = (a * b + c).rectify() * Value(5.0) n = backward(L) print(f" a=2 b={b_value:+.0f} c=4 -> L={L.v:5.1f}, {n} nodes; dL/da {a.grad:>6.1f}" f" dL/db {b.grad:>6.1f} dL/dc {c.grad:>6.1f}") ``` ``` machinery 32 lines, 4 operations; L = 5 * rectify(a*b + c) a=2 b=+3 c=4 -> L= 50.0, 8 nodes; dL/da 15.0 dL/db 10.0 dL/dc 5.0 a=2 b=-3 c=4 -> L= 0.0, 8 nodes; dL/da 0.0 dL/db 0.0 dL/dc 0.0 ``` Thirty-two lines and four operations. The derivatives can be checked by hand: $L = 5\,(ab+c)$ gives $\partial L/\partial a = 5b = 15$, $\partial L/\partial b = 5a = 10$ and $\partial L/\partial c = 5$ in the positive region; the output says so. In the second call only $b$ became negative three: the sum went negative, the rectifier gate closed, and **all three derivatives became zero**. The graph is the same, the code is the same; what changed is only the path the data opens in the graph. #### The Same Gradient, Two Routes ```python # The gradient of the same network by two routes: walking the graph and writing the chain # rule BY HAND. The parameters are one flat list: p[j*G+i] weight, p[H*G+j] bias, p[H*G+H+j] # and p[-1] output layer. G, H = len(FIELDS), 8 def init_weights(seed): # gain-scaled initialization r = rng(seed) return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(H * G)] + [0.0] * H \ + [(r() * 2 - 1) * math.sqrt(2.0 / H) for _ in range(H)] + [0.0] def by_graph(p, X, y, idx): nodes, total = [Value(v) for v in p], None for k in idx: x, o = X[k], nodes[-1] for j in range(H): z = nodes[H * G + j] for i in range(G): z = z + nodes[j * G + i] * x[i] o = o + nodes[H * G + H + j] * z.rectify() L = o.loss(y[k]) total = L if total is None else total + L root = total * (1.0 / len(idx)) return root, backward(root), [u.grad for u in nodes] def by_hand(p, X, y, idx): g, loss_sum = [0.0] * len(p), 0.0 for k in idx: x, hh = X[k], [0.0] * H for j in range(H): s = p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G)) hh[j] = s if s > 0 else 0.0 o = p[-1] + sum(p[H * G + H + j] * hh[j] for j in range(H)) loss_sum += max(0.0, o) + math.log(1.0 + math.exp(-abs(o))) - o * y[k] d = 1.0 / (1.0 + math.exp(-o)) - y[k] # local derivative of the loss node g[-1] += d for j in range(H): g[H * G + H + j] += d * hh[j] if hh[j] > 0: # a closed rectifier gate does not pass it g[H * G + j] += d * p[H * G + H + j] for i in range(G): g[j * G + i] += d * p[H * G + H + j] * x[i] return loss_sum / len(idx), [v / len(idx) for v in g] P0, BATCH = init_weights(SEED + 1), list(range(30)) root, NODES, auto = by_graph(P0, X_train, y_train, BATCH) loss_value, manual = by_hand(P0, X_train, y_train, BATCH) print(f"parameters {len(P0)}; for a 30-example batch the graph has {NODES} nodes, " f"{NODES / 30:.1f} per example") print(f"loss graph {root.v:.10f} by hand {loss_value:.10f}") print(f"largest absolute gradient difference {max(abs(u - v) for u, v in zip(auto, manual)):.2e}" f" (largest gradient {max(abs(v) for v in auto):.4f})") print(f"hand-written route {len(inspect.getsource(by_hand).splitlines())} lines " f"and belongs to this network only; machinery {LINES} lines and knows nothing of the network") ``` ``` parameters 81; for a 30-example batch the graph has 6622 nodes, 220.7 per example loss graph 0.7366651135 by hand 0.7366651135 largest absolute gradient difference 2.78e-17 (largest gradient 0.3097) hand-written route 18 lines and belongs to this network only; machinery 32 lines and knows nothing of the network ``` This is the lesson's first number. The gradient of eighty-one parameters was computed by two routes and the largest absolute difference came out as **2.78e-17**: a difference produced by the order of floating-point addition. Since the largest gradient is 0.3097, the two routes give the same number **to fifteen digits**; equality is checked not with `==` but with the ratio of the difference to the largest gradient. The second number says the opposite. The hand-written route took eighteen lines, and these lines belong **to this network only**; if a layer is added they are rewritten. The thirty-two-line machinery knows nothing of the network's structure; whoever writes the forward pass, the same `backward` call gives the gradient. Its cost is in the third number: for a batch of thirty examples the graph built **6,622 nodes**, 220.7 per example. The hand-written route holds no nodes; it accumulates the intermediate values in place. #### The Course's Three Numbers ```python # Training runs with the HAND-WRITTEN backpropagation; the graph was built for verification. The two # runs differ ONLY IN THE INITIAL VALUES: the example order is the same in both. STEP, EPOCHS, BATCH_SIZE, ORDER_SEED = 0.30, 40, 30, SEED + 700 def train_model(seed): p = init_weights(seed) for t in range(EPOCHS): indices = mix(range(len(X_train)), ORDER_SEED + t) for start in range(0, len(indices), BATCH_SIZE): _, g = by_hand(p, X_train, y_train, indices[start:start + BATCH_SIZE]) p = [a - STEP * b for a, b in zip(p, g)] return p def accuracy(p, X, y): n = 0 for k in range(len(X)): x = X[k] o = p[-1] + sum(p[H * G + H + j] * max(0.0, p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G))) for j in range(H)) n += (o >= 0.0) == (y[k] == 1) return n / len(X) S = [accuracy(train_model(SEED + t), X_test, y_test) for t in range(1, 11)] print(f"network {G}-{H}-1, {len(P0)} parameters, {EPOCHS} epochs, batch size {BATCH_SIZE}, " f"learning rate {STEP}, {EPOCHS * (len(X_train) // BATCH_SIZE)} updates") print(f" baseline {BASELINE:.4f}") print(f" network's number {S[0]:.4f} (difference {S[0] - BASELINE:+.4f})") print(f" second run {S[1]:.4f} (difference {S[1] - BASELINE:+.4f})") print(f"range over ten initializations {min(S):.4f} - {max(S):.4f}, width " f"{max(S) - min(S):.4f} ({round((max(S) - min(S)) * len(X_test))} subscribers)") TREE, LINEAR = 198, 202 # the Supervised Learning course's numbers, on the same 252-row test set low, high = round(min(S) * len(X_test)), round(max(S) * len(X_test)) print(f"comparison lines (with the 756-row training set): decision tree {TREE / len(X_test):.4f}, " f"linear model {LINEAR / len(X_test):.4f}; the network's range {low}-{high} subscribers" f" -> {sum(low <= v <= high for v in (TREE, LINEAR))}/2 inside") print(f"had the same training run through the graph, " f"{EPOCHS * (len(X_train) // BATCH_SIZE) * NODES:,} nodes would have been built") ``` ``` network 8-8-1, 81 parameters, 40 epochs, batch size 30, learning rate 0.3, 400 updates baseline 0.7579 network's number 0.8095 (difference +0.0516) second run 0.7937 (difference +0.0357) range over ten initializations 0.7857 - 0.8095, width 0.0238 (6 subscribers) comparison lines (with the 756-row training set): decision tree 0.7857, linear model 0.8016; the network's range 198-204 subscribers -> 2/2 inside had the same training run through the graph, 2,648,800 nodes would have been built ``` Three numbers side by side: the baseline 0.7579, the network's number 0.8095, the second run 0.7937. The difference between them is 0.0158, that is, four subscribers; this difference between the two runs is close to a third of the gain over the baseline (+0.0516). Over ten initializations the range spreads **between 0.7857 and 0.8095**, six subscribers wide. With **nothing changing** in capacity, data, learning rate or number of epochs, the network's number can land anywhere in this range. The Supervised Learning course's decision tree got 198 and its linear model 202 subscribers right on the test set; the network's range over ten runs is 198–204 and **both are inside**. The comparison is not exact, since those numbers came from the 756-row training set; but the direction is clear: saying that the network passed the two older families with the number it produced in a single run is reading run-to-run variability as a finding. This is why the course's rule exists — **a result that does not state that the same capacity gave two different numbers in two runs counts as unmeasured.** #### Summary - The computation graph is a data structure: each node holds its value, its parents and its local derivative; the backward pass walks the graph in reverse and the chain rule is written only once. - The thirty-two-line machinery with four operations gives the gradient of eighty-one parameters with a difference of **2.78e-17** from the eighteen-line hand-written backpropagation; the largest gradient is 0.3097. - Thirty-two lines do not know the structure of the network, eighteen lines belong to this network only; the cost is 6,622 nodes for a batch of thirty examples, 220.7 per example. - The course's three numbers: baseline 0.7579, the network's number 0.8095, second run 0.7937; the difference between them is 0.0158, close to a third of the gain over the baseline. - The range of ten initializations is 0.7857–0.8095, and the numbers of the Supervised Learning course's decision tree (0.7857) and linear model (0.8016) fall within this range; an advantage read from a single run cannot be separated from run-to-run variability. #### Next Step The gradient was computed correctly, but which examples were fed to the network in which order passed in a single line: the training set was shuffled every epoch and split into batches of thirty. That line is a decision. The next lesson addresses the data loading pipeline: how much the batch generation order moves the number, what data augmentation adds, and in which case augmentation produces leakage into the held-out set are measured. ### Data Loading Pipeline Source: https://academia.sh/en/courses/deep-learning/data-loading-pipeline Measuring the batch generation order, data augmentation and the leakage augmentation produces separately: with the same network and the same 400 updates, four batch orders give four numbers between 0.8016 and 0.8095 on the test set, while the difference rises to 0.0159 when only the initial values change under the same order — the order decision moves half as much as run-to-run variability. The difference augmentation adds is +0.0026, +0.0013 and −0.0079 in the mean of three runs; all three are smaller than the run range of a single order. When augmentation is done before the split, the twins of 138 rows of the test set enter the training set and the twinned-clean subset gap shifts from −0.0211 to −0.0008: a shift of +0.0203. The previous lesson computed the gradient by two routes and showed that the two gave the same number to machine precision. One line of that lesson's training loop passed without question: the training set was shuffled every epoch and split into batches of thirty. This line determines not **what** the network sees but **in which order** it sees it. The data loading pipeline consists of three decisions: which examples, in which order, how many together. The third is the batch size and is left to the next lesson. This lesson measures the first two and adds a fourth: **data augmentation**. The measure is two numbers: the difference augmentation adds, and the inflation in the case where augmentation **produces leakage**. - **TP7.** The dataset, the network, the baseline and the hand-written backpropagation are inherited from the previous lesson: the fictional subscriber table, the 300-row training set, the 8-8-1 network, 81 parameters, a baseline of 0.7579 on the test set. - **TP8.** So that the comparisons can run, **the number of updates is held constant at 400**. Since the augmented set is twice as large, the number of epochs drops from forty to twenty; the network takes the same number of gradient steps in both setups. - **TP9.** Augmentation here is **adding noise to tabular data** and is a model: copy rows are produced by adding noise, measured in units of standard deviation, to the four continuous columns; the label is copied and not changed. Domain-specific augmentation procedures are left to the Natural Language Processing and Computer Vision course. - **TP10.** In the leakage setup **the test set is the same 252 original rows in both cases**; what changes is only which rows' twins enter the training set. Leakage was measured in the Data Preparation and Feature Engineering course, and its procedure is not repeated. - **TP11.** Run-to-run variability is not noise but a measured quantity: every comparison is printed with **at least two runs** and the difference it adds is read together with this range. #### The Dataset, the Network and the Baseline ```python # COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through # Unsupervised Learning courses is generated with the same seed (20260218); # the training set is reduced from 756 to 300 rows. The network and the hand-written # backpropagation are inherited from the previous lesson: 8-8-1, 81 parameters, 400 updates, step 0.30. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "mean"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def mix(items, seed): r, s = rng(seed), list(items) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue subscriber_id = 10001 + i r, v = rng(SEED + 7000 + subscriber_id), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean": round(mean, 2), "suspicious": int(z > 1.35)} for q in REGION[1:]: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) K = mix(DATA, SEED + 90000) TRAIN, TEST = K[:300], K[1008:] def table(rows, stats=None): X = [[float(s[a]) for a in FIELDS] for s in rows] if stats is None: # scaling is learned from the training set only stats = [] for j in range(4, 8): m = sum(v[j] for v in X) / len(X) stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X)))) for v in X: for j, m, sd in stats: v[j] = (v[j] - m) / sd return X, [s["suspicious"] for s in rows], stats X_train, y_train, STATS = table(TRAIN) X_test, y_test, _ = table(TEST, STATS) BASELINE = sum(v == 0 for v in y_test) / len(y_test) G, H = len(FIELDS), 8 def init_weights(seed): # gain-scaled initialization r = rng(seed) return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(H * G)] + [0.0] * H \ + [(r() * 2 - 1) * math.sqrt(2.0 / H) for _ in range(H)] + [0.0] def gradient(p, X, y, idx): # the hand-written backpropagation of the previous lesson g = [0.0] * len(p) for k in idx: x, hh = X[k], [0.0] * H for j in range(H): s = p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G)) hh[j] = s if s > 0 else 0.0 s = p[-1] + sum(p[H * G + H + j] * hh[j] for j in range(H)) d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k] g[-1] += d for j in range(H): g[H * G + H + j] += d * hh[j] if hh[j] > 0: g[H * G + j] += d * p[H * G + H + j] for i in range(G): g[j * G + i] += d * p[H * G + H + j] * x[i] return [v / len(idx) for v in g] STEP, BATCH_SIZE = 0.30, 30 def train_model(X, y, seed, order, updates=400): p, t = init_weights(seed), 0 while t * (len(X) // BATCH_SIZE) < updates: perm = order(t, len(X)) for b in range(0, len(perm) - BATCH_SIZE + 1, BATCH_SIZE): p = [a - STEP * v for a, v in zip(p, gradient(p, X, y, perm[b:b + BATCH_SIZE]))] t += 1 return p def accuracy(p, X, y): n = 0 for k in range(len(X)): x = X[k] o = p[-1] + sum(p[H * G + H + j] * max(0.0, p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G))) for j in range(H)) n += (o >= 0.0) == (y[k] == 1) return n / len(X) print(f"fictional: train {len(X_train)} rows (reduced from 756), test {len(X_test)}, " f"features {G}, parameters {len(init_weights(SEED))}") print(f"baseline (predicting the most frequent class, on the test set) {BASELINE:.4f}") ``` ``` fictional: train 300 rows (reduced from 756), test 252, features 8, parameters 81 baseline (predicting the most frequent class, on the test set) 0.7579 ``` #### Batch Generation Order There are four order decisions and all four are defensible. **Unordered**: rows are given in the order of the file, the same batches are built every epoch. **Once**: the set is shuffled once at the start and that order is kept for forty epochs. **Every epoch**: the set is shuffled again every epoch and the batches are built from different rows every epoch; this last one is tried with two separate seeds. ```python # The batch generation ORDER is a decision: same data, same network, same 400 updates. # Four orders, each with two initializations. ORDERS = {"unordered": lambda t, n: list(range(n)), "once": lambda t, n: mix(range(n), SEED + 700), "every epoch A": lambda t, n: mix(range(n), SEED + 700 + t), "every epoch B": lambda t, n: mix(range(n), SEED + 800 + t)} S = {} print(f"{'batch order':<13} {'run 1':>8} {'run 2':>8} {'diff':>8}") for name, o in ORDERS.items(): S[name] = [accuracy(train_model(X_train, y_train, SEED + k, o), X_test, y_test) for k in (1, 2)] print(f"{name:<13} {S[name][0]:>8.4f} {S[name][1]:>8.4f} {S[name][1] - S[name][0]:>+8.4f}") first = [v[0] for v in S.values()] widest = max(abs(v[1] - v[0]) for v in S.values()) print(f"moved by the order (same initialization, four orders) {max(first) - min(first):.4f} " f"({round((max(first) - min(first)) * len(X_test))} subscribers)") print(f"moved by the initialization (same order, two runs) at most {widest:.4f} " f"({round(widest * len(X_test))} subscribers)") ORDER = ORDERS["every epoch A"] ``` ``` batch order run 1 run 2 diff unordered 0.8056 0.7897 -0.0159 once 0.8016 0.8016 +0.0000 every epoch A 0.8095 0.7937 -0.0159 every epoch B 0.8016 0.8056 +0.0040 moved by the order (same initialization, four orders) 0.0079 (2 subscribers) moved by the initialization (same order, two runs) at most 0.0159 (4 subscribers) ``` Four orders, the same network, the same 400 updates: test numbers of 0.8056, 0.8016, 0.8095 and 0.8016. What the order moves is **0.0079**, two subscribers. This is not a finding but it is a decision: whichever order is chosen, its justification must be written, because it moves the result. Read through the second column, the same table falls into scale. When the order is held constant and only the initial values change, the difference rises to **0.0159**, four subscribers — twice what the order moves. In the "unordered" and "every epoch A" rows the second run falls four subscribers below the first, while in the "once" row the two runs give exactly the same number. Two runs coming out equal in the same row does not show that the order has no effect; it shows that **two runs are not all runs**. #### Data Augmentation ```python # Data augmentation: COPY rows with noise added to the four continuous columns; the noise # scale is in units of standard deviation. The NUMBER OF UPDATES is HELD CONSTANT at 400 — # since the augmented set is twice as large, the number of epochs is halved. def augment(X, y, seed, noise): r, X_aug, y_aug = rng(seed), list(X), list(y) for k in range(len(X)): v = list(X[k]) for j in range(4, 8): v[j] += (r() + r() + r() - 1.5) * 2 * noise X_aug.append(v) y_aug.append(y[k]) return X_aug, y_aug MODELS, MEANS = {}, {} print(f"{'augmentation':<13} {'rows':>6} {'epochs':>6} {'run1':>7} {'run2':>7} " f"{'run3':>7} {'mean':>7}") for noise in (0.0, 0.05, 0.15, 0.40): X_aug, y_aug = (X_train, y_train) if noise == 0.0 else augment(X_train, y_train, SEED + 11000, noise) MODELS[noise] = [train_model(X_aug, y_aug, SEED + k, ORDER) for k in (1, 2, 3)] r = [accuracy(p, X_test, y_test) for p in MODELS[noise]] MEANS[noise] = sum(r) / len(r) name = "none" if noise == 0.0 else f"noise {noise:.2f}" print(f"{name:<13} {len(X_aug):>6} {400 // (len(X_aug) // 30):>6} " + " ".join(f"{v:>7.4f}" for v in r) + f" {MEANS[noise]:>7.4f}") for noise in (0.05, 0.15, 0.40): print(f"added by augmentation (noise {noise:.2f}) {MEANS[noise] - MEANS[0.0]:+.4f}") ``` ``` augmentation rows epochs run1 run2 run3 mean none 300 40 0.8095 0.7937 0.8016 0.8016 noise 0.05 600 20 0.8016 0.8095 0.8016 0.8042 noise 0.15 600 20 0.8016 0.8095 0.7976 0.8029 noise 0.40 600 20 0.7937 0.7937 0.7937 0.7937 added by augmentation (noise 0.05) +0.0026 added by augmentation (noise 0.15) +0.0013 added by augmentation (noise 0.40) -0.0079 ``` The difference augmentation adds is read from the mean of three runs: **+0.0026** at noise 0.05, **+0.0013** at 0.15 and **−0.0079** at 0.40. The first two are positive and the third negative; but all three are smaller than the setup without augmentation's own run range (0.7937–0.8095, that is, 0.0158). The honest sentence on this table is: **augmentation does not add a measurable difference on this data.** The contribution cannot be told apart from zero, because the difference between two runs of the same setup is larger than the contribution. Something else shows in the last row. When the noise rises to 0.40, all three runs give **exactly the same number**: 0.7937. Strong noise behaves like regularization and erases the difference the initial values opened, but together with the difference it erases it also erases the gain. The number of rows the network sees doubled; since the number of epochs halved, the gradient steps it took stayed the same, and the diversity augmentation gained did not turn out larger than the signal it lost. #### Where Augmentation Leaks Augmentation is an order decision. The right order is **split, then augment**: copies are derived only from training rows. The wrong order is **augment, then split**: the noisy twin of a test row falls into the training set and the network has nearly seen that row. ```python # LEAKAGE: if augmentation is done BEFORE the split, the noisy twin of a test row enters # the training set (the Data Preparation and Feature Engineering course, data leakage). The test set is the SAME # 252 original rows in both setups; the shift is read from the subset comparison. def leaky(noise, seed): r, X_aug, y_aug = rng(seed), list(X_train), list(y_train) pool = mix(range(len(X_train) + len(X_test)), seed + 5)[:len(X_train)] for k in pool: x, e = (X_train[k], y_train[k]) if k < len(X_train) else (X_test[k - len(X_train)], y_test[k - len(X_train)]) v = list(x) for j in range(4, 8): v[j] += (r() + r() + r() - 1.5) * 2 * noise X_aug.append(v) y_aug.append(e) return X_aug, y_aug, sorted(k - len(X_train) for k in pool if k >= len(X_train)) def subset(p, idx): return accuracy(p, [X_test[k] for k in idx], [y_test[k] for k in idx]) X_leak, y_leak, TWINS = leaky(0.05, SEED + 11000) CLEAN = [k for k in range(len(X_test)) if k not in set(TWINS)] print(f"{len(TWINS)} rows of the test set have a twin in the training set, " f"{len(CLEAN)} do not") print(f"{'training':<11} {'run':>5} {'all':>7} {'twinned':>7} {'clean':>7} {'diff':>8}") K = {} for name, P in (("honest", MODELS[0.05]), ("leaky", [train_model(X_leak, y_leak, SEED + k, ORDER) for k in (1, 2, 3)])): K[name] = [] for k, p in enumerate(P, 1): a, b = subset(p, TWINS), subset(p, CLEAN) K[name].append(a - b) print(f"{name:<11} {k:>5} {accuracy(p, X_test, y_test):>7.4f} {a:>7.4f} {b:>7.4f} " f"{a - b:>+8.4f}") d, s = sum(K["honest"]) / 3, sum(K["leaky"]) / 3 print(f"mean of the twinned-clean gap: honest {d:+.4f}, leaky {s:+.4f}; " f"shift {s - d:+.4f}") ``` ``` 138 rows of the test set have a twin in the training set, 114 do not training run all twinned clean diff honest 1 0.8016 0.7899 0.8158 -0.0259 honest 2 0.8095 0.8043 0.8158 -0.0114 honest 3 0.8016 0.7899 0.8158 -0.0259 leaky 1 0.8095 0.8116 0.8070 +0.0046 leaky 2 0.8095 0.8043 0.8158 -0.0114 leaky 3 0.8095 0.8116 0.8070 +0.0046 mean of the twinned-clean gap: honest -0.0211, leaky -0.0008; shift +0.0203 ``` The overall numbers cannot be read on their own: the three runs of leaky training sit at 0.8095, the three runs of honest training between 0.8016 and 0.8095. The difference stays inside the run range and **at this scale leakage is not visible from the overall number**. In the Data Preparation and Feature Engineering course a leakage of the same kind had inflated the held-out set number by 0.0476. The subset comparison cancels run-to-run variability, because the two subsets are read **from the same run**. The difference between the 138 rows whose twin is in the training set and the 114 rows with no twin is, in honest training, **−0.0211** on average: the twinned group is slightly harder to begin with. In leaky training the same difference rises to **−0.0008**. The shift is **+0.0203**, that is, five subscribers; and this shift is the trace left by the leakage that does not show in the overall number. The presence of leakage is read not from the overall number but from **who inflated**. #### Summary - When the batch generation order is built with four separate decisions, the test number varies between 0.8016 and 0.8095; what the order moves is 0.0079, two subscribers. - When only the initial values change under the same order, the difference rises to 0.0159, four subscribers: run-to-run variability moves twice as much as the order decision. - The difference augmentation adds is +0.0026, +0.0013 and −0.0079 in the mean of three runs; all three are smaller than the setup without augmentation's run range, that is, the contribution cannot be told apart from zero. - At noise 0.40 all three runs give 0.7937: strong augmentation erases both the initial difference and the gain. - When augmentation is done before the split, the twins of 138 rows of the test set enter the training set; the overall number does not show the inflation but the twinned-clean gap shifts from −0.0211 to −0.0008, a shift of +0.0203. #### Next Step The batch size was held at thirty in this lesson, and the number of updates at four hundred. Both are decisions and both are tied to a resource: how many examples' forward and backward passes can stay in memory **at the same time** in one update. The next lesson addresses hardware usage: the relation between the batch size and the number of stored values in memory is computed, and that the batch size needs to change together with the learning rate is shown with numbers. ### Hardware Usage Source: https://academia.sh/en/courses/deep-learning/hardware-usage Computing the relation between the batch size and the number of stored values in memory, and measuring that the batch size must change together with the learning rate: a one-layer eight-unit network holds 1,182 values at batch size 30 and 0.8629 of them are activations; in a four-layer sixty-four-unit network the share drops to 0.0198 at batch size 1 and rises to 0.8583 of 185,242 values at batch size 300. In a budget of one hundred thousand values the largest batch size is 2,936, 676 and 139 respectively. If the learning rate is left at 0.30 while the number of epochs is constant, the spread of four batch sizes becomes 0.0476 (twelve subscribers) and the lowest, 0.7619, stays only 0.0040 above the baseline; when the step is scaled with the batch size, the spread falls to 0.0159. The previous lesson held the batch size at thirty. This number is tied to a resource: how many examples' forward and backward passes can stay in memory **at the same time** in one update. An accelerator is a class of hardware, and the subject of this lesson is not the properties of any product but **the computation that is the same on every accelerator**: as the batch size grows, which quantity grows by how much. The second question stands in the shadow of the first. The batch size looks like a memory decision, but it also changes the size of the step gradient descent takes: the gradient of a batch is the **mean** of that batch, and as the batch grows the mean calms down. The lesson's measure is two numbers: how much the batch size grows the **number of stored values**, and how much the test number drops when the batch size is changed alone. - **TP12.** The dataset, the network, the baseline and the backpropagation are inherited from Lesson 01; the number of epochs is **held constant at 40** in this lesson and the number of updates changes accordingly as the batch size changes. - **TP13.** The measure is **the number of stored values**, not bytes. How many bytes a value takes is a **type decision** and was measured in the memory lesson of the Data Analysis with Python course: there the same column took 173,837, 19,251 and 3,628 bytes by representation, and when the width was reduced from 8 bytes to 1 byte the memory shrank 31.4 times while the total shifted by 6.9 m³. **That computation is not repeated**; it is taken as input. - **TP14.** The three stacks counted are: the **parameters**, the **gradients of the parameters** and the **activations** of every example in the batch together with their gradients. Activations are stored for the backward pass; if they were not stored, the forward pass would have to run again. - **TP15.** The optimizer's own state is counted separately: gradient descent with momentum asks for one more value per parameter. This lesson runs with plain descent and the state is zero. - **TP16.** The memory budget is a **fictional** number (100,000 values) and is not the memory of any product; its purpose is to show the computation from the budget to the batch size. - **TP17.** The scaled step rule is **directly proportional to the batch size**: the step of 0.30 for a batch of thirty becomes 3.00 for a batch of three hundred. This is not a rule but a **candidate**, and it is measured. #### The Dataset, the Network and the Baseline ```python # COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through # Unsupervised Learning courses is generated with the same seed (20260218); # the training set is reduced from 756 to 300 rows. The network and the backpropagation # are inherited from Lesson 01; the number of epochs is 40 and constant. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "mean"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def mix(items, seed): r, s = rng(seed), list(items) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue subscriber_id = 10001 + i r, v = rng(SEED + 7000 + subscriber_id), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean": round(mean, 2), "suspicious": int(z > 1.35)} for q in REGION[1:]: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) K = mix(DATA, SEED + 90000) TRAIN, TEST = K[:300], K[1008:] def table(rows, stats=None): X = [[float(s[a]) for a in FIELDS] for s in rows] if stats is None: # scaling is learned from the training set only stats = [] for j in range(4, 8): m = sum(v[j] for v in X) / len(X) stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X)))) for v in X: for j, m, sd in stats: v[j] = (v[j] - m) / sd return X, [s["suspicious"] for s in rows], stats X_train, y_train, STATS = table(TRAIN) X_test, y_test, _ = table(TEST, STATS) BASELINE = sum(v == 0 for v in y_test) / len(y_test) G, H = len(FIELDS), 8 def init_weights(seed, h=H): # gain-scaled initialization r = rng(seed) return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(h * G)] + [0.0] * h \ + [(r() * 2 - 1) * math.sqrt(2.0 / h) for _ in range(h)] + [0.0] def gradient(p, X, y, idx, h=H): # the hand-written backpropagation of Lesson 01 g = [0.0] * len(p) for k in idx: x, hh = X[k], [0.0] * h for j in range(h): s = p[h * G + j] + sum(p[j * G + i] * x[i] for i in range(G)) hh[j] = s if s > 0 else 0.0 s = p[-1] + sum(p[h * G + h + j] * hh[j] for j in range(h)) d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k] g[-1] += d for j in range(h): g[h * G + h + j] += d * hh[j] if hh[j] > 0: g[h * G + j] += d * p[h * G + h + j] for i in range(G): g[j * G + i] += d * p[h * G + h + j] * x[i] return [v / len(idx) for v in g] def train_model(X, y, seed, batch=30, step=0.30, epochs=40, h=H): p = init_weights(seed, h) for t in range(epochs): perm = mix(range(len(X)), SEED + 700 + t) for b in range(0, len(perm) - batch + 1, batch): p = [a - step * v for a, v in zip(p, gradient(p, X, y, perm[b:b + batch], h))] return p def accuracy(p, X, y, h=H): n = 0 for k in range(len(X)): x = X[k] o = p[-1] + sum(p[h * G + h + j] * max(0.0, p[h * G + j] + sum(p[j * G + i] * x[i] for i in range(G))) for j in range(h)) n += (o >= 0.0) == (y[k] == 1) return n / len(X) print(f"fictional: train {len(X_train)} rows (reduced from 756), test {len(X_test)}, " f"features {G}, hidden units {H}, parameters {len(init_weights(SEED))}") print(f"baseline (predicting the most frequent class, on the test set) {BASELINE:.4f}") ``` ``` fictional: train 300 rows (reduced from 756), test 252, features 8, hidden units 8, parameters 81 baseline (predicting the most frequent class, on the test set) 0.7579 ``` #### How Many Values Are Held in One Update The number of parameters is independent of the batch size; the number of activations is **directly proportional** to the batch size. The ratio of the two quantities depends on the shape of the network and the table reads it on three networks. ```python # The NUMBER OF STORED VALUES. Three stacks are held for an update: the parameters, their # gradients, and the activations of every example in the batch (the intermediate values of # the forward pass are stored for the backward pass) together with the gradients of those activations. def count_values(g, h, layers, batch, state=0): params = g * h + h + (layers - 1) * (h * h + h) + h + 1 return params, params * (1 + state), 2 * batch * (g + layers * h + 1) print(f"{'layers':>7} {'hidden':>6} {'batch':>6} {'parameters':>10} {'gradient':>8} " f"{'activations':>14} {'total':>9} {'activation share':>16}") for layers, h in ((1, 8), (1, 64), (4, 64)): for batch in (1, 30, 300): p, e, a = count_values(G, h, layers, batch) print(f"{layers:>7} {h:>6} {batch:>6} {p:>10} {e:>8} {a:>14} {p + e + a:>9} " f"{a / (p + e + a):>16.4f}") p, e, a = count_values(G, H, 1, 30) print(f"\nthis lesson's network (1 layer, {H} units, batch size 30): {p + e + a} values; " f"at batch size 300 {sum(count_values(G, H, 1, 300)):,}") print(f"descent with momentum asks for one more value per parameter for the optimizer " f"state: {sum(count_values(G, H, 1, 30, 1))} values") BUDGET = 100000 print(f"\nif the budget were {BUDGET:,} values, the largest batch size:") for layers, h in ((1, 8), (1, 64), (4, 64)): params = count_values(G, h, layers, 1)[0] print(f" layers {layers}, units {h:>2}: {(BUDGET - 2 * params) // (2 * (G + layers * h + 1)):>6,}") ``` ``` layers hidden batch parameters gradient activations total activation share 1 8 1 81 81 34 196 0.1735 1 8 30 81 81 1020 1182 0.8629 1 8 300 81 81 10200 10362 0.9844 1 64 1 641 641 146 1428 0.1022 1 64 30 641 641 4380 5662 0.7736 1 64 300 641 641 43800 45082 0.9716 4 64 1 13121 13121 530 26772 0.0198 4 64 30 13121 13121 15900 42142 0.3773 4 64 300 13121 13121 159000 185242 0.8583 this lesson's network (1 layer, 8 units, batch size 30): 1182 values; at batch size 300 10,362 descent with momentum asks for one more value per parameter for the optimizer state: 1263 values if the budget were 100,000 values, the largest batch size: layers 1, units 8: 2,936 layers 1, units 64: 676 layers 4, units 64: 139 ``` Three groups of rows show three different arrangements. In the one-layer eight-unit network the activation share is 0.1735 at batch size 1; it rises to 0.8629 at batch size 30 and 0.9844 at batch size 300. This network is so small that **data** fills almost all of the memory. In the four-layer sixty-four-unit network the table turns around: at batch size 1 the share is 0.0198, that is, the memory belongs to the parameters and gradients; but when the batch size goes up to 300 the share becomes 0.8583 in the same network and the total grows from 26,772 to 185,242, more than sixfold. The number of parameters **did not change at all** between these two rows. The useful side of the computation appears when it is read in reverse. When a memory budget is given, the largest batch size can be solved: in a fictional budget of a hundred thousand values the small network can take 2,936 examples, the wide one-layer network 676 and the four-layer network 139. The same budget, the same hardware, **a twentyfold difference**. The batch size must be chosen not as a habit but as the result of this split; and the chosen number does not come alone, as the next section shows. #### Batch Size and Learning Rate Change Together ```python # The batch size and the learning rate change TOGETHER. The number of epochs is constant at 40; # as the batch size grows, the number of updates falls. The scaled step is directly # proportional to the batch size. print(f"{'batch':>6} {'updates':>11} {'fixed step 0.30':>25} {'scaled step':>25}") FIXED, SCALED = [], [] for batch in (10, 30, 100, 300): a = [accuracy(train_model(X_train, y_train, SEED + k, batch, 0.30), X_test, y_test) for k in (1, 2)] o = 0.30 * batch / 30 b = [accuracy(train_model(X_train, y_train, SEED + k, batch, o), X_test, y_test) for k in (1, 2)] FIXED += a SCALED += b print(f"{batch:>6} {40 * (len(X_train) // batch):>11} {a[0]:>13.4f} {a[1]:>6.4f}" f" {'step ' + format(o, '.2f'):>12} {b[0]:>6.4f} {b[1]:>6.4f}") print(f"spread over four batch sizes, fixed step {max(FIXED) - min(FIXED):.4f} " f"({round((max(FIXED) - min(FIXED)) * len(X_test))} subscribers)") print(f"same spread, scaled step {max(SCALED) - min(SCALED):.4f} " f"({round((max(SCALED) - min(SCALED)) * len(X_test))} subscribers)") print(f"baseline {BASELINE:.4f}; lowest with fixed step {min(FIXED):.4f} " f"({min(FIXED) - BASELINE:+.4f}), lowest with scaled step {min(SCALED):.4f} " f"({min(SCALED) - BASELINE:+.4f})") ``` ``` batch updates fixed step 0.30 scaled step 10 1200 0.7619 0.7857 step 0.10 0.8056 0.7937 30 400 0.8095 0.7937 step 0.30 0.8095 0.7937 100 120 0.7897 0.8056 step 1.00 0.8016 0.8016 300 40 0.7619 0.7738 step 3.00 0.8056 0.7937 spread over four batch sizes, fixed step 0.0476 (12 subscribers) same spread, scaled step 0.0159 (4 subscribers) baseline 0.7579; lowest with fixed step 0.7619 (+0.0040), lowest with scaled step 0.7937 (+0.0357) ``` The left column shows the cost of changing the batch size alone. With the step left at 0.30, the spread of the four batch sizes is **0.0476**, twelve subscribers: three times run-to-run variability (four subscribers). There are two separate reasons at the two ends. At batch size 300 only forty updates are made and the network stops before training is finished; the result is 0.7619, **only 0.0040 above the baseline**, that is, one subscriber. At batch size 10 the number of updates rises to 1,200 but since each gradient is the mean of only ten examples it is noisy; a step of 0.30 stays too large together with that noise and the result again falls to 0.7619. The right column scales the step with the batch size: 0.10, 0.30, 1.00 and 3.00. The spread of the four batch sizes falls to **0.0159**, four subscribers — that is, the same size as the difference between two runs of the same setup. The lowest number rises from 0.7619 to 0.7937 and the difference above the baseline from 0.0040 to 0.0357. The batch size alone is not a hyperparameter: when it is changed the learning rate must change too, otherwise what is measured is not the memory decision itself but the step forgotten beside that decision. #### Summary - The values held in one update are three stacks: the parameters, their gradients, and the activations of the examples in the batch with their gradients; the first two are independent of the batch size, the third is directly proportional. - A one-layer eight-unit network holds 1,182 values at batch size 30 and 0.8629 of them are activations; in the four-layer sixty-four-unit network the share is 0.0198 at batch size 1 and 0.8583 at batch size 300. - Going from batch size 1 to 300 in the same network carries the total from 26,772 to 185,242; the number of parameters does not change at all. - In a fictional budget of a hundred thousand values the largest batch size is 2,936, 676 and 139 for the three networks; the same budget produces a twentyfold difference. - If the step is left at 0.30 while the number of epochs is constant, the spread of the four batch sizes is 0.0476 (twelve subscribers) and the lowest, 0.7619, stays one subscriber above the baseline; when the step is scaled with the batch size the spread comes to 0.0159 and the lowest number to 0.7937. #### Next Step The number of items pinned over three lessons grew quickly: seed, example order, batch size, learning rate, number of epochs, augmentation scale, from which set the scaling statistics are read. None of them is the model itself and all of them move the result; in this lesson, leaving a single item — the step — in place lowered the number by twelve subscribers. How many items need to be pinned for a training to give the same number a second time was never counted. The next lesson addresses experiment tracking: the items to pin are counted on top of the lists of the earlier courses, and how much the result moves when one of them is released is printed. ### Experiment Tracking Source: https://academia.sh/en/courses/deep-learning/experiment-tracking Counting the items that must be pinned for the same training to give the same number a second time: on top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course and the 11 of the Visualization, Reporting and Analysis at Scale course, 8 training-specific items are added, making 67. Six of the eight items move the result when released alone, and the largest shift is 0.0159, four subscribers — that item is not a setting of the model but its starting values. The summation order moves the parameters by 6.66e-16 and changes no decision, while the number of epochs moves the parameters by 0.2098 and again changes no decision; the number of epochs, which does not move the result alone, shifts it by two subscribers when released together with the batch size. In the previous three lessons the setup of the training was opened up piece by piece, and every piece produced a decision: by which route the gradient is computed, in which order the examples are given, how many go together, what the step is. Every decision moved the result and none of them was the model itself. How many of these decisions have to be written down for a training to give the **same** number a second time was not asked. The question is not new and part of it is closed. The Data Collection and Preparation course wrote the preparation notebook as thirty-nine decisions; the Data Analysis with Python course counted the working environment as nine items; the Visualization, Reporting and Analysis at Scale course counted the report itself as eleven items and found the total to be fifty-nine. In the pipeline lesson of the Data Preparation and Feature Engineering course one more number was placed next to this list: the preparation chain learned **131 numbers** from the training set, and all of them had to be read from the right set. **These computations are not repeated**; they are taken as input. What this lesson adds is **the training itself**. - **TP18.** The dataset, the network, the baseline and the backpropagation are inherited from Lesson 01. The items of the training loop are gathered in a single dictionary and the run is read from this dictionary; only the scaling item is applied outside the loop, while the data is being prepared. - **TP19.** To **release** an item does not mean to delete it but to move it to another value that is **equally defensible**. A batch of twenty-five instead of thirty, a step of 0.25 instead of 0.30, fifty epochs instead of forty: none of them is wrong. - **TP20.** Items are released **one by one** while the others stay pinned. The criterion is the accuracy on the test set, and the test set does not change in any setup. - **TP21.** The "summation order" item is **a trace of concurrency**: when the gradients of a batch are summed in parallel, which term is added first can change from run to run. Here this is modeled by summing the examples of the batch in reverse order; since floating-point addition is not associative, the result separates in the last digits. - **TP22.** Two quantities are counted separately: the shift of the **parameters** and the shift of the **result**. While the first is not zero the second can be zero; the threshold is how far from the decision boundary one stands. - **TP23.** No experiment tracking tool, notebook application or repository name is mentioned. Tracking here is **the item list itself**: if the list can be carried, the run can be carried. #### The Dataset, the Network and the Item Dictionary ```python # COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through # Unsupervised Learning courses is generated with the same seed (20260218); # the training set is reduced from 756 to 300 rows. The items of the training loop are # gathered in a single dictionary and the run is read from this dictionary. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "mean"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def mix(items, seed): r, s = rng(seed), list(items) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue subscriber_id = 10001 + i r, v = rng(SEED + 7000 + subscriber_id), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean": round(mean, 2), "suspicious": int(z > 1.35)} for q in REGION[1:]: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) K = mix(DATA, SEED + 90000) TRAIN, TEST = K[:300], K[1008:] def table(rows, stats=None): X = [[float(s[a]) for a in FIELDS] for s in rows] if stats is None: # scaling is learned from the training set only stats = [] for j in range(4, 8): m = sum(v[j] for v in X) / len(X) stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X)))) for v in X: for j, m, sd in stats: v[j] = (v[j] - m) / sd return X, [s["suspicious"] for s in rows], stats X_train, y_train, STATS = table(TRAIN) X_test, y_test, _ = table(TEST, STATS) BASELINE = sum(v == 0 for v in y_test) / len(y_test) G, H = len(FIELDS), 8 def init_weights(seed): # gain-scaled initialization r = rng(seed) return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(H * G)] + [0.0] * H \ + [(r() * 2 - 1) * math.sqrt(2.0 / H) for _ in range(H)] + [0.0] def gradient(p, X, y, idx): # the hand-written backpropagation of Lesson 01 g = [0.0] * len(p) for k in idx: x, hh = X[k], [0.0] * H for j in range(H): s = p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G)) hh[j] = s if s > 0 else 0.0 s = p[-1] + sum(p[H * G + H + j] * hh[j] for j in range(H)) d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k] g[-1] += d for j in range(H): g[H * G + H + j] += d * hh[j] if hh[j] > 0: g[H * G + j] += d * p[H * G + H + j] for i in range(G): g[j * G + i] += d * p[H * G + H + j] * x[i] return [v / len(idx) for v in g] DEFAULTS = {"seed": SEED + 1, "order": SEED + 700, "batch": 30, "step": 0.30, "epochs": 40, "split": "slice", "summing": "forward", "scale": "train"} def train_model(X, y, **item): a = dict(DEFAULTS, **item) p = init_weights(a["seed"]) for t in range(a["epochs"]): perm = mix(range(len(X)), a["order"] + t) n = len(perm) // a["batch"] for b in range(n): idx = perm[b * a["batch"]:(b + 1) * a["batch"]] if a["split"] == "slice" \ else perm[b::n] if a["summing"] == "backward": idx = idx[::-1] p = [u - a["step"] * v for u, v in zip(p, gradient(p, X, y, idx))] return p def accuracy(p, X, y): n = 0 for k in range(len(X)): x = X[k] o = p[-1] + sum(p[H * G + H + j] * max(0.0, p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G))) for j in range(H)) n += (o >= 0.0) == (y[k] == 1) return n / len(X) print(f"fictional: train {len(X_train)} rows (reduced from 756), test {len(X_test)}, " f"parameters {len(init_weights(SEED))}; baseline {BASELINE:.4f}") ``` ``` fictional: train 300 rows (reduced from 756), test 252, parameters 81; baseline 0.7579 ``` #### How Many Items ```python # The items to pin. The first three sources are INHERITED FROM EARLIER COURSES and are not # repeated here; this lesson adds only those SPECIFIC TO TRAINING. INHERITED = [("preparation notebook (Data Collection and Preparation)", 39), ("working environment (Data Analysis with Python)", 9), ("report (Visualization, Reporting and Analysis at Scale)", 11)] TRAINING = ["initialization seed", "example order seed", "batch size", "batch split style", "learning rate", "number of epochs", "summation order (concurrency)", "set the scale is read from"] for name, n in INHERITED: print(f" {name:<55} {n:>3} items") print(f" {'training itself (this lesson)':<55} {len(TRAINING):>3} items") print(f" {'total':<55} {sum(n for _, n in INHERITED) + len(TRAINING):>3} items") print("\ntraining-specific items and their default values:") print(" " + ", ".join(f"{a}={DEFAULTS[k]}" for a, k in zip(TRAINING, ("seed", "order", "batch", "split", "step", "epochs", "summing", "scale")))) ``` ``` preparation notebook (Data Collection and Preparation) 39 items working environment (Data Analysis with Python) 9 items report (Visualization, Reporting and Analysis at Scale) 11 items training itself (this lesson) 8 items total 67 items training-specific items and their default values: initialization seed=20260219, example order seed=20260918, batch size=30, batch split style=slice, learning rate=0.3, number of epochs=40, summation order (concurrency)=forward, set the scale is read from=train ``` The total is **sixty-seven**. Fifty-nine were counted before this lesson and are not repeated here; eight belong to the training itself. None of the eight is the architecture of the model — the number of layers, the number of units and the activation do not enter this list, because they are **the definition of the reported model**. What is on the list are decisions about **how** the reported number was produced, and most of them fit in a single line. #### Each Item One by One ```python # Each item is released ALONE: it is moved to another equally defensible value and the # others stay pinned. When the scale item is released, the scale statistics are read # from the training+test set instead of the training set. X_train2, y_train2, _ = table(TRAIN, table(TRAIN + TEST)[2]) REFERENCE, P0 = accuracy(train_model(X_train, y_train), X_test, y_test), train_model(X_train, y_train) ALTERNATIVES = [({"seed": SEED + 2}, "20260220"), ({"order": SEED + 800}, "20261018"), ({"batch": 25}, "25"), ({"split": "strided"}, "strided"), ({"step": 0.25}, "0.25"), ({"epochs": 50}, "50"), ({"summing": "backward"}, "backward"), (None, "train+test")] print(f"baseline {BASELINE:.4f}, with all items pinned {REFERENCE:.4f}\n") print(f"{'released item':<30} {'new value':<14} {'test':>7} {'shift':>8} " f"{'subscribers':>11} {'parameters':>10}") SHIFT = [] for name, (d, new) in zip(TRAINING, ALTERNATIVES): p = train_model(X_train, y_train, **d) if d else train_model(X_train2, y_train2) v = accuracy(p, X_test, y_test) SHIFT.append(abs(v - REFERENCE)) print(f"{name:<30} {new:<14} {v:>7.4f} {v - REFERENCE:>+8.4f} " f"{round(abs(v - REFERENCE) * len(X_test)):>11} " f"{max(abs(a - b) for a, b in zip(p, P0)):>10.2e}") print(f"\n{sum(1 for u in SHIFT if u > 1e-9)} of {len(SHIFT)} items move the result alone; " f"largest shift {max(SHIFT):.4f} " f"({round(max(SHIFT) * len(X_test))} subscribers)") for d, name in (({"epochs": 50, "summing": "backward"}, "epochs 50 + summing backward"), ({"epochs": 50, "batch": 25}, "epochs 50 + batch 25")): v = accuracy(train_model(X_train, y_train, **d), X_test, y_test) print(f" {name:<28} {v:.4f} shift {v - REFERENCE:+.4f} " f"({round(abs(v - REFERENCE) * len(X_test))} subscribers)") ``` ``` baseline 0.7579, with all items pinned 0.8095 released item new value test shift subscribers parameters initialization seed 20260220 0.7937 -0.0159 4 1.46e+00 example order seed 20261018 0.8016 -0.0079 2 7.45e-02 batch size 25 0.8056 -0.0040 1 2.76e-01 batch split style strided 0.8135 +0.0040 1 4.80e-01 learning rate 0.25 0.8056 -0.0040 1 1.98e-01 number of epochs 50 0.8095 +0.0000 0 2.10e-01 summation order (concurrency) backward 0.8095 +0.0000 0 6.66e-16 set the scale is read from train+test 0.8056 -0.0040 1 3.18e-01 6 of 8 items move the result alone; largest shift 0.0159 (4 subscribers) epochs 50 + summing backward 0.8095 shift +0.0000 (0 subscribers) epochs 50 + batch 25 0.8016 shift -0.0079 (2 subscribers) ``` Six of the eight items move the result alone and the largest shift is **0.0159**, four subscribers. That item is the **initialization seed**. It is not a setting of the model and not a preparation decision; it is the place where the weights start. The third number, the signature of this course, is also the largest row in the reproducibility list. The one subscriber each that the learning rate and the batch size move is a quarter of what the seed moves. The last column separates two items from each other. When the summation order is released the parameters move by **6.66e-16**, that is, by the last digit of floating-point addition; not one decision of the test set changes. When the number of epochs goes from forty to fifty the parameters move by **0.2098** — fourteen digits above the summation order — and again not one decision of the test set changes. The two zeros are not the same zero: in the first the network is the same network, in the second the network is another network and only none of the decisions has crossed the boundary. The price of this separation is in the last two rows. The number of epochs alone moves zero, the batch size alone moves one subscriber; released together they move **two subscribers**. An item that does not move the result alone is not an item that does not need to be pinned. The item for the set the scale is read from confirms this as well: when the test set is mixed into the scale statistics the number moves by only one subscriber, yet the same leakage had produced a 0.0476 inflation with the decision tree in the Data Preparation and Feature Engineering course. That a leakage looks small is not proof that it is absent. #### Summary - Reproducing a training requires pinning 67 items: on top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course and the 11 of the Visualization, Reporting and Analysis at Scale course, 8 training-specific items are added. - Six of the eight items move the result when released alone; with all items pinned the test number is 0.8095 and the baseline is 0.7579. - The largest shift is 0.0159 (four subscribers) and its source is the initialization seed — not a setting of the model, the third number of this course. - The shift of the parameters and the shift of the result are separate things: the summation order moves the parameters by 6.66e-16 and the number of epochs by 0.2098, and neither changes a single decision. - The number of epochs, which moves zero alone, moves two subscribers when released together with the batch size; and reading the scale from the test set moves one subscriber here, while in the Data Preparation and Feature Engineering course the same leakage had inflated the number by 0.0476. #### Next Step In this lesson the items were all held **pinned** and released one by one. In a real training the opposite is done: a few items are released on purpose, a candidate list is built and the best is chosen. The search methods were measured in the Supervised Learning course and will not be repeated; but in that course the number of every candidate was read from **a single run**. What was measured in this lesson makes one ask: if the difference between two runs of the same candidate is four subscribers and the difference between two candidates is one subscriber, what is the search choosing. The next lesson takes up the hyperparameter search with this question and closes the course. ### Hyperparameter Search Source: https://academia.sh/en/courses/deep-learning/tuning-variable-search Measuring how run-to-run variability misleads a search: when ten candidates are run twice with the same budget, the spread between candidates on the validation set is 0.0317 while the difference between two runs of a single candidate reaches 0.0238, and twenty-three of the forty-five candidate pairs cannot be told apart, in seven of them the order flips. The search looking from the first run chooses the sixteen-unit candidate and gets 0.8135 on the test set, while the search looking from the second run chooses the four-unit candidate and gets 0.8016; the test mean of the ten candidates is 0.8036 and the baseline is 0.7579. The course wrap-up gathers the baseline, the network's number and the second run of twenty-four lessons in a single table. The previous lesson counted the items of the training and pinned all of them; the item that made the largest shift was the starting values. In a real training some of the items are released on purpose: a candidate list is built, all of it is run, and the best is chosen from the validation set. The search methods were measured in the Evaluation and Tuning topic of the Supervised Learning course — grid, random and model-based search had given 0.7937, 0.7857 and 0.8016 on the test set in the same fifty-candidate budget — and are **not repeated here**. The only thing that changes is this: there the number of each candidate was read from **a single run**. The question of the lesson is this: with the number of candidates the same, when the difference between two runs of the same candidate is larger than the difference between candidates, what is the search choosing. - **TP24.** The dataset, the network and the baseline are inherited from Lesson 01; in addition a **validation set** is used (252 rows), and the choice is read from it, the report from the test set. - **TP25.** The number of candidates is **ten**, the budget is equal for every candidate (40 epochs) and the candidates are listed by hand; the choice of the search method is not the subject of this lesson. - **TP26.** The tuning space has three axes: hidden units (4, 8, 16), learning rate (0.15, 0.30, 0.60) and batch size (20, 30, 50). Every candidate is run **twice** and the two runs separate only in **their starting values**; the other items are pinned as in Lesson 04. - **TP27.** Two candidates are counted as indistinguishable **if their difference is smaller than the run difference of both**; the threshold comes from the measure itself. A tie goes to the first candidate on the list, and this too is a decision. #### The Dataset, the Network and the Candidate List ```python # COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through # Unsupervised Learning courses is generated with the same seed (20260218); # the training set is reduced from 756 to 300 rows. The VALIDATION set is also used in this # lesson: the choice is read from it, the report from the test set. The network comes from # Lesson 01; the number of hidden units h changes with the candidate. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "mean"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 def mix(items, seed): r, s = rng(seed), list(items) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue subscriber_id = 10001 + i r, v = rng(SEED + 7000 + subscriber_id), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "mean": round(mean, 2), "suspicious": int(z > 1.35)} for q in REGION[1:]: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) K = mix(DATA, SEED + 90000) TRAIN, VALID, TEST = K[:300], K[756:1008], K[1008:] def table(rows, stats=None): X = [[float(s[a]) for a in FIELDS] for s in rows] if stats is None: # scaling is learned from the training set only stats = [] for j in range(4, 8): m = sum(v[j] for v in X) / len(X) stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X)))) for v in X: for j, m, sd in stats: v[j] = (v[j] - m) / sd return X, [s["suspicious"] for s in rows], stats X_train, y_train, STATS = table(TRAIN) X_valid, y_valid, _ = table(VALID, STATS) X_test, y_test, _ = table(TEST, STATS) BASELINE = sum(v == 0 for v in y_test) / len(y_test) G = len(FIELDS) def init_weights(seed, h): r = rng(seed) return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(h * G)] + [0.0] * h \ + [(r() * 2 - 1) * math.sqrt(2.0 / h) for _ in range(h)] + [0.0] def gradient(p, X, y, idx, h): g = [0.0] * len(p) for k in idx: x, hh = X[k], [0.0] * h for j in range(h): s = p[h * G + j] + sum(p[j * G + i] * x[i] for i in range(G)) hh[j] = s if s > 0 else 0.0 s = p[-1] + sum(p[h * G + h + j] * hh[j] for j in range(h)) d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k] g[-1] += d for j in range(h): g[h * G + h + j] += d * hh[j] if hh[j] > 0: g[h * G + j] += d * p[h * G + h + j] for i in range(G): g[j * G + i] += d * p[h * G + h + j] * x[i] return [v / len(idx) for v in g] def train_model(seed, h, step, batch, epochs): p = init_weights(seed, h) for t in range(epochs): perm = mix(range(len(X_train)), SEED + 700 + t) for b in range(0, len(perm) - batch + 1, batch): p = [u - step * v for u, v in zip(p, gradient(p, X_train, y_train, perm[b:b + batch], h))] return p def accuracy(p, X, y, h): n = 0 for k in range(len(X)): x = X[k] o = p[-1] + sum(p[h * G + h + j] * max(0.0, p[h * G + j] + sum(p[j * G + i] * x[i] for i in range(G))) for j in range(h)) n += (o >= 0.0) == (y[k] == 1) return n / len(X) print(f"fictional: train {len(X_train)} rows (reduced from 756), validation {len(X_valid)}, " f"test {len(X_test)}; baseline {BASELINE:.4f}") ``` ``` fictional: train 300 rows (reduced from 756), validation 252, test 252; baseline 0.7579 ``` #### Ten Candidates, Two Runs ```python # TEN CANDIDATES, each with TWO RUNS. The search methods were measured in the Supervised # Learning course and are not repeated here; the only thing that changes is HOW MANY RUNS # the number of each candidate is read from. The choice is made on the validation set, # the report on the test set. CANDIDATES = [(4, 0.15, 30), (4, 0.30, 30), (8, 0.15, 30), (8, 0.30, 30), (8, 0.60, 30), (8, 0.30, 20), (8, 0.30, 50), (16, 0.15, 30), (16, 0.30, 30), (16, 0.60, 30)] VAL_SCORES, TEST_SCORES = [], [] print(f"{'cand':>4} {'hidden':>6} {'rate':>5} {'batch':>6} {'val r1':>7} {'val r2':>7} " f"{'|diff|':>7} {'test r1':>7} {'test r2':>7}") for no, (h, step, batch) in enumerate(CANDIDATES, 1): P = [train_model(SEED + k, h, step, batch, 40) for k in (1, 2)] d = [accuracy(p, X_valid, y_valid, h) for p in P] s = [accuracy(p, X_test, y_test, h) for p in P] VAL_SCORES.append(d) TEST_SCORES.append(s) print(f"{no:>4} {h:>6} {step:>5.2f} {batch:>6} {d[0]:>7.4f} {d[1]:>7.4f} " f"{abs(d[0] - d[1]):>7.4f} {s[0]:>7.4f} {s[1]:>7.4f}") ``` ``` cand hidden rate batch val r1 val r2 |diff| test r1 test r2 1 4 0.15 30 0.8135 0.8175 0.0040 0.8095 0.8056 2 4 0.30 30 0.8016 0.8254 0.0238 0.8095 0.8016 3 8 0.15 30 0.8135 0.8175 0.0040 0.8016 0.8056 4 8 0.30 30 0.8095 0.8175 0.0079 0.8095 0.7937 5 8 0.60 30 0.8016 0.8056 0.0040 0.7857 0.8056 6 8 0.30 20 0.7976 0.8095 0.0119 0.7976 0.7976 7 8 0.30 50 0.8095 0.8175 0.0079 0.8016 0.8056 8 16 0.15 30 0.8294 0.8254 0.0040 0.8135 0.8056 9 16 0.30 30 0.8294 0.8175 0.0119 0.8095 0.8016 10 16 0.60 30 0.8016 0.8016 0.0000 0.7976 0.7976 ``` The first validation column gives a familiar result: the two highest numbers are 0.8294 and both belong to the sixteen-unit candidates, so capacity seems to have paid off. The second column breaks this: candidate number two — the **smallest** network on the list — gives 0.8016 in the first run, then goes up to 0.8254 in the second and draws level with candidate number eight. #### What the Search Chooses ```python # The candidate the search chooses depends on which run it looks from. SPREAD = max(d[0] for d in VAL_SCORES) - min(d[0] for d in VAL_SCORES) RUN_GAP = [abs(d[0] - d[1]) for d in VAL_SCORES] print(f"spread between candidates (run 1, validation) {SPREAD:.4f}") print(f"difference between the two runs of the same candidate: at most {max(RUN_GAP):.4f}, " f"median {sorted(RUN_GAP)[len(RUN_GAP) // 2]:.4f}") PAIRS = [(i, j) for i in range(len(CANDIDATES)) for j in range(i + 1, len(CANDIDATES))] tied = sum(1 for i, j in PAIRS if abs(VAL_SCORES[i][0] - VAL_SCORES[j][0]) <= max(RUN_GAP[i], RUN_GAP[j])) flipped = sum(1 for i, j in PAIRS if (VAL_SCORES[i][0] - VAL_SCORES[j][0]) * (VAL_SCORES[i][1] - VAL_SCORES[j][1]) < 0) print(f"of {len(PAIRS)} candidate pairs, {tied} cannot be told apart by the run difference; " f"in {flipped} pairs the order FLIPS between the two runs") for k in (0, 1): best = max(range(len(CANDIDATES)), key=lambda i: (VAL_SCORES[i][k], -i)) print(f"chosen by run {k + 1}: candidate {best + 1} (hidden {CANDIDATES[best][0]}, " f"rate {CANDIDATES[best][1]:.2f}, batch {CANDIDATES[best][2]}): validation " f"{VAL_SCORES[best][k]:.4f}, test {TEST_SCORES[best][k]:.4f}") avg = sum(s[0] for s in TEST_SCORES) / len(TEST_SCORES) print(f"test mean of the ten candidates {avg:.4f}; baseline {BASELINE:.4f}") ``` ``` spread between candidates (run 1, validation) 0.0317 difference between the two runs of the same candidate: at most 0.0238, median 0.0079 of 45 candidate pairs, 23 cannot be told apart by the run difference; in 7 pairs the order FLIPS between the two runs chosen by run 1: candidate 8 (hidden 16, rate 0.15, batch 30): validation 0.8294, test 0.8135 chosen by run 2: candidate 2 (hidden 4, rate 0.30, batch 30): validation 0.8254, test 0.8016 test mean of the ten candidates 0.8036; baseline 0.7579 ``` The number of the lesson is in the first two lines. The spread between candidates is **0.0317**, the largest difference between two runs of the same candidate is **0.0238**; the two are of the same order of magnitude. **Twenty-three** of the forty-five candidate pairs, that is, more than half, cannot be told apart by the run difference, and in seven pairs the order **flips** between the two runs. The search that looks from the first run chooses candidate number eight: sixteen units. The search that looks from the second run chooses candidate number two: four units. The same list, the same budget and **two different answers to the question of capacity**; the test numbers of the chosen ones are 0.8135 and 0.8016. The test mean of the ten candidates is 0.8036: the candidate chosen from the first run is 0.0099 above it, the one chosen from the second run 0.0020 below. Depending on which run it looks from, the search earns two and a half subscribers or loses half a subscriber. The tuning budget produced the inflation of the number in the Supervised Learning course; here there is a problem that comes before it — **if the order of the search is built from noise, there is no gain to inflate.** The remedy is expensive: measuring every candidate with several runs and writing its number as the mean of the runs; the budget goes down from ten candidates to five. #### Summary - The spread between candidates on the validation set is 0.0317 and the largest difference between two runs of a single candidate is 0.0238; the two are of the same order of magnitude. - Twenty-three of the forty-five candidate pairs cannot be told apart by the run difference, and in seven the order flips. - The search that looks from the first run chooses the sixteen-unit candidate, the one that looks from the second run the four-unit candidate: the same list gives two different answers to the question of capacity, with test numbers of 0.8135 and 0.8016. - The test mean of the ten candidates is 0.8036 and the baseline is 0.7579; depending on which run it looks from, the search earns two and a half subscribers or loses half a subscriber. #### Course Wrap-Up The computation of twenty-four lessons is in a single table, and every number was read from its own lesson. **The rows do not carry a common unit**: the course used three separate fictional datasets and the last column writes which one it was read from. The criterion is **accuracy** in twenty-three rows and larger is better; the **Architectures 10** row is unlabeled, is read by **reconstruction error** and there smaller is better. The second run is retraining the same setup with only the starting values changed. | Lesson | Baseline | Network's number | Second run | Set | |---|---|---|---|---| | Fundamentals 01 | 0.7579 | 0.7857 | 0.7698 | subscriber | | Fundamentals 02 | 0.7579 | 0.7937 | 0.7976 | subscriber | | Fundamentals 03 | 0.7579 | 0.7540 | 0.7738 | subscriber | | Fundamentals 04 | 0.7579 | 0.7619 | 0.7897 | subscriber | | Fundamentals 05 | 0.7579 | 0.7857 | 0.7698 | subscriber | | Fundamentals 06 | 0.7579 | 0.7857 | 0.7698 | subscriber | | Fundamentals 07 | 0.7579 | 0.7817 | 0.7778 | subscriber | | Fundamentals 08 | 0.7579 | 0.7937 | 0.7738 | subscriber | | Fundamentals 09 | 0.7579 | 0.7698 | 0.7857 | subscriber | | Architectures 01 | 0.5850 | 0.8000 | 0.8150 | grid | | Architectures 02 | 0.5850 | 0.8050 | 0.8050 | grid | | Architectures 03 | 0.5850 | 0.8150 | 0.8900 | grid | | Architectures 04 | 0.7100 | 0.7350 | 0.7150 | sequence | | Architectures 05 | 0.7100 | 0.9750 | 0.9150 | sequence | | Architectures 06 | 0.6333 | 0.8889 | 0.9944 | sequence | | Architectures 07 | 0.6611 | 0.8778 | 0.8833 | sequence | | Architectures 08 | 0.6444 | 0.9722 | 0.9833 | event code | | Architectures 09 | 0.6444 | 0.8333 | 0.8500 | event code | | Architectures 10 | 2.9007 | 0.7949 | 0.7638 | sequence (error) | | Training 01 | 0.7579 | 0.8095 | 0.7937 | subscriber | | Training 02 | 0.7579 | 0.8016 | 0.8095 | subscriber | | Training 03 | 0.7579 | 0.8056 | 0.7937 | subscriber | | Training 04 | 0.7579 | 0.8095 | 0.7937 | subscriber | | Training 05 | 0.7579 | 0.8135 | 0.8016 | subscriber | **The rule of the course is this: the number of a network can come from its training and not from its capacity; a result that does not write that the same capacity gave two different numbers in two runs counts as unmeasured.** The third column comes down to a single number: of the twenty-one lessons in which the difference made by capacity or architecture was written in its own lesson, in **fourteen** the difference of the second run came out larger than that difference — in the convolutional architecture 0.0750 against the 0.0575 added by depth, in multi-head attention 0.0500 against the 0.0250 added by the number of heads, in the perceptron four subscribers against the one subscriber added by capacity. In the remaining seven the difference made by capacity was equal to the run difference or larger than it. The second claim of the course is read from the same table and is not hidden. In fourteen rows of the subscriber table the comparison lines come from the Supervised Learning course: decision tree 0.7857, linear model 0.8016. In the nine lessons of Neural Network Fundamentals the number of the network **did not exceed 0.8016 in any run**; the highest of the eighteen runs is 0.7976 and in three lessons the number of the network stopped exactly at the number of the tree. In the five lessons of Training Practice the number of the network stayed between 0.8016 and 0.8135, that is, at or above the line of the linear model; but they ran with a feature set of eight columns and the ten initializations of the first lesson spread between 0.7857–0.8095, with both lines inside this range. **The deep network could not pass the decision tree and the linear model on this fictional tabular data in a way that is separable from run-to-run variability.** The gain came not from capacity but from training practice, and in at least one more lesson a deeper network could not pass a shallower one. In twenty-four lessons the architecture was built and the second run was written next to every number. But the inputs were always **ready-made numbers**: a column, a grid cell, a reading sequence, an event code. How a text or an image is turned into numbers, **against which baseline** that conversion is measured and what the measures of the field itself are was never asked. The next course, Natural Language Processing and Computer Vision, opens with this question.