Lesson 04 / 24
Loss Functions
Measuring how the choice of loss moves the gradient the network sees and the number it reaches: when the network says 0.99 and the true label is 0, the gradient that cross-entropy gives to the output layer is 0.9900 and that of squared error 0.0196 — a ratio of 50.5. When the same network is trained with two losses on four candidates, validation picks squared error and 25 epochs; the test number of that candidate is 0.7619, its second run 0.7897, a difference of seven subscribers. What the choice of loss adds is three subscribers. While the training loss always falls from 0.4861 to 0.3279 and the training accuracy rises from 0.78 to 0.86, the test accuracy peaks at 0.7937 at epoch 50 and falls to 0.7817; in the same run the number of suspicious subscribers caught rises from 1 to 25.
Published Updated
In this lesson
- Compares the gradient that cross-entropy and squared error give to the output layer at the same output values and writes down their ratio
- Trains the same network with two losses and places the number that the choice of loss moves next to the run-to-run variability
- Proves that loss is not a metric by showing a range in which the loss falls while the held-out set metric gets worse
Contents
In three lessons the output of the network was always a probability, and that probability was always compared with the target in the same way: the difference was taken and passed back. This comparison is a choice. The number passed back is the gradient of the loss function with respect to the output layer, and when the loss changes, that number changes too.
The loss does two jobs at once: during training it tells the network which way to go, and when training is over it is read, or one tries to read it, as the number that reports performance. It was not designed for the second job, and the second measure of the lesson shows this.
- NN22. The set, the split, the seed, the twelve columns and the training set reduced to 300
rows are as in
01–03; the network is 12–8–1, 113 parameters. The baseline on the test set is 0.7579. - NN23. The classification losses are cross-entropy and squared error. Both are computed over the same sigmoid output; the only thing that changes is the gradient given to the output layer.
- NN24. The output gradient of cross-entropy is ; the derivative of the sigmoid cancels against the derivative of the loss. The gradient of squared error is and it keeps carrying the derivative of the sigmoid.
- NN25. In cross-entropy the probability is bounded from below by as overflow protection. The bound does not affect the number, because no probability the network produces comes near that value.
- NN26. The regression losses are not trained in this lesson; only the gradient behavior is shown by calculation. In the Regression Metrics lesson of the Supervised Learning course two fits were measured: explained variance 0.4770 and 0.4751.
- NN27. The number of candidates tried is 4: two losses times two epoch budgets (25 and 100). The choice is made from the validation set, by looking at the sum of the two runs.
- NN28. The second metric is the number of suspicious subscribers caught: the real suspicious subscribers flagged at the 0.5 threshold, out of the 61 suspicious ones in the test set. The design of the metric was measured in the Classification Metrics lesson of the Supervised Learning course.
Choosing the Loss Changes the Gradient
The difference between the two losses shows not in the loss value but in the derivative. The network says 0.99 to a subscriber and the true label is 0. This is one of the worst errors possible; how the loss reacts to it can be measured.
# loss.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split; # the training set is reduced to 300 rows as in 01. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def make_set(data): return ([[(x[a] - MEAN[a]) / STD[a] for a in FIELDS] for x in data], [float(x["suspicious"]) for x in data]) X_TR, Y_TR = make_set(TRAIN[:300]) # REDUCED training set X_VA, Y_VA = make_set(VAL) X_TE, Y_TE = make_set(TEST) def squash(z): # S-shaped function, overflow-protected if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def rectified(z): return z if z > 0.0 else 0.0 def init_weights(seed, sizes): # gain-scaled initialization, bias zero r, net = rng(seed), [] for a, b in zip(sizes, sizes[1:]): scale = math.sqrt(2.0 / a) net.append(([[(r() * 2 - 1) * scale for _ in range(a)] for _ in range(b)], [0.0] * b)) return net def matmul(W, bias, a): # MATRIX MULTIPLICATION by hand: each row is a unit return [sum(w * v for w, v in zip(row, a)) + b for row, b in zip(W, bias)] def forward(net, x): # FORWARD PASS: layer by layer a = x for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(t) for t in z] return a LOSSES = { # (loss, gradient at the output layer) "cross-entropy": ( lambda a, y: -(math.log(max(a, 1e-12)) if y else math.log(max(1 - a, 1e-12))), lambda a, y: a - y), "squared error": ( lambda a, y: (a - y) ** 2, lambda a, y: 2.0 * (a - y) * a * (1.0 - a)), } print(f"train {len(X_TR)}, validation {len(X_VA)}, test {len(X_TE)}, " f"baseline {BASELINE:.4f}\n") print("classification, true label 0:") print(f"{'output':>7} {'cross loss':>13} {'cross grad':>12} {'squared loss':>12} " f"{'squared grad':>12} {'ratio':>7}") for a in (0.01, 0.10, 0.50, 0.90, 0.99): cl, cg = LOSSES["cross-entropy"] sl, sg = LOSSES["squared error"] print(f"{a:>7.2f} {cl(a, 0.0):>13.4f} {cg(a, 0.0):>12.4f} {sl(a, 0.0):>12.4f} " f"{sg(a, 0.0):>12.4f} {cg(a, 0.0) / sg(a, 0.0):>7.1f}") print("\nregression, residual r:") print(f"{'r':>7} {'squared loss':>12} {'squared grad':>12} {'absolute loss':>13} " f"{'absolute grad':>13} {'ratio':>7}") for r in (0.5, 1.0, 2.0, 5.0, 10.0): print(f"{r:>7.1f} {r * r:>12.4f} {2 * r:>12.4f} {r:>13.4f} {1.0:>13.4f} " f"{2 * r:>7.1f}")
train 300, validation 252, test 252, baseline 0.7579
classification, true label 0:
output cross loss cross grad squared loss squared grad ratio
0.01 0.0101 0.0100 0.0001 0.0002 50.5
0.10 0.1054 0.1000 0.0100 0.0180 5.6
0.50 0.6931 0.5000 0.2500 0.2500 2.0
0.90 2.3026 0.9000 0.8100 0.1620 5.6
0.99 4.6052 0.9900 0.9801 0.0196 50.5
regression, residual r:
r squared loss squared grad absolute loss absolute grad ratio
0.5 0.2500 1.0000 0.5000 1.0000 1.0
1.0 1.0000 2.0000 1.0000 1.0000 2.0
2.0 4.0000 4.0000 2.0000 1.0000 4.0
5.0 25.0000 10.0000 5.0000 1.0000 10.0
10.0 100.0000 20.0000 10.0000 1.0000 20.0
The last row decides the classification question. When the network says 0.99 and is wrong,
cross-entropy sends 0.9900 to the output layer and squared error 0.0196 — there is a 50.5-fold
difference between them. Because the gradient of squared error carries the derivative of the sigmoid,
the more confidently the network is wrong, the weaker the correction becomes. The saturation measured
in 02 meets us here once more, this time through the loss.
The regression table shows the same logic in the opposite direction. When the residual is 10 units, the gradient of squared error is 20 and that of absolute error 1. Squared error brings large residuals to the front and a single outlying observation pulls the fit toward itself; absolute error gives every residual the same weight. Which one is right is a question not of the data but of the cost of the error.
The Same Network, Two Losses
Whether the difference in derivative turns into a number can be understood only by training. The same network, the same starting values, the same learning rate; the only things that change are the loss and the epoch budget.
def train_model(net, X, Y, rate, epochs, loss_name, watch=()): # MODEL: full-batch gradient descent loss_fn, output_grad = LOSSES[loss_name] n, trace = len(X), [] for t in range(epochs): gW = [[[0.0] * len(W[0]) for _ in W] for W, _ in net] gb = [[0.0] * len(bias) for _, bias in net] total = 0.0 for i in range(n): a, acts = X[i], [X[i]] for k, (W, bias) in enumerate(net): z = matmul(W, bias, a) a = [squash(z[0])] if k == len(net) - 1 else [rectified(q) for q in z] acts.append(a) total += loss_fn(acts[-1][0], Y[i]) d = [output_grad(acts[-1][0], Y[i])] for k in range(len(net) - 1, -1, -1): W, a = net[k][0], acts[k] for j, dj in enumerate(d): if dj: s = gW[k][j] for p in range(len(a)): s[p] += dj * a[p] gb[k][j] += dj if k: prev = [0.0] * len(a) for j, dj in enumerate(d): if dj: s = W[j] for p in range(len(a)): prev[p] += dj * s[p] d = [prev[p] if a[p] > 0 else 0.0 for p in range(len(a))] for k, (W, bias) in enumerate(net): for j in range(len(W)): s, g = W[j], gW[k][j] for p in range(len(s)): s[p] -= rate * g[p] / n bias[j] -= rate * gb[k][j] / n if (t + 1) in watch: trace.append((t + 1, total / n, accuracy(net, X_TR, Y_TR), accuracy(net, X_VA, Y_VA), accuracy(net, X_TE, Y_TE), caught(net, X_TE, Y_TE))) return net, trace def accuracy(net, X, Y): return sum((forward(net, x)[0] > 0.5) == (y > 0.5) for x, y in zip(X, Y)) / len(X) def caught(net, X, Y): # real suspicious subscribers flagged at the 0.5 threshold return sum(1 for x, y in zip(X, Y) if y > 0.5 and forward(net, x)[0] > 0.5) CANDIDATES = {} print(f"{'loss':<20} {'epoch':>5} {'run':>6} {'train_loss':>10} {'validation':>10} " f"{'test':>8}") for loss_name in LOSSES: for run in (1, 2): _, trace = train_model(init_weights(SEED + run, [12, 8, 1]), X_TR, Y_TR, 0.3, 100, loss_name, watch=(25, 100)) for epoch, loss, _, val, test, _ in trace: CANDIDATES.setdefault((loss_name, epoch), []).append((val, test)) print(f"{loss_name:<20} {epoch:>5} {run:>6} {loss:>10.4f} {val:>10.4f} {test:>8.4f}") CHOSEN = max(CANDIDATES, key=lambda k: sum(t[0] for t in CANDIDATES[k])) print(f"\ncandidates tried {len(CANDIDATES)}, chosen from validation: {CHOSEN[0]}, " f"{CHOSEN[1]} epochs") print(f"baseline {BASELINE:.4f}, network number {CANDIDATES[CHOSEN][0][1]:.4f}, " f"second run {CANDIDATES[CHOSEN][1][1]:.4f}, difference " f"{round(abs(CANDIDATES[CHOSEN][0][1] - CANDIDATES[CHOSEN][1][1]) * len(X_TE))} subscribers") best = {k: max(t[1] for t in CANDIDATES[k]) for k in CANDIDATES} print(f"loss choice (best run of each candidate) {min(best.values()):.4f}-{max(best.values()):.4f} " f"= {round((max(best.values()) - min(best.values())) * len(X_TE))} subscribers")
loss epoch run train_loss validation test cross-entropy 25 1 0.4341 0.8135 0.7778 cross-entropy 100 1 0.3601 0.8214 0.7857 cross-entropy 25 2 0.4152 0.8333 0.7778 cross-entropy 100 2 0.3648 0.8056 0.7698 squared error 25 1 0.1556 0.8135 0.7619 squared error 100 1 0.1312 0.8214 0.7857 squared error 25 2 0.1536 0.8373 0.7897 squared error 100 2 0.1232 0.8056 0.7738 candidates tried 4, chosen from validation: squared error, 25 epochs baseline 0.7579, network number 0.7619, second run 0.7897, difference 7 subscribers loss choice (best run of each candidate) 0.7778-0.7897 = 3 subscribers
The loss columns cannot be compared directly, because the two losses produce numbers on two different scales: 0.4341 against 0.1556. What can be compared is the test column, and the difference between the two losses is three subscribers, while the difference between the two runs of the same candidate is seven subscribers.
The epoch budget makes the gradient difference visible. At twenty-five epochs squared error stays at 0.7619 while cross-entropy reaches 0.7778; at a hundred epochs both rise to 0.7857. The weak gradient of squared error does not stop learning, it delays it.
Loss Is Not a Metric
Until now the loss only steered the training. The table below follows the same run epoch by epoch and puts three numbers side by side: the training loss, the test accuracy and the number of suspicious subscribers caught.
WATCH = (10, 25, 50, 100, 200) print(f"suspicious subscribers in the test set {int(sum(Y_TE))}") print(f"{'run':>6} {'epoch':>5} {'train_loss':>10} {'on_train':>9} {'on_test':>9} " f"{'caught':>10}") TRACES = {} for run in (1, 2): _, trace = train_model(init_weights(SEED + run, [12, 8, 1]), X_TR, Y_TR, 0.3, WATCH[-1], "cross-entropy", watch=WATCH) TRACES[run] = trace for epoch, loss, acc_train, _, acc_test, n_caught in trace: print(f"{run:>6} {epoch:>5} {loss:>10.4f} {acc_train:>9.4f} {acc_test:>9.4f} " f"{n_caught:>10}") for run in (1, 2): trace = TRACES[run] peak = max(trace, key=lambda t: t[4]) print(f"run {run}: loss {trace[0][1]:.4f} -> {trace[-1][1]:.4f} (always falling), " f"test highest at epoch {peak[0]}: {peak[4]:.4f}, " f"at epoch {WATCH[-1]}: {trace[-1][4]:.4f}") print(f" caught suspicious {trace[0][5]} -> {trace[-1][5]}, " f"highest {max(t[5] for t in trace)}")
suspicious subscribers in the test set 61
run epoch train_loss on_train on_test caught
1 10 0.4861 0.7800 0.7619 1
1 25 0.4341 0.7933 0.7778 7
1 50 0.3899 0.8133 0.7937 20
1 100 0.3601 0.8400 0.7857 21
1 200 0.3279 0.8600 0.7817 25
2 10 0.4995 0.7967 0.7817 10
2 25 0.4152 0.8233 0.7778 16
2 50 0.3805 0.8233 0.7738 23
2 100 0.3648 0.8367 0.7698 25
2 200 0.3518 0.8400 0.7738 25
run 1: loss 0.4861 -> 0.3279 (always falling), test highest at epoch 50: 0.7937, at epoch 200: 0.7817
caught suspicious 1 -> 25, highest 25
run 2: loss 0.4995 -> 0.3518 (always falling), test highest at epoch 10: 0.7817, at epoch 200: 0.7738
caught suspicious 10 -> 25, highest 25
The loss column is one-directional in both runs: it always falls. The training accuracy is one-directional too, from 0.78 to 0.86. The test accuracy is not. In the first run it peaks at 0.7937 at epoch 50 and drops to 0.7817 at epoch 200; in the second run the peak is at epoch 10 and after that it only goes down. The epochs that keep pushing the loss down make the accuracy on the held-out set worse.
The last column completes the distinction. The number of suspicious subscribers caught rises in both runs: from 1 to 25 and from 10 to 25. At the same moment of the same training, one metric gets worse while another gets better. At epoch 10 the network catches only one of the 61 suspicious subscribers, yet its accuracy is 0.7619; at epoch 200 it catches 25 of them and its accuracy is 0.7817. The sentence “the network was best at epoch 50” is true for the one who reads accuracy and false for the one who reads the suspicious subscribers caught. At which epoch to stop is a question not of the loss but of the metric and of the cost behind the metric; that calculation is the subject of the Threshold Selection and Curves lesson of the Supervised Learning course.
Summary
- The choice of loss is decisive not in the loss value but in the derivative. When the network says 0.99 and is wrong, the output gradient of cross-entropy is 0.9900 and that of squared error 0.0196 — a ratio of 50.5.
- In regression the same logic works in the opposite direction: when the residual is 10 units, the gradient of squared error is 20 and that of absolute error 1. Squared error brings large residuals to the front, absolute error gives all of them equal weight.
- The triple: baseline 0.7579, network number 0.7619, second run 0.7897. What the choice of loss adds is three subscribers, the run difference of the same candidate is seven subscribers.
- The weak gradient of squared error does not stop learning, it delays it: at 25 epochs they separate at 0.7619 and 0.7778, at 100 epochs both come to 0.7857.
- Loss is not a metric. While the training loss always falls from 0.4861 to 0.3279, the test accuracy peaks at epoch 50 and drops, and the number of suspicious subscribers caught rises from 1 to 25.
Next Step
In four lessons the gradient always arrived ready-made. How the number at the output layer is found was written in this lesson, but how that number is distributed over the weights of the hidden layer was never opened. The next lesson opens that box: the chain rule is applied layer by layer over the network and the resulting gradient is compared with the numerical derivative. The measure is two numbers — the largest difference between the hand-computed gradient and the numerical derivative, and how many forward passes the two methods take.
To keep your progress and take notes, Log in
My notes
Log in to take notes.