--- title: 'Supervised Learning' source: 'https://academia.sh/en/courses/supervised-learning' language: en updated: '2026-10-21T05:03:29+00:00' license: 'CC BY-SA 4.0' lessons: 20 outcomes: - 'Applies regression and classification families with their assumptions and measures the difference each family adds on the held-out set' - 'Chooses the appropriate evaluation metric for the problem context and shows how the metric changes the ranking' - 'Runs a tuning search and separates the difference the budget inflates by writing the number of candidates tried next to the number found' --- # Supervised Learning Reading the difference a model family adds when the features are held fixed and only the model is changed, together with the number of candidates tried to find that difference: measuring least squares as an estimate, the two numbers that polynomial degree separates, the sparsity produced by squared, absolute and mixed penalties, the continuity of a probability output, the cost of interpretability, the ratio of the neighbor count to the curse of dimensionality, probability and ranking diverging when conditional independence breaks, margin width and the number of support vectors, how the split criterion and pruning differ from a depth limit, the variance that bagging reduces, the trade-off between learning rate and round count in boosting, which set the second layer of stacking is learned from, the different error distributions of two models with the same accuracy, the ranking that changes when the metric changes, the two numbers a threshold moves, the gap between the stated probability and the observed rate, the swaps among three regression metrics, the fold count moving the estimate, the validation count that inflates as the search budget grows, and the slice where the error concentrates. ## Linear Models Measuring by prediction performance the families that build the decision as a line or a plane: the assumptions checked with the least squares number on the held-out set and the structure of the residuals, the divergence of the train and held-out numbers as the polynomial degree is swept from one to eight and the set from which the choice was read, how many coefficients the ridge, lasso and elastic net penalties set exactly to zero and what sparsity is worth on the held-out set, the continuity of the probability output independent of the threshold and the decision boundary, and the score paid for a readable coefficient together with what the coefficient does not say. ### Linear Regression Source: https://academia.sh/en/courses/supervised-learning/linear-regression Measuring least squares as a prediction procedure rather than an explanation tool and establishing the three numbers of the course: on the test set the baseline gives a root error of 6.5324 m³ while the linear model gives 5.3096, the difference is 1.2228 m³ and the number of candidates tried is 1. The mean of the residuals is exactly zero per region on the train set and wanders between -1.0769 and +1.3685 on the test set; the residual spread rises from 3.566 to 7.137 as the prediction grows. When the 127 subsets of the seven columns are tried, the subset chosen on validation falls behind the full model with 5.3157 on test; had it been chosen on test, 5.2931 would have been reported and the difference is 0.0226 m³. The previous course measured every step of the features in twelve lessons and left two accounts open at its close. The first: throughout all those measurements the model was always chosen from the same two or three families — a decision tree of depth 6, a gradient-based linear model, a distance-based procedure — and **what the family itself contributes** was never asked. The second: why the depth is six and why the number of neighbors is fifteen, and **which set those numbers should be read from**, was never measured. The same closing counted that the test set had been looked at at least twenty-four times. This course takes up both questions together and changes the measure for that purpose. The features are **held constant**; the only thing that changes is the model. In every lesson three numbers stand side by side: the **baseline**, the **number of the family on the held-out set**, and the **number of candidates tried** to find that number. The third is new and is the rule of the course: **a model's number cannot be read without writing how many candidates were tried to find it.** A number whose tuning budget is not written is an unmeasured number. This lesson establishes the triple and starts with the simplest family, least squares. - **LM1.** The set is the **fictional** subscriber table of the Introduction to Machine Learning and Data Preparation and Feature Engineering courses; it is rebuilt with the same seed (20260218). The split is 756 train, 252 validation and 252 test rows and does not change throughout the course. - **LM2.** In the classification lessons the target is the subscriber carrying an illicit-use or fault suspicion, and the baseline is **saying the most frequent class**: 0.7579 on the test set. - **LM3.** In the regression lessons the target is the subscriber's **mean consumption over up to three periods of readings** (m³) and the baseline is **the mean of the train set**: saying the same number to every subscriber. - **LM4.** The regression feature is not derived from any reading that enters the computation of the target. Volatility, zero readings and per-person consumption come out of the same reading series, so they are **not used**; what remains is region, household size, satisfaction and period count. - **LM5.** The sum of the five region indicators equals one in every row, that is, the constant term. One of them (north) is left out and becomes the **reference**. - **LM6.** The measure is the **root mean squared error** (m³); the mean absolute error is printed next to it. Where the two diverge is left to `evaluation-and-tuning/05`. - **LM7.** The coefficient of determination is defined **relative to the train mean**: zero means making the same error as the baseline. It can also come out negative on a held-out set. - **LM8.** The **causal** reading of a coefficient is not the subject of this course. The sign flipping under a confounding variable was measured in the Applied Statistics topic of the Statistical Analysis and Exploratory Data Analysis course; it is not repeated here. - **LM9.** The test set is looked at in each lesson only after the choice is finished, and how many times it was looked at is written. #### Three Numbers The setup produces the set, splits it and prints the two baselines. ```python # regression.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the target is the subscriber's three-period mean consumption (m3). import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "region_name": k["region"][0], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "target": round(mean, 2), "suspicious": int(z > 1.35)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] MEAN = sum(x["target"] for x in TRAIN) / len(TRAIN) def rmse(predict, s): return math.sqrt(sum((predict(x) - x["target"]) ** 2 for x in s) / len(s)) def mae(predict, s): return sum(abs(predict(x) - x["target"]) for x in s) / len(s) print(f"subscribers {len(DATA)}, train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") print(f"classification baseline (most frequent class, on test) " f"{sum(x['suspicious'] == 0 for x in TEST) / len(TEST):.4f}") print(f"regression baseline (train mean) {MEAN:.4f} m3\n") print(f"{'split':<12} {'rmse':>9} {'mean abs err':>12}") for name, s in (("train", TRAIN), ("validation", VAL), ("test", TEST)): print(f"{name:<12} {rmse(lambda x: MEAN, s):>9.4f} {mae(lambda x: MEAN, s):>12.4f}") ``` ``` subscribers 1260, train 756, validation 252, test 252 classification baseline (most frequent class, on test) 0.7579 regression baseline (train mean) 19.7665 m3 split rmse mean abs err train 6.1995 4.9017 validation 6.4193 5.1148 test 6.5324 5.0806 ``` Both baselines are non-learning procedures and both stay constant throughout this course. In classification the number 0.7579 is carried over as it is from the previous two courses. In regression the baseline says 19.7665 m³ to every subscriber and gives a root error of 6.5324 m³ on the test set: the zero point against which all the remaining numbers will be measured. #### Least Squares Is a Prediction Procedure Least squares solves for the coefficients that minimize the sum of squared errors. The solution is in closed form: a system of equations built from the products of the columns with one another is solved by elimination. ```python FIELDS = ["area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period"] def solve(A, b): # elimination with partial pivoting n = len(b) for i in range(n): p = max(range(i, n), key=lambda k: abs(A[k][i])) A[i], A[p], b[i], b[p] = A[p], A[i], b[p], b[i] for k in range(i + 1, n): f = A[k][i] / A[i][i] for j in range(i, n): A[k][j] -= f * A[i][j] b[k] -= f * b[i] w = [0.0] * n for i in range(n - 1, -1, -1): w[i] = (b[i] - sum(A[i][j] * w[j] for j in range(i + 1, n))) / A[i][i] return w def row(x, fields): return [1.0] + [float(x[a]) for a in fields] def least_squares(s, fields): # MODEL: linear regression X = [row(x, fields) for x in s] y = [x["target"] for x in s] p = len(X[0]) A = [[sum(X[i][a] * X[i][b] for i in range(len(X))) for b in range(p)] for a in range(p)] return solve(A, [sum(X[i][a] * y[i] for i in range(len(X))) for a in range(p)]) def predictor(w, fields): return lambda x: sum(a * b for a, b in zip(w, row(x, fields))) def determination(predict, s): # coefficient of determination return 1 - (sum((predict(x) - x["target"]) ** 2 for x in s) / sum((MEAN - x["target"]) ** 2 for x in s)) W = least_squares(TRAIN, FIELDS) f = predictor(W, FIELDS) print(f"intercept {W[0]:>8.3f} (north is the reference)") for i, a in enumerate(FIELDS): print(f"{a:<12} {W[i + 1]:>8.3f}") print(f"\n{'split':<12} {'baseline rmse':>13} {'model rmse':>10} {'difference':>10} " f"{'mae':>8} {'determination':>13}") for name, s in (("train", TRAIN), ("validation", VAL), ("test", TEST)): t = rmse(lambda x: MEAN, s) print(f"{name:<12} {t:>13.4f} {rmse(f, s):>10.4f} {t - rmse(f, s):>+10.4f} " f"{mae(f, s):>8.4f} {determination(f, s):>13.4f}") print(f"\ncandidates tried 1") ``` ``` intercept 21.820 (north is the reference) area_south -3.383 area_east 4.598 area_west -5.851 area_central 2.340 household 0.062 satisfaction -0.148 period -0.550 split baseline rmse model rmse difference mae determination train 6.1995 5.0757 +1.1238 3.7461 0.3297 validation 6.4193 5.3225 +1.0969 4.0005 0.3125 test 6.5324 5.3096 +1.2228 4.0020 0.3393 candidates tried 1 ``` The triple is complete: baseline 6.5324, the family's number 5.3096, candidates tried 1. The difference is 1.2228 m³, that is, 18.7% of the baseline. That the number of candidates is one is the privilege of this lesson — the column set was written in advance, there is no hyperparameter, and the solution came in closed form in a single pass. In the remaining lessons this number will never be one again. The same coefficient list can be put to two different jobs, and the two are not the same thing. The region coefficients give the order of the base consumption levels in the setup correctly: east is the highest, west the lowest. But the measure of this lesson is not the correctness of a coefficient, it is **the error made on a subscriber not seen before**. The coefficient of determination on the test set is 0.3393 and is defined relative to the baseline: zero means making the same error as saying the train mean to every subscriber. #### Assumptions Are Checked with Numbers Linear regression has assumptions: the mean of the residuals is zero, their spread is independent of the prediction, the tails are symmetric. These are checked not as theory but as numbers. ```python def residuals(predict, s): return [predict(x) - x["target"] for x in s] def resid_mean(v): # a residual at machine precision is zero m = sum(v) / len(v) return m if abs(m) > 5e-5 else 0.0 print(f"{'region':<9} {'train mean':>13} {'test n':>11} {'test mean':>13}") for b in REGION: e = [x for x in TRAIN if x["region_name"] == b[0]] v = [x for x in TEST if x["region_name"] == b[0]] print(f"{b[0]:<9} {resid_mean(residuals(f, e)):>+13.4f} {len(v):>11} " f"{resid_mean(residuals(f, v)):>+13.4f}") ordered = sorted(TEST, key=f) n = len(ordered) // 4 print(f"\n{'slice':<7} {'mean prediction':>16} {'mean residual':>14} {'residual spread':>16}") for k in range(4): d = ordered[k * n:(k + 1) * n] if k < 3 else ordered[3 * n:] v = residuals(f, d) m = sum(v) / len(v) sd = math.sqrt(sum((q - m) ** 2 for q in v) / len(v)) print(f"slice {k + 1:<1} {sum(f(x) for x in d) / len(d):>16.2f} {m:>+14.3f} " f"{sd:>16.3f}") v = residuals(f, TEST) m = sum(v) / len(v) sd = math.sqrt(sum((q - m) ** 2 for q in v) / len(v)) print(f"\nresiduals beyond two spreads on test " f"{sum(1 for q in v if abs(q - m) > 2 * sd)}, " f"negative {sum(1 for q in v if q - m < -2 * sd)}, " f"positive {sum(1 for q in v if q - m > 2 * sd)}") ``` ``` region train mean test n test mean north +0.0000 68 -1.0370 south +0.0000 58 -0.0391 east +0.0000 36 -1.0769 west +0.0000 32 +1.3685 central +0.0000 58 +0.5708 slice mean prediction mean residual residual spread slice 1 15.33 +1.192 3.566 slice 2 18.67 -1.007 4.746 slice 3 21.29 -0.237 4.883 slice 4 23.85 -0.498 7.137 residuals beyond two spreads on test 17, negative 11, positive 6 ``` The first column is not a check but an identity. On the train set the mean residual of every region is **exactly zero**, and the reason is not that the model is good: the region indicators are inside the model as columns, and the least squares solution makes every column orthogonal to the residuals. This number can never come out different from zero, so it says nothing. The column that speaks is the third: the same means wander between -1.0769 and +1.3685 on the test set. On the thirty-two subscribers in the west region the model says 1.37 m³ too much on average. The second table shows a sharper breakdown. The spread of the residuals doubles as the prediction grows, from 3.566 to 7.137. The model is far more accurate on low-consumption subscribers and far coarser on high-consumption ones. The reason the equal-spread assumption fails lies in the setup itself: consumption is generated from a multiplicative distribution, so the scatter of large values is also large. The last line gives the same asymmetry from the other direction — of the seventeen residuals beyond two spreads, eleven lie in one direction and six in the other. Broken assumptions would invalidate the margin of play of the coefficients in an explanation study; the Applied Statistics topic of the Statistical Analysis and Exploratory Data Analysis course did that computation. For a prediction target the measure is single and has already been taken: 5.3096. The assumption check here does not invalidate the number, it says **where the number is bad**. #### A Number Whose Tuning Budget Is Not Written Is an Unmeasured Number In this lesson a single candidate was tried. What happens if all the non-empty subsets of the seven columns are tried? ```python import itertools best_val, best_test, candidates = None, None, 0 for r in range(1, len(FIELDS) + 1): for c in itertools.combinations(FIELDS, r): candidates += 1 g = predictor(least_squares(TRAIN, list(c)), list(c)) d, s = rmse(g, VAL), rmse(g, TEST) if best_val is None or d < best_val[0]: best_val = (d, s, c) if best_test is None or s < best_test[1]: best_test = (d, s, c) print(f"candidates tried {candidates}") print(f"full model validation {rmse(f, VAL):.4f} test {rmse(f, TEST):.4f}") print(f"chosen on validation validation {best_val[0]:.4f} " f"test {best_val[1]:.4f} {len(best_val[2])} columns") print(f"chosen on test validation {best_test[0]:.4f} " f"test {best_test[1]:.4f} {len(best_test[2])} columns") print(f"number reported if chosen on test {best_test[1]:.4f}, " f"honest {best_val[1]:.4f}, difference {best_val[1] - best_test[1]:.4f} m3") print(f"excluded column: " f"{[a for a in FIELDS if a not in best_val[2]]}") ``` ``` candidates tried 127 full model validation 5.3225 test 5.3096 chosen on validation validation 5.3073 test 5.3157 6 columns chosen on test validation 5.4150 test 5.2931 6 columns number reported if chosen on test 5.2931, honest 5.3157, difference 0.0226 m3 excluded column: ['satisfaction'] ``` Three rows give three different numbers, and all three come from the same model family, the same features, the same split. The only differences among them are **how many candidates were tried** and **which set the choice was read from**. The choice read from the validation set drops the satisfaction column and beats the full model's 5.3225 on validation with 5.3073. The same choice gives 5.3157 on test — that is, **behind the full model**. Choosing among a hundred and twenty-seven candidates by looking at validation produced a worse result than not choosing at all; searching 127 times for the smallest value on a set of 252 rows also selects that set's own scatter. The last two lines turn the rule of the course into a number. Had the choice been made on the test set, the reported number would have been 5.2931; the honest number is 5.3157 and the difference is 0.0226 m³. The sign of the difference is fixed — the number on the set where the choice was made can never come out worse. This is why the comparison between 5.3096 and 5.2931 carries meaning only when 1 and 127 are written next to them. #### Summary - The three numbers of the course were established: the baseline, the number of the family on the held-out set, and the number of candidates tried. The baseline is 0.7579 in classification and 6.5324 m³ in regression. - Least squares gives 5.3096 m³ on the test set with seven columns; the difference is 1.2228 m³ and the number of candidates tried is 1. This is the only lesson in the course where the number of candidates is one. - The mean residual is exactly zero per region on the train set, and this is not a check but the identity of the least squares solution; the same mean wanders between -1.0769 and +1.3685 on the test set. - The equal-spread assumption does not hold: the residual spread rises from 3.566 to 7.137 as the prediction grows. For a prediction target this does not invalidate the number; it says where the number is bad. - When the 127 subsets of the seven columns are tried, the subset chosen on validation falls behind the full model with 5.3157 on test; had it been chosen on test, 5.2931 would have been reported and the difference is 0.0226 m³. #### Next Step In this lesson the model was a plane and the columns were used as given. Yet the relation between household size and consumption need not be a line; the consumption that grows in going from a two-person household to a four-person one may not grow at the same rate in going from four to six. There is a way to build a nonlinear relation with a linear model, and it consists only of adding columns: the square, the cube, the pairwise products. The next lesson measures this expansion by degree — the degree is swept from 1 to 8, the train and held-out numbers of each degree are printed side by side, the choice is read from the validation set, and which degree would have been chosen had it been read from the test set is shown in the same table. ### Polynomial Regression Source: https://academia.sh/en/courses/supervised-learning/polynomial-regression Building a nonlinear relation with a linear model by adding columns, and the degree pulling two numbers apart: when the degree is swept from 1 to 8 the number of columns rises from 7 to 168, the train root error falls from 5.0757 to 4.8437 m³, and the validation root error bottoms out at 5.2569 and climbs to 5.6877. The degree read from validation is 4 and gives 5.3775 on test; had it been read from test, degree 5 would have been chosen and 5.2009 reported, a difference of 0.1766 m³. The degree chosen on validation is 0.0679 m³ behind degree 1 on test. Four features produce only 450 discrete combinations; the ceiling model that gives every cell the train mean yields 3.8418 on train and 6.3887 on test, and 33 test subscribers fall into a cell never seen in training. The previous lesson built the model as a plane and used the columns as given: household size one column, satisfaction one column, period count one column. What that model says is that when household size rises by one, consumption rises by **the same amount everywhere**. There is no reason for this to be true. The consumption that grows in going from a two-person household to a four-person one may not grow at the same rate in going from four to six; the effect of the region may change with household size. Building this relation does not require changing the model family. A linear model is linear in its **coefficients**, not in its columns. When the square of household size is added as a column, the model is still solved by least squares, but now it carries curvature. **Polynomial features** is the name of this expansion, and it has a single hyperparameter: the degree. This lesson sweeps the degree from 1 to 8, prints the train and held-out numbers of each degree side by side, and shows which set the choice was read from as two separate numbers. - **LM10.** The set, the split and the target are as in the previous lesson: the same fictional subscriber table, seed 20260218, 756/252/252, the target is mean consumption (m³), the baseline is the train mean and a root error of 6.5324 m³ on the test set. - **LM11.** The polynomial is expanded over three numeric columns only: household size, satisfaction, period count. For degree $d$, all the monomials that satisfy $1 \le i+j+k \le d$ are added. - **LM12.** The four region indicators stay as they are at every degree; they do not enter the polynomial. North is the reference as in the previous lesson. - **LM13.** The columns are scaled with the mean and spread learned from the train set. The eighth power of household size exceeds one million; without scaling the elimination collapses numerically. - **LM14.** At high degrees the columns turn into linear combinations of one another, because household size takes only six discrete values, satisfaction five and period count three. To make the system of equations solvable, $10^{-6}$ is added to the diagonal. This is not a regularization decision but a numerical threshold set against a singular system; the penalty decision is the subject of the next lesson. - **LM15.** The number of candidates tried is **8**: eight degrees, no other tuning. - **LM16.** The choice is read from the **validation set**. The test set numbers appear in the table but do not enter the choice; the reason they appear is to measure the difference between the two choices. - **LM17.** The ceiling model is the procedure that gives every discrete combination of the same four features the train set mean. No model built with these features can be better than it on the train set. #### As the Degree Grows, Two Numbers Diverge The setup is the setup of the previous lesson; polynomial features and the degree sweep are added on top of it. ```python # polynomial.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue x = {"region_name": k["region"][0], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "target": round(sum(v) / len(v), 2)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] MEAN = sum(x["target"] for x in TRAIN) / len(TRAIN) def rmse(predict, s): return math.sqrt(sum((predict(x) - x["target"]) ** 2 for x in s) / len(s)) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, " f"baseline on test {rmse(lambda x: MEAN, TEST):.4f} m3") ``` ``` train 756, validation 252, test 252, baseline on test 6.5324 m3 ``` Polynomial features, scaling and the solution stand together. The scaling step follows the rule of the previous course and has two faces: **fitting** only from the train set, **applying** to every set. ```python import itertools NUMERIC = ["household", "satisfaction", "period"] INDICATORS = ["area_south", "area_east", "area_west", "area_central"] def powers(degree): # all monomials with 1 <= i+j+k <= degree return [u for u in itertools.product(range(degree + 1), repeat=len(NUMERIC)) if 1 <= sum(u) <= degree] def raw(x, pw): return ([float(x[a]) for a in INDICATORS] + [math.prod(x[a] ** k for a, k in zip(NUMERIC, u)) for u in pw]) def fit_scale(train, pw): # LEARN: from the train set only H = [raw(x, pw) for x in train] p = len(H[0]) o = [sum(r[j] for r in H) / len(H) for j in range(p)] s = [max(1e-9, math.sqrt(sum((r[j] - o[j]) ** 2 for r in H) / len(H))) for j in range(p)] return o, s def apply(x, pw, o, s): # APPLY h = raw(x, pw) return [1.0] + [(h[j] - o[j]) / s[j] for j in range(len(h))] def solve(A, b): n = len(b) for i in range(n): p = max(range(i, n), key=lambda k: abs(A[k][i])) A[i], A[p], b[i], b[p] = A[p], A[i], b[p], b[i] for k in range(i + 1, n): f = A[k][i] / A[i][i] for j in range(i, n): A[k][j] -= f * A[i][j] b[k] -= f * b[i] w = [0.0] * n for i in range(n - 1, -1, -1): w[i] = (b[i] - sum(A[i][j] * w[j] for j in range(i + 1, n))) / A[i][i] return w EPSILON = 1e-6 # numerical threshold that makes the singular system solvable def polynomial(train, pw, o, s): # MODEL: polynomial regression X = [apply(x, pw, o, s) for x in train] y = [x["target"] for x in train] p = len(X[0]) A = [[sum(X[i][a] * X[i][b] for i in range(len(X))) + (EPSILON if a == b and a else 0.0) for b in range(p)] for a in range(p)] return solve(A, [sum(X[i][a] * y[i] for i in range(len(X))) for a in range(p)]) def predictor(w, pw, o, s): return lambda x: sum(a * b for a, b in zip(w, apply(x, pw, o, s))) RESULTS = [] print(f"{'degree':>7} {'columns':>7} {'train':>9} {'validation':>10} {'test':>8} " f"{'weight':>10}") for degree in range(1, 9): pw = powers(degree) o, s = fit_scale(TRAIN, pw) w = polynomial(TRAIN, pw, o, s) g = predictor(w, pw, o, s) RESULTS.append((degree, len(pw) + len(INDICATORS), rmse(g, TRAIN), rmse(g, VAL), rmse(g, TEST))) print(f"{degree:>7} {RESULTS[-1][1]:>7} {RESULTS[-1][2]:>9.4f} {RESULTS[-1][3]:>10.4f} " f"{RESULTS[-1][4]:>8.4f} {math.sqrt(sum(v * v for v in w[1:])):>10.2f}") print(f"\nbaseline on test {rmse(lambda x: MEAN, TEST):.4f} m3, candidates tried 8") ``` ``` degree columns train validation test weight 1 7 5.0757 5.3225 5.3096 3.26 2 13 5.0531 5.3219 5.2648 5.12 3 23 5.0355 5.3184 5.3183 17.22 4 38 5.0013 5.2569 5.3775 212.08 5 59 4.9228 5.2635 5.2009 1520.80 6 87 4.8933 5.3610 5.3224 3487.91 7 123 4.8733 5.4923 5.5168 6959.24 8 168 4.8437 5.6877 5.9085 9344.45 baseline on test 6.5324 m3, candidates tried 8 ``` The train column is one-directional and **improves without interruption**: from 5.0757 to 4.8437. This is not a finding but an identity. The columns of degree $d$ contain the columns of degree $d-1$, so the solution of the smaller degree is also reachable at the larger degree; least squares can only improve on it. It is impossible for the train number to get worse along a growing column set, and for that reason the train number can never be used in choosing the degree. The validation column is two-directional: it falls from 5.3225 to 5.2569, then climbs to 5.6877. The weight column shows the mechanism of the climb. The coefficient magnitude rises from 3.26 to 9,344.45, that is, about three thousand times. High-degree monomials are nearly linear combinations of one another — the sixth power of a column that takes six discrete values is a combination of its lower powers — and the model gives these nearly identical columns huge coefficients that cancel one another. On the train set this is free; on the held-out set those coefficients are not cancelled exactly and the difference turns into error. #### Which Set the Choice Was Read From One degree has to be chosen from the table of eight rows, and this choice can be made from two different sets. ```python best_val = min(RESULTS, key=lambda t: t[3]) best_test = min(RESULTS, key=lambda t: t[4]) first = RESULTS[0] print(f"degree chosen on validation {best_val[0]}: validation " f"{best_val[3]:.4f}, test {best_val[4]:.4f}") print(f"degree if chosen on test {best_test[0]}: validation " f"{best_test[3]:.4f}, test {best_test[4]:.4f}") print(f"difference between the two choices {best_val[4] - best_test[4]:.4f} m3") print(f"degree 1 on test {first[4]:.4f}; degree chosen on validation " f"{best_val[4] - first[4]:+.4f} m3 behind it\n") print(f"{'degree':>7} {'divergence (validation - train)':>31}") for d, cols, e, v, t in RESULTS: print(f"{d:>7} {v - e:>31.4f}") ``` ``` degree chosen on validation 4: validation 5.2569, test 5.3775 degree if chosen on test 5: validation 5.2635, test 5.2009 difference between the two choices 0.1766 m3 degree 1 on test 5.3096; degree chosen on validation +0.0679 m3 behind it degree divergence (validation - train) 1 0.2467 2 0.2687 3 0.2828 4 0.2556 5 0.3407 6 0.4677 7 0.6189 8 0.8440 ``` The first three lines show in a single example why the rule of the course is necessary. The validation set chooses degree 4; that degree gives 5.3775 on test. Had the choice been made from the test set, degree 5 would have come and 5.2009 would have been reported. The 0.1766 m³ between the two numbers is the price of the lesson's eight-candidate budget and comes from nothing the model learned. The fourth line is more unsettling. The degree 4 chosen from validation is **0.0679 m³ behind** degree 1 on test. That is, trying eight candidates and choosing by looking at the validation set produced a worse result than not trying and staying with the simplest model. This is not a calculation error; it is the consequence of the validation set also being 252 rows; the smallest number on that set includes that set's own scatter. The second claim of the course is measured for the first time on this line: **a stronger family did not beat the weaker one.** The divergence column reduces the table to a single number. The difference between validation and train stays between 0.25 and 0.28 up to degree 4, rises to 0.3407 at degree 5 and becomes 0.8440 at degree 8. The place where the divergence begins is degree 5; one step after the place where the validation number bottoms out. #### The Ceiling of These Features At degree 8 there are 168 columns. All of these columns are functions of four features, and those four features produce only a finite number of combinations. ```python cell = {} for x in TRAIN: cell.setdefault((x["region_name"], x["household"], x["satisfaction"], x["period"]), []).append(x["target"]) CELL_MEANS = {k: sum(v) / len(v) for k, v in cell.items()} def cell_prediction(x): # every cell gets the train mean: the ceiling model return CELL_MEANS.get((x["region_name"], x["household"], x["satisfaction"], x["period"]), MEAN) print(f"discrete feature combinations 5*6*5*3 = {5 * 6 * 5 * 3}, " f"filled cells in train {len(cell)}") print(f"cell mean model: train {rmse(cell_prediction, TRAIN):.4f}, " f"test {rmse(cell_prediction, TEST):.4f}") print(f"cells seen in test but absent from train " f"{sum(1 for x in TEST if (x['region_name'], x['household'], x['satisfaction'], x['period']) not in CELL_MEANS)}") ``` ``` discrete feature combinations 5*6*5*3 = 450, filled cells in train 268 cell mean model: train 3.8418, test 6.3887 cells seen in test but absent from train 33 ``` The cell mean model is the most flexible procedure that can be built with these four features: it gives every combination its own train mean and carries no shape assumption. On the train set it gives 3.8418, that is, it comfortably beats the 4.8437 of the degree 8 polynomial. On the test set it gives 6.3887 and comes closer to the baseline's 6.5324 by only 0.1437 m³. This chasm between the two numbers says where the degree sweep is heading. 756 rows fall into 268 filled cells, that is, three subscribers per cell on average; the mean of three subscribers is not a prediction, it is those three subscribers themselves. As the polynomial degree grows, the model approaches this procedure and the test number goes toward its number. The last line shows the same limit from the other direction: of the 252 subscribers in the test set, 33 fall into a cell never seen in the train set, and the only thing that can be said to them is the baseline. #### Summary - Polynomial features do not change the model family; a linear model is linear in its coefficients, not in its columns. Going from degree 1 to 8 the number of columns rises from 7 to 168. - The train root error falls without interruption from 5.0757 to 4.8437 m³ and this is an identity: the columns of the larger degree contain those of the smaller. The train number cannot be used in choosing the degree. - The validation root error bottoms out at 5.2569 and climbs to 5.6877; the weight magnitude rises over the same range from 3.26 to 9,344.45. The mechanism of the climb is the huge coefficients that cancel one another, given to nearly identical columns. - The degree read from validation, 4, gives 5.3775 on test; had it been read from test, degree 5 would have reported 5.2009 and the difference is 0.1766 m³. The same choice stays 0.0679 m³ behind degree 1 on test. - Four features produce 450 discrete combinations and 268 of them are filled in train; the ceiling model that gives every cell its own mean yields 3.8418 on train and 6.3887 on test. The 33 subscribers in the test set fall into a cell never seen in training. #### Next Step The degree sweep turned a knob in whole-number steps: there is no intermediate stop between degree 4 and 5, and moving to degree 5 adds twenty-one columns at once. Between adding columns and deleting them there is a third way — keeping them all and **penalizing** their coefficients. The Introduction to Machine Learning course had built this intuition with a single penalty type and had measured that the two numbers go in opposite directions as the penalty coefficient grows. The next lesson changes the **form** of the penalty: the ridge, lasso and elastic net penalties are applied separately to the same column set and the measure is no longer the error but **sparsity** — how many coefficients each penalty sets exactly to zero, which one shrinks them all and leaves them, and what the zeroed columns are worth on the held-out set. ### Regularization Penalties Source: https://academia.sh/en/courses/supervised-learning/regularization-penalties Applying the ridge, lasso and elastic net penalties to the same 87 columns and measuring them by sparsity: the ridge penalty sets not a single coefficient exactly to zero in any of six settings, the lasso penalty zeroes from 42 up to 84, and the elastic net penalty stays between 24 and 83. At lasso 10, 65 of the 87 coefficients are exactly zero and the remaining 22 columns give 5.2909 m³ on test; the ridge penalty reaches the same number by keeping all 87 columns, at 5.2954. The choice read from validation is ridge 3 and it gives 5.2954 on test; had it been read from test, lasso 10 would have been chosen and the difference would have been 0.0046 m³. At lasso 3000 all 87 coefficients are zeroed and the model returns exactly to the baseline, 6.5324; the ridge penalty still stays at 6.5086 even at 100000. The previous lesson turned a knob in whole-number steps. There was no intermediate stop between degree 4 and 5, and moving to degree 5 added twenty-one columns at once. Besides adding or deleting columns there is a third way: keeping the columns and **penalizing their coefficients**. The Generalization topic of the Introduction to Machine Learning course had built this intuition with a single penalty type, had measured that as the penalty coefficient grows the train number worsens without interruption while the held-out number first improves and then worsens, and had shown that the best point is read from the validation set. That sweep is not repeated here. The question of this lesson is not the **size** of the penalty but its **form**. Penalizing the sum of squares of the coefficients and penalizing the sum of absolute values produce two different procedures, and the difference shows not in the error but in **how many coefficients are exactly zero**. This is the measure: **sparsity**. How many columns each penalty drops from the model, which one shrinks them all and leaves them, and what the dropped columns are worth on the held-out set. - **LM18.** The set, the split, the target and the baseline are as in the previous two lessons: the fictional subscriber table, seed 20260218, 756/252/252, a root error of 6.5324 m³ on the test set. - **LM19.** The column set is the **degree 6 polynomial**: the four region indicators and all the monomials of the three numeric columns whose total degree does not exceed six, 87 columns in all. The degree does not change in this lesson; the only thing that changes is the penalty. - **LM20.** The columns are scaled with the mean and spread learned from the train set. The penalty is sensitive to scale: on unscaled columns the same penalty coefficient is applied to different columns with different force. - **LM21.** The intercept is not penalized. Had it been penalized, the model would say zero when all the coefficients were zeroed; when it is not penalized, it returns to the baseline, that is, to the train mean. - **LM22.** Three penalty forms: the **ridge penalty** penalizes the sum of squares of the coefficients, the **lasso penalty** the sum of absolute values, the **elastic net penalty** an even mixture of the two. No product or figurative name is used; the penalty is referred to by its form. - **LM23.** The solution is **coordinate descent**: each coefficient is updated in turn, with the others held fixed. Because the derivative of the absolute penalty is undefined at zero, there is no closed form; the descent sets a coefficient that stays below the threshold **exactly to zero**. - **LM24.** The number of rounds is 120 and is fixed in all settings. The number of rounds is not swept as a hyperparameter; had it been swept, the number of candidates would have tripled. - **LM25.** The "exactly zero" comparison is not made with `==`; the threshold is $10^{-9}$, and since the descent assigns exactly `0.0` to the coefficient it zeroes, this threshold is only a safeguard. - **LM26.** The number of candidates tried is **18**: three penalty forms, six penalty coefficients. #### Three Penalty Forms The setup is the setup of the previous two lessons. ```python # penalty.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue x = {"region_name": k["region"][0], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "target": round(sum(v) / len(v), 2)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] MEAN = sum(x["target"] for x in TRAIN) / len(TRAIN) def rmse(predict, s): return math.sqrt(sum((predict(x) - x["target"]) ** 2 for x in s) / len(s)) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, " f"baseline on test {rmse(lambda x: MEAN, TEST):.4f} m3") ``` ``` train 756, validation 252, test 252, baseline on test 6.5324 m3 ``` Coordinate descent gathers the three penalty forms in a single procedure. For each coefficient the best value is computed with the others held fixed; the share of the lasso penalty **pushes** this value toward zero, and a coefficient that cannot pass the threshold lands exactly on zero. ```python import itertools NUMERIC = ["household", "satisfaction", "period"] INDICATORS = ["area_south", "area_east", "area_west", "area_central"] DEGREE = 6 POWERS = [u for u in itertools.product(range(DEGREE + 1), repeat=3) if 1 <= sum(u) <= DEGREE] # h household, s satisfaction, p period; h^2s^1 = household squared times satisfaction NAMES = INDICATORS + ["".join(f"{a}^{k}" for a, k in zip("hsp", u) if k) for u in POWERS] def raw(x): return ([float(x[a]) for a in INDICATORS] + [math.prod(x[a] ** k for a, k in zip(NUMERIC, u)) for u in POWERS]) H = [raw(x) for x in TRAIN] P = len(NAMES) O = [sum(r[j] for r in H) / len(H) for j in range(P)] S = [max(1e-9, math.sqrt(sum((r[j] - O[j]) ** 2 for r in H) / len(H))) for j in range(P)] def apply(x): h = raw(x) return [(h[j] - O[j]) / S[j] for j in range(P)] COLS = [[(H[i][j] - O[j]) / S[j] for i in range(len(H))] for j in range(P)] SQ = [sum(v * v for v in COLS[j]) for j in range(P)] Y = [x["target"] for x in TRAIN] def descent(penalty, absolute_share, rounds=120): # MODEL: penalized linear regression w, intercept = [0.0] * P, sum(Y) / len(Y) resid = [y - intercept for y in Y] for _ in range(rounds): for j in range(P): if SQ[j] < 1e-12: continue r = sum(COLS[j][i] * resid[i] for i in range(len(resid))) + SQ[j] * w[j] c1, c2 = penalty * absolute_share, penalty * (1 - absolute_share) if r > c1: updated = (r - c1) / (SQ[j] + c2) elif r < -c1: updated = (r + c1) / (SQ[j] + c2) else: updated = 0.0 delta = updated - w[j] if delta: for i in range(len(resid)): resid[i] -= COLS[j][i] * delta w[j] = updated d = sum(resid) / len(resid) intercept += d for i in range(len(resid)): resid[i] -= d return intercept, w def predictor(m): return lambda x: m[0] + sum(a * b for a, b in zip(m[1], apply(x))) PENALTIES = (3, 10, 30, 100, 300, 1000) FORMS = (("ridge", 0.0), ("lasso", 1.0), ("elastic net", 0.5)) RESULTS = [] print(f"columns {P}, baseline on test {rmse(lambda x: MEAN, TEST):.4f} m3") print(f"\n{'penalty form':<12} {'penalty':>8} {'exact zero':>10} {'train':>9} " f"{'validation':>10} {'test':>8}") for name, share in FORMS: for c in PENALTIES: m = descent(c, share) g = predictor(m) zeros = sum(1 for v in m[1] if abs(v) < 1e-9) RESULTS.append((name, c, zeros, rmse(g, VAL), rmse(g, TEST))) print(f"{name:<12} {c:>8} {zeros:>10} {rmse(g, TRAIN):>9.4f} " f"{rmse(g, VAL):>10.4f} {rmse(g, TEST):>8.4f}") print(f"\ncandidates tried {len(RESULTS)}") ``` ``` columns 87, baseline on test 6.5324 m3 penalty form penalty exact zero train validation test ridge 3 0 5.0348 5.2982 5.2954 ridge 10 0 5.0367 5.3009 5.2968 ridge 30 0 5.0408 5.3088 5.3053 ridge 100 0 5.0581 5.3313 5.3342 ridge 300 0 5.1289 5.4180 5.4387 ridge 1000 0 5.3907 5.6894 5.7325 lasso 3 42 5.0371 5.3028 5.2933 lasso 10 65 5.0429 5.3092 5.2909 lasso 30 74 5.0515 5.3179 5.3077 lasso 100 79 5.0810 5.3486 5.3443 lasso 300 83 5.1447 5.4056 5.4420 lasso 1000 84 5.6036 5.8955 5.9568 elastic net 3 24 5.0360 5.3015 5.2947 elastic net 10 47 5.0405 5.3058 5.2941 elastic net 30 68 5.0477 5.3158 5.3022 elastic net 100 72 5.0722 5.3458 5.3483 elastic net 300 75 5.1431 5.4192 5.4509 elastic net 1000 83 5.4757 5.7450 5.8260 candidates tried 18 ``` The third column is the answer of the lesson, and the difference between the three blocks is beyond dispute. In the rows of the ridge penalty that column is **zero in all six settings**: even when the penalty is raised a thousandfold, not one of the 87 coefficients becomes exactly zero. The lasso penalty rises from 42 to 84 over the same range. The elastic net penalty stands right in the middle: from 24 to 83. The distinction comes from the force a coefficient feels as it approaches zero. The derivative of the ridge penalty is proportional to the coefficient; as the coefficient shrinks, the force trying to shrink it further shrinks too, and it never reaches zero. The derivative of the lasso penalty is independent of magnitude — however small the coefficient, it is pushed with the same constant force — and the moment that force exceeds the coefficient's contribution to lowering the error, the coefficient stays at zero. The last two columns read the same table from the error side, and there the difference is small. The best validation numbers of the three forms are 5.2982, 5.3028 and 5.3015; the distance between them is under 0.005 m³. The choice of penalty form is **not visible in the error number**; it is visible in the number of columns. #### What Sparsity Is Worth One of the eighteen candidates has to be chosen, and the price of sparsity is read right here. ```python best_val = min(RESULTS, key=lambda t: t[3]) best_test = min(RESULTS, key=lambda t: t[4]) print(f"chosen on validation {best_val[0]} penalty {best_val[1]}: " f"{best_val[2]} exact zero, test {best_val[4]:.4f}") print(f"if chosen on test {best_test[0]} penalty {best_test[1]}: " f"{best_test[2]} exact zero, test {best_test[4]:.4f}") print(f"difference between the two choices {best_val[4] - best_test[4]:.4f} m3\n") ridge3, lasso10 = descent(3, 0.0), descent(10, 1.0) print(f"{'threshold':>9} {'ridge penalty 3':>16} {'lasso penalty 10':>16} " f"(coefficients below this absolute value)") for e in (1e-9, 0.001, 0.01, 0.1): print(f"{e:>9} {sum(1 for v in ridge3[1] if abs(v) < e):>16} " f"{sum(1 for v in lasso10[1] if abs(v) < e):>16}") kept = [NAMES[j] for j in range(P) if abs(lasso10[1][j]) > 1e-9] print(f"\nlasso penalty 10 leaves {len(kept)} columns: {', '.join(kept[:9])} ...") for c in (3000, 100000): for name, share in (("lasso", 1.0), ("ridge", 0.0)): mm = descent(c, share) print(f"{name} penalty {c}: exact zero {sum(1 for v in mm[1] if abs(v) < 1e-9)}/{P}, " f"intercept {mm[0]:.4f}, test {rmse(predictor(mm), TEST):.4f}") print(f"train mean {MEAN:.4f}, baseline on test {rmse(lambda x: MEAN, TEST):.4f}") ``` ``` chosen on validation ridge penalty 3: 0 exact zero, test 5.2954 if chosen on test lasso penalty 10: 65 exact zero, test 5.2909 difference between the two choices 0.0046 m3 threshold ridge penalty 3 lasso penalty 10 (coefficients below this absolute value) 1e-09 0 65 0.001 0 65 0.01 5 66 0.1 29 71 lasso penalty 10 leaves 22 columns: area_south, area_east, area_west, area_central, p^1, s^1p^4, s^1p^5, s^3p^1, h^1 ... lasso penalty 3000: exact zero 87/87, intercept 19.7665, test 6.5324 ridge penalty 3000: exact zero 0/87, intercept 19.7665, test 6.0935 lasso penalty 100000: exact zero 87/87, intercept 19.7665, test 6.5324 ridge penalty 100000: exact zero 0/87, intercept 19.7665, test 6.5086 train mean 19.7665, baseline on test 6.5324 ``` The validation set chooses the ridge penalty, and that choice keeps all 87 coefficients. Had it been chosen from the test set, lasso 10 would have come: 65 coefficients exactly zero, the remaining 22 columns and 5.2909. The difference between the two choices is 0.0046 m³, and it is the price of searching eighteen times for the smallest on a 252-row set. But this is not the difference the table is really telling. **The same error level is reached with two different column counts**: 5.2954 with 87 columns for the ridge penalty, 5.2909 with 22 columns for the lasso penalty. What sparsity is worth on the held-out set is **zero** on this set — 65 columns leave the model and the error number does not worsen. Sparsity is not a performance gain but a **cost gain**: the number of columns to be computed, stored and explained for a prediction falls to a quarter. The threshold table gives the difference between the two penalties more sharply. Not one of the ridge penalty's 87 coefficients is below $10^{-9}$, but 29 of them are below 0.1. That is, the ridge penalty also disables columns — it only **keeps them in the model**. The lasso penalty really drops 65 of them and rises only to 71 at the 0.1 threshold; it does not shrink the ones outside those it dropped. The last four lines show the limit of the two forms. At lasso 3000 all 87 coefficients are zeroed, the intercept stays at the train mean, and the model gives **exactly 6.5324** on test, that is, the baseline itself. The ridge penalty is at 6.0935 at the same setting, and at 6.5086 at a coefficient of a hundred thousand. The ridge penalty approaches the baseline but at a finite setting does **not reach** it; the lasso penalty does. The difference between the two forms shows not at any point of the error curve but at its ends. #### Summary - The form of the penalty shows not in the error number but in the number of columns: the best validation numbers of the three forms, 5.2982, 5.3028 and 5.3015, are within 0.005 m³ of one another. - The ridge penalty does not set a single coefficient exactly to zero in any of the six settings; the lasso penalty rises from 42 to 84, the elastic net penalty from 24 to 83. The reason is the shape of the derivative: the force of the ridge penalty shrinks as the coefficient shrinks, that of the lasso penalty stays constant. - The same error level is reached with two different column counts: 5.2954 with 87 columns for the ridge penalty, 5.2909 with 22 columns for the lasso penalty. What sparsity is worth on this set is a cost, not performance. - The ridge penalty disables columns but does not drop them: 29 of its 87 coefficients are below 0.1 and none is exactly zero. - At lasso 3000 all the coefficients are zeroed and the model returns exactly to the baseline, 6.5324; the ridge penalty stays at 6.5086 even at a coefficient of a hundred thousand and does not reach the baseline at a finite setting. - The choice read from validation is ridge 3 (5.2954 on test); had it been read from test, lasso 10 would have been chosen (5.2909) and the price of the eighteen-candidate budget would have been 0.0046 m³. #### Next Step In three lessons the target was a continuous number and the model said that number directly. The real through-line problem of the course, though, is a classification: whether a subscriber carries illicit use or fault suspicion, and the baseline there is 0.7579. The same linear model cannot be used directly on this problem, because the output of the plane stretches from minus infinity to plus infinity and the label is only zero and one. The bridge between them is squeezing the output into a probability. The next lesson writes this squeeze, and its measure is this: the same model gives one number at a 0.5 threshold and other numbers at other thresholds; choosing the threshold is a separate job and is left to the `evaluation-and-tuning` topic. What will be shown here is that the output is continuous **before the threshold** and that this continuity carries more than a single class label. ### Logistic Regression Source: https://academia.sh/en/courses/supervised-learning/logistic-regression Squeezing the output of a linear plane into a probability and measuring that the probability is continuous before the threshold: with twelve fixed columns the model gives 0.7937 on the test set and passes the baseline by 0.0357, that is, nine subscribers. When nine candidates are tried, five tie at 0.8373 on validation and the test numbers of those five split between 0.7937 and 0.8016; the setting read from validation gives 0.7937 on test, and the difference is two subscribers. All 252 of the 252 subscribers in the test set receive a different probability; when the threshold is swept from 0.20 to 0.80, a single model gives seven different accuracy numbers between 0.7183 and 0.8016, the number of flagged subscribers falls from 102 to 5, and 27 subscribers change class between 0.40 and 0.60. In three lessons the target was a continuous number and the model said that number directly. The through-line problem of the course, though, is a classification: whether a subscriber carries illicit use or fault suspicion, and the baseline there is 0.7579. The same linear plane cannot be used directly on this problem. The output of the plane stretches from minus infinity to plus infinity; the label is only zero and one. Saying "-3.4 suspicious" of a subscriber is meaningless. The bridge is an S-shaped function that squeezes the output of the plane between zero and one. The squeezed number is read as a **probability**, and in this form the model takes the name **logistic regression** — it is a regression procedure, because it predicts a continuous number; it becomes a classifier only after a **threshold** is applied to that number. The measure of this lesson lies exactly here: what the output carries before the threshold is applied. Choosing the threshold is a separate job and is left to the `evaluation-and-tuning` topic. - **LM27.** The set, the split and the seed are as in the previous three lessons. The target is the **suspicious** label and the baseline is saying the most frequent class: 0.7579 on the test set. - **LM28.** The feature set is the **twelve columns** left by the Data Preparation and Feature Engineering course and stays fixed throughout the course: mean consumption, volatility, household size, satisfaction, period count, zero-reading indicator, per-person consumption and five region indicators. The columns that the regression lessons left out are used here, because the target is no longer consumption but a label. - **LM29.** The columns are scaled with the mean and spread learned from the train set. The Preparation Steps topic of the Data Preparation and Feature Engineering course had measured that this step is mandatory in a gradient-based model; here it is assumed. - **LM30.** All five region indicators are used. In the regression lessons one had been left out as the reference; since the gradient-based solution does not solve a singular system, it is not needed here. - **LM31.** The solution is **gradient descent**: the weights are updated step by step in the direction of the slope of the loss surface. The hyperparameters are the learning rate and the number of epochs. - **LM32.** The number of candidates tried is **9**: three learning rates, three epoch counts. - **LM33.** The measure is accuracy and the test set is 252 subscribers; one subscriber is worth 0.0040. What accuracy hides and by which criterion it should be replaced are left to `evaluation-and-tuning/01`–`02`. - **LM34.** The threshold sweep in this lesson **does not make a choice**; it is printed to show that a single model is independent of the threshold. Choosing the operating point by a cost calculation is the subject of `evaluation-and-tuning/03`. - **LM35.** Whether the stated probability sits on the realized rate is a separate question and is left to `evaluation-and-tuning/04`. #### From the Plane to a Probability The setup generates the set, fixes the twelve columns and prints two rates. ```python # logistic.py — MODEL: the same FICTIONAL subscriber table, the same seed and the same split; # in this lesson the target is the illicit use or fault suspicion label. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEANS = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD_DEVS = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def scaled(x): # FIT from train, APPLY to every set return [1.0] + [(x[a] - MEANS[a]) / STD_DEVS[a] for a in FIELDS] print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, columns {len(FIELDS)}") print(f"suspicious rate on train {sum(x['suspicious'] for x in TRAIN) / len(TRAIN):.4f}, " f"on test {sum(x['suspicious'] for x in TEST) / len(TEST):.4f}") print(f"baseline (most frequent class) on test {BASELINE:.4f}") ``` ``` train 756, validation 252, test 252, columns 12 suspicious rate on train 0.2222, on test 0.2421 baseline (most frequent class) on test 0.7579 ``` The squeeze and gradient descent stand together. The squeezed number is the same thing as reading the linear score as the **log-odds**: when the score is zero the probability is 0.5, as the score grows it goes toward one, as it shrinks toward zero, and it never touches the ends. ```python def squash(z): # S-shaped function, overflow-safe if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) X = [scaled(x) for x in TRAIN] Y = [x["suspicious"] for x in TRAIN] P = len(X[0]) def logistic(rate, epochs): # MODEL: logistic regression w = [0.0] * P for _ in range(epochs): g = [0.0] * P for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(P))) - Y[i] for j in range(P): g[j] += h * X[i][j] for j in range(P): w[j] -= rate * g[j] / len(X) return w def probability(w, x): return squash(sum(a * b for a, b in zip(w, scaled(x)))) def accuracy(w, s, threshold=0.5): return sum((probability(w, x) > threshold) == x["suspicious"] for x in s) / len(s) RESULTS = [] print(f"{'learning rate':>13} {'epochs':>6} {'train':>9} {'validation':>10} {'test':>8}") for rate in (0.1, 0.3, 1.0): for epochs in (100, 300, 600): w = logistic(rate, epochs) RESULTS.append((rate, epochs, accuracy(w, VAL), accuracy(w, TEST), w)) print(f"{rate:>13} {epochs:>6} {accuracy(w, TRAIN):>9.4f} " f"{RESULTS[-1][2]:>10.4f} {RESULTS[-1][3]:>8.4f}") best_val = max(RESULTS, key=lambda t: t[2]) best_test = max(RESULTS, key=lambda t: t[3]) print(f"\nbaseline {BASELINE:.4f}, candidates tried {len(RESULTS)}") print(f"setting chosen on validation ({best_val[0]}, {best_val[1]}): validation {best_val[2]:.4f}, " f"test {best_val[3]:.4f}") print(f"setting if chosen on test ({best_test[0]}, {best_test[1]}): validation {best_test[2]:.4f}, " f"test {best_test[3]:.4f}") print(f"difference between the two choices {best_test[3] - best_val[3]:.4f}, that is " f"{round((best_test[3] - best_val[3]) * len(TEST))} subscribers") print(f"candidates with the same highest number on validation " f"{sum(1 for t in RESULTS if abs(t[2] - best_val[2]) < 1e-12)}") W = best_val[4] ``` ``` learning rate epochs train validation test 0.1 100 0.8175 0.8056 0.8016 0.1 300 0.8201 0.8373 0.7937 0.1 600 0.8188 0.8373 0.8016 0.3 100 0.8201 0.8373 0.7937 0.3 300 0.8175 0.8333 0.8016 0.3 600 0.8148 0.8333 0.8016 1.0 100 0.8161 0.8333 0.8016 1.0 300 0.8148 0.8373 0.8016 1.0 600 0.8148 0.8373 0.8016 baseline 0.7579, candidates tried 9 setting chosen on validation (0.1, 300): validation 0.8373, test 0.7937 setting if chosen on test (0.1, 100): validation 0.8056, test 0.8016 difference between the two choices 0.0079, that is 2 subscribers candidates with the same highest number on validation 5 ``` The triple is complete: baseline 0.7579, the family's number 0.7937, candidates tried 9. The difference is 0.0357, that is, nine subscribers on the test set. The test column of the nine candidates takes only two distinct values — 0.7937 and 0.8016 — and the gap between them is two subscribers. The learning rate and the epoch count substitute for each other in this problem: six hundred epochs with a small rate and three hundred with a large one arrive at the same place. The last line shows the rule of the course from an unexpected place. **Five** of the nine candidates give exactly the same number on the validation set, 0.8373. The validation set is 252 rows and measures in steps of 0.0040; it is ordinary for five different settings to fall on the same digit. But those five candidates do not give the same number on the test set. The rule that breaks the tie — taking the first place in the table — is not a property of the model, and it is worth two subscribers. #### The Output Is Continuous Before the Threshold The chosen model is a single setting, but it does not give a single number. What it gives is 252 probabilities, and extracting a class from those probabilities is a separate decision. ```python tied = [t for t in RESULTS if abs(t[2] - best_val[2]) < 1e-12] print(f"the {len(tied)} candidates tied on validation, test numbers: " f"{sorted({round(t[3], 4) for t in tied})}") p = sorted((probability(W, x), x["suspicious"]) for x in TEST) print(f"distinct probability values on test {len({round(q, 9) for q, _ in p})}/{len(TEST)}, " f"lowest {p[0][0]:.4f}, highest {p[-1][0]:.4f}\n") print(f"{'probability range':<18} {'subscribers':>11} {'suspicious':>10} {'suspicious rate':>15}") for k in range(5): a, b = k / 5, (k + 1) / 5 d = [q for q in p if a <= q[0] < b or (k == 4 and q[0] >= b)] g = sum(t[1] for t in d) print(f"[{a:.1f}, {b:.1f})".ljust(18) + f" {len(d):>11} {g:>10} {g / len(d) if d else 0.0:>15.4f}") print(f"\n{'threshold':>9} {'accuracy':>9} {'flagged':>12} {'caught':>10} " f"{'missed':>10}") for e in (0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80): flagged = sum(1 for q, _ in p if q > e) caught = sum(1 for q, y in p if q > e and y) print(f"{e:>9.2f} {accuracy(W, TEST, e):>9.4f} {flagged:>12} {caught:>10} " f"{sum(t[1] for t in p) - caught:>10}") print(f"\nsubscribers who change class between 0.40 and 0.60 " f"{sum(1 for q, _ in p if 0.40 < q <= 0.60)}") print(f"threshold fixed at 0.5: accuracy {accuracy(W, TEST):.4f}, " f"baseline {BASELINE:.4f}, best threshold at 0.60: {accuracy(W, TEST, 0.60):.4f}") ``` ``` the 5 candidates tied on validation, test numbers: [0.7937, 0.8016] distinct probability values on test 252/252, lowest 0.0196, highest 0.9838 probability range subscribers suspicious suspicious rate [0.0, 0.2) 150 15 0.1000 [0.2, 0.4) 52 18 0.3462 [0.4, 0.6) 27 11 0.4074 [0.6, 0.8) 18 12 0.6667 [0.8, 1.0) 5 5 1.0000 threshold accuracy flagged caught missed 0.20 0.7183 102 46 15 0.30 0.7659 68 35 26 0.40 0.7817 50 28 33 0.50 0.7937 31 20 41 0.60 0.8016 23 17 44 0.70 0.7897 16 12 49 0.80 0.7778 5 5 56 subscribers who change class between 0.40 and 0.60 27 threshold fixed at 0.5: accuracy 0.7937, baseline 0.7579, best threshold at 0.60: 0.8016 ``` The second line is the plainest finding of the lesson: **all 252 of the 252 subscribers receive a different probability**. The model does not produce a label, it produces a ranking. The lowest probability is 0.0196 and the highest 0.9838; the squeeze does not touch the ends, because an S-shaped function reaches zero and one only at infinity. The range table shows that the ranking is not empty. Of the 150 subscribers in the lowest range, 15, that is one in ten, are suspicious; all five subscribers in the highest range are suspicious. The ranking goes in the same direction as the label. Whether the stated probability **sits on** the realized rate — for example, the rate coming out as 0.4074 in the 0.4–0.6 range — is a separate question and is left to `evaluation-and-tuning/04`. The threshold table gives the number that carries the title of the lesson. A single model, a single weight set, a single training — and seven different accuracy numbers: from 0.7183 to 0.8016. At a threshold of 0.20 the model flags 102 subscribers and catches 46 of the 61 suspicious ones; at a threshold of 0.80 only five subscribers are flagged and all five are truly suspicious, but 56 suspicious ones escape. None of these is a different model. For this reason the sentence "the accuracy of the model is 0.7937" is incomplete. The correct sentence is: **it is 0.7937 at a 0.5 threshold.** The threshold is not a property of the model but a decision, and if the decision is moved between 0.40 and 0.60, 27 subscribers change class. Which threshold to choose depends on the cost of the illicit use caught against the cost of the wasted field visit; that calculation is the subject of `evaluation-and-tuning/03` and is not done here. #### Summary - Logistic regression is not a classifier but a regression procedure that predicts a probability; it becomes a classifier only after a threshold is applied. - With twelve fixed columns the model gives 0.7937 on the test set; the baseline is 0.7579, the difference 0.0357 and the number of candidates tried 9. - Five of the nine candidates tie at exactly 0.8373 on the validation set but split on test between 0.7937 and 0.8016; the rule that breaks the tie is not a property of the model and is worth two subscribers. - All 252 of the 252 subscribers in the test set receive a different probability, the lowest 0.0196, the highest 0.9838. The model produces a ranking, not a label, and the ranking goes in the same direction as the label. - When the threshold is swept from 0.20 to 0.80, the same model gives seven different accuracies between 0.7183 and 0.8016; the number of flagged subscribers falls from 102 to 5 and 27 subscribers change class between 0.40 and 0.60. #### Next Step The model of this lesson carries twelve coefficients, and those coefficients are readable: each belongs to a column and has a sign and a magnitude. This is the most often cited advantage of linear models. The next lesson treats that advantage not as praise but as **a number**: how many subscribers is the difference between the held-out number of a model with readable coefficients and the number of the best family in this topic, and what can and cannot be said by looking at a coefficient. The close of the topic is there too. ### Coefficient Interpretation Source: https://academia.sh/en/courses/supervised-learning/coefficient-interpretation Treating readability as a number rather than a praise: the readable model chosen from eight candidates on the validation set carries twelve coefficients and gives 0.8016 on the test set, the best family chosen from the same set gives 0.8056 with 33 columns, and the price of interpretability is 0.0040, that is, a single subscriber. The twelve coefficients give the order of the generating structure in the fiction correctly. The same coefficients are not fixed: when the per-person consumption column is removed, the coefficient of mean consumption rises from 0.772 to 0.808 and the test number does not change at all; when mean consumption is removed, the coefficient of per-person consumption jumps from 0.071 to 0.442, that is, to six times. The ridge penalty brings the coefficients down from 0.772 to 0.326 while the test number falls by only 0.0079. The model of the previous lesson carried twelve coefficients, and those coefficients are readable: each belongs to a column and has a sign and a magnitude. This is the most frequently cited advantage of linear models, and it is usually written down as a praise. This lesson treats it not as a praise but as **a number**: how many subscribers separate the held-out number of a model with readable coefficients from the number of the best family in this topic. The second question is separate from the first and harder. What has been said when a coefficient is read. The Applied Statistics topic of the Statistical Analysis and Exploratory Data Analysis course had measured the **causal** face of this question: a one-variable coefficient changing sign when a confounding variable is added, and the within-group slopes turning against the aggregated slope, were shown there with numbers. It is not repeated here. The question here is specific to the prediction target: **how many different coefficient sets give the same held-out number**, and how much the value of a coefficient depends on which of those sets was chosen. - **LM36.** The set, the split, the seed, the target and the fixed twelve-column feature set are as in the previous lesson; the baseline is 0.7579 on the test set. - **LM37.** All candidates are learned with **the same training setting**: learning rate 0.3, 300 epochs. The tuning sweep of the previous lesson is not repeated; the only thing that changes in this lesson is **the column set**. - **LM38.** A readable candidate is a model whose coefficients can be listed one by one and interpreted: the candidates with one, two, four and twelve columns. - **LM39.** The expanded candidate is the model raised to 33 columns with the pairwise products and the squares of the six numeric columns. Its coefficients can be listed but not read one by one, because the same magnitude is spread across several columns. - **LM40.** The ridge penalty is applied to the expanded candidate at four levels. The difference between the penalty forms was measured in the previous lessons; only the ridge penalty is used here. - **LM41.** The number of candidates tried is **8**: four readable candidates and four expanded candidates with a penalty. The choice is read from the validation set. - **LM42.** The coefficients are read on **scaled columns**, so they are comparable with one another. A coefficient in raw units is the change per cubic meter and cannot be ranked, because the columns have different units. - **LM43.** This lesson does not deal with **for whom** a coefficient is read, what happens in which subgroup, or how interpretability is built as a method; those questions belong to the Model Evaluation, Interpretability and Ethics course. Here interpretability is only the score paid. #### The Price of Interpretability The setup is the setup of the previous lesson. ```python # interpret.py — MODEL: the same FICTIONAL subscriber table, the same seed, the same split and # the same twelve-column feature set; the target is the suspicious label. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, " f"baseline {BASELINE:.4f}") ``` ``` train 756, validation 252, test 252, baseline 0.7579 ``` The eight candidates are ranked in a single ladder: readable models from one column to twelve columns, then the expanded model with 33 columns at four penalty levels. ```python import itertools def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) PRODUCTS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "per_person"] def expand(x): d = dict(x) for a, b in itertools.combinations(PRODUCTS, 2): d[a + "*" + b] = x[a] * x[b] for a in PRODUCTS: d[a + "^2"] = x[a] * x[a] return d EXPANDED = FIELDS + [a + "*" + b for a, b in itertools.combinations(PRODUCTS, 2)] \ + [a + "^2" for a in PRODUCTS] EX_TRAIN, EX_VAL, EX_TEST = [[expand(x) for x in s] for s in (TRAIN, VAL, TEST)] def scale(train, cols): return ({a: statistics.fmean(x[a] for x in train) for a in cols}, {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in cols}) def train_model(train, cols, mu, sd, rate=0.3, epochs=300, penalty=0.0): # MODEL: logistic regression X = [[1.0] + [(x[a] - mu[a]) / sd[a] for a in cols] for x in train] Y = [x["suspicious"] for x in train] w = [0.0] * len(X[0]) for _ in range(epochs): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= rate * (g[j] / len(X) + (penalty * w[j] if j else 0.0)) return w def score(w, cols, mu, sd, x): return w[0] + sum(w[j + 1] * (x[a] - mu[a]) / sd[a] for j, a in enumerate(cols)) def accuracy(w, cols, mu, sd, rows): return sum((squash(score(w, cols, mu, sd, x)) > 0.5) == x["suspicious"] for x in rows) / len(rows) LADDER = [("1 column", ["mean_consumption"]), ("2 columns", ["mean_consumption", "zero_readings"]), ("4 columns", ["mean_consumption", "zero_readings", "household", "satisfaction"]), ("12 columns", FIELDS)] RESULTS, tried = [], 0 print(f"{'model':<24} {'coefficients':>12} {'validation':>10} {'test':>8}") for name, cols in LADDER: mu, sd = scale(TRAIN, cols) w = train_model(TRAIN, cols, mu, sd) tried += 1 RESULTS.append((name, len(cols), accuracy(w, cols, mu, sd, VAL), accuracy(w, cols, mu, sd, TEST))) print(f"{name:<24} {len(cols):>12} {RESULTS[-1][2]:>10.4f} {RESULTS[-1][3]:>8.4f}") emu, esd = scale(EX_TRAIN, EXPANDED) for penalty in (0.0, 0.01, 0.05, 0.2): w = train_model(EX_TRAIN, EXPANDED, emu, esd, penalty=penalty) tried += 1 RESULTS.append((f"33 columns penalty {penalty}", len(EXPANDED), accuracy(w, EXPANDED, emu, esd, EX_VAL), accuracy(w, EXPANDED, emu, esd, EX_TEST))) print(f"{RESULTS[-1][0]:<24} {len(EXPANDED):>12} {RESULTS[-1][2]:>10.4f} " f"{RESULTS[-1][3]:>8.4f}") print(f"\nbaseline {BASELINE:.4f}, candidates tried {tried}") readable = max(RESULTS[:4], key=lambda t: t[2]) overall = max(RESULTS, key=lambda t: t[2]) print(f"readable candidate chosen from validation: {readable[0]}, test {readable[3]:.4f}") print(f"all candidates, chosen from validation: {overall[0]}, test {overall[3]:.4f}") print(f"price of interpretability {overall[3] - readable[3]:.4f}, that is " f"{round((overall[3] - readable[3]) * len(TEST))} subscriber") ``` ``` model coefficients validation test 1 column 1 0.8254 0.7976 2 columns 2 0.8095 0.7976 4 columns 4 0.8095 0.7897 12 columns 12 0.8333 0.8016 33 columns penalty 0.0 33 0.8333 0.8056 33 columns penalty 0.01 33 0.8373 0.7976 33 columns penalty 0.05 33 0.8492 0.8056 33 columns penalty 0.2 33 0.8214 0.7937 baseline 0.7579, candidates tried 8 readable candidate chosen from validation: 12 columns, test 0.8016 all candidates, chosen from validation: 33 columns penalty 0.05, test 0.8056 price of interpretability 0.0040, that is 1 subscriber ``` The last line is the number that carries the title of the lesson. Choosing among the readable models by looking at the validation set gives the twelve-column model, and 0.8016 is read on the test set. Choosing among all candidates by looking at the same set gives the expanded model with a penalty, and 0.8056 is read on the test set. The price of interpretability is 0.0040, that is, **one in 252 subscribers**. The smallness of this number is specific to this set and is not a general law; but what has to be read is not its size but **that it was measured**. When it is said that performance was given up for the sake of interpretability and the amount given up is not written, that sentence is not a measurement but a declaration of preference. Here the amount was written and its budget was put next to it: eight candidates. The ladder itself says something separate. The one-column model beats the four-column model on validation with 0.8254, and on the test set both stay below the twelve-column one. Adding columns is not a one-way improvement, and this was the finding of the twelve lessons of the Data Preparation and Feature Engineering course. #### What a Coefficient Says and Does Not Say The twelve coefficients of the chosen readable model are read on scaled columns, and in this form they are comparable with one another. ```python mu, sd = scale(TRAIN, FIELDS) W = train_model(TRAIN, FIELDS, mu, sd) print(f"{'column':<16} {'coefficient':>11} (scaled columns, intercept " f"{W[0]:+.3f})") for j in sorted(range(len(FIELDS)), key=lambda j: -abs(W[j + 1])): print(f"{FIELDS[j]:<16} {W[j + 1]:>+11.3f}") print() TWIN = [("full 12 columns", FIELDS), ("per_person removed", [a for a in FIELDS if a != "per_person"]), ("mean_consumption removed", [a for a in FIELDS if a != "mean_consumption"])] print(f"{'model':<24} {'mean_consumption':>16} {'per_person':>10} {'test':>8}") for name, cols in TWIN: mu2, sd2 = scale(TRAIN, cols) w = train_model(TRAIN, cols, mu2, sd2) k = {a: w[j + 1] for j, a in enumerate(cols)} print(f"{name:<24} {k.get('mean_consumption', 0.0):>+16.3f} " f"{k.get('per_person', 0.0):>+10.3f} {accuracy(w, cols, mu2, sd2, TEST):>8.4f}") print() print(f"{'ridge penalty':>13} {'mean_consumption':>16} {'volatility':>10} " f"{'zero_readings':>13} {'test':>8}") for penalty in (0.0, 0.02, 0.1, 0.5): w = train_model(TRAIN, FIELDS, mu, sd, penalty=penalty) k = {a: w[j + 1] for j, a in enumerate(FIELDS)} print(f"{penalty:>13} {k['mean_consumption']:>+16.3f} {k['volatility']:>+10.3f} " f"{k['zero_readings']:>+13.3f} {accuracy(w, FIELDS, mu, sd, TEST):>8.4f}") ``` ``` column coefficient (scaled columns, intercept -1.643) mean_consumption +0.772 zero_readings +0.684 volatility +0.380 area_east +0.306 satisfaction -0.273 household +0.156 period +0.148 area_west -0.147 area_south -0.109 per_person +0.071 area_central -0.048 area_north -0.005 model mean_consumption per_person test full 12 columns +0.772 +0.071 0.8016 per_person removed +0.808 +0.000 0.8016 mean_consumption removed +0.000 +0.442 0.7778 ridge penalty mean_consumption volatility zero_readings test 0.0 +0.772 +0.380 +0.684 0.8016 0.02 +0.573 +0.397 +0.523 0.7937 0.1 +0.326 +0.342 +0.310 0.7937 0.5 +0.133 +0.178 +0.127 0.7619 ``` The first table shows what the coefficient says, and what it says is true. Mean consumption, the zero-reading indicator and volatility are in the first three places; these are exactly the three heaviest terms that the fiction used when generating the label. The sign of satisfaction is minus, that of household size is plus, and among the regions east is the highest. The coefficient of north sits at zero with -0.005; in the fiction the region contribution of north is also exactly zero. The twelve numbers give the order of the generating structure in a readable form. The second table shows **what the same coefficients do not say**. In the twelve-column model per-person consumption stands near the end of the list with +0.071; someone who looks at this number concludes that the column is almost useless. When the column is removed from the model, the test number **does not change at all**, it stays at 0.8016; up to here the reading is correct. But when mean consumption is removed, the coefficient of the same column jumps from +0.071 to +0.442, that is, to six times. Per-person consumption was not a useless column; it **looked useless while mean consumption was next to it**, because the two are two forms of the same quantity, and the model divides the weight belonging to that quantity between the two as it pleases. The third table measures the same uncertainty through the penalty. The coefficient of mean consumption falls from 0.772 to 0.326, that is, to below half; the zero-reading indicator falls from 0.684 to 0.310. The test number, on the other hand, falls from 0.8016 to 0.7937, that is, by only two subscribers. **These three models are almost the same as predictions, and their coefficients hardly resemble one another.** The held-out number is not enough to determine which coefficient set was chosen. The rule that comes out of this is limited and has to be written exactly. The **place of a coefficient in the ranking** is readable information to the extent that the columns are independent of one another. The **value** of a coefficient, on the other hand, is meaningful only together with the set of columns beside it and the penalty applied. The **causal** reading of a coefficient does not come out of any of these tables; the Applied Statistics topic had measured that limit separately. #### Summary - The price of interpretability in this set is 0.0040, that is, one in 252 subscribers: the readable model chosen from validation gives 0.8016 on the test set, the best family chosen from the same set gives 0.8056, and the number of candidates tried is 8. - The ladder is not one-way: the one-column model passes the four-column one on validation, and on the test set both stay below the twelve-column one. - The twelve scaled coefficients give the order of the generating structure in the fiction correctly; the first three places are mean consumption, zero readings and volatility, and the coefficient of north sits at zero with -0.005. - The value of a coefficient depends on the columns beside it: per-person consumption is +0.071, and it jumps to +0.442 when mean consumption is removed, that is, it rises to six times. - The ridge penalty brings the coefficients of mean consumption and zero readings down to below half and hardly changes volatility, while the test number falls by only 0.0079; the coefficients of models that are almost the same as predictions hardly resemble one another. #### Next Step Across five lessons the model family changed but the **form** of the model never did. Least squares was a plane; polynomial regression bent the columns, but the decision was still a linear combination of the columns; the three penalties only shrank the coefficients of that combination; logistic regression squeezed the output of the plane, but the squeeze is a one-way increasing function, so the decision boundary is still a plane. In all five lessons the subscriber being classified fell on one side or the other according to a single weighted sum of twelve numbers. This is a limit, and it has never been tested. In a set where the boundary itself is **curved** — a set where suspicion rises in a certain range of two columns and falls on both sides of the range — no plane works, and polynomial columns catch that curve only if we know its shape in advance. The next topic opens with families that do not build the decision with a plane: an instance-based procedure that decides by looking at its neighbors, and what happens to the distance measure itself as the number of columns grows. ## Nonlinear Models Counting the difference and the cost that the families which come into play when the decision boundary is curved add: the number that the neighbor count and the distance measure move and the nearest and farthest neighbors drawing closer as columns increase, the probability output running to the extremes when the conditional independence assumption breaks while the ranking survives, the margin width and the number of support vectors that carry the decision and the result the kernel transformation gives without being expanded, the split a splitting criterion chooses and how pruning differs from a depth limit, the number that settles and the variability that falls as the number of trees grows together with the out-of-bag estimate, the trade-off between learning rate and number of rounds in sequential boosting, the difference three combinations add and the number of examples on which the ensembles err together. ### Nearest Neighbor Source: https://academia.sh/en/courses/supervised-learning/nearest-neighbor Measuring a family that computes nothing and stores the training set as it is: twenty-four candidates are tried over the neighbor count and the distance metric, the candidate chosen on the validation set gives 0.7937 on the test set and beats the baseline by 0.0357, yet the same candidate scores 0.8294 on the set it was chosen on. A procedure that looks at one neighbor falls below the baseline on all three metrics. As the column count grows, the ratio of the farthest neighbor to the nearest drops from 26.91 to 1.49 and the model settles exactly on the baseline. The Linear Models topic worked on a single form across five lessons: the model drew a **line** or a **plane** in the feature space. Polynomial features stretched this form, but the stretching still stayed inside an equation with fixed coefficients. The closing of the topic left a question open: what happens in a set where the boundary separating two classes is itself **curved**, and how many numbers does a family that finds that curve without writing it by hand cost. This lesson starts with the answer that is furthest from the question: a model that builds no equation at all. An **instance-based procedure** computes nothing during training; it stores the training set as it is, and when classifying a subscriber it looks at the training rows **nearest** to that subscriber and gives the label of the majority. The decision boundary is not written here, it arises from the placement of the examples, and nothing extra is done to make it curved. It has a price, and the lesson measures that price with the course's three numbers: the baseline, the family number and the number of candidates tried to find that number. - **NM1.** The set is the **fictional** subscriber table coming from the Introduction to Machine Learning and Data Preparation and Feature Engineering courses: 1,260 subscribers, seed 20260218, split 756/252/252, baseline 0.7579 on the test set. The baseline is the procedure that says the most frequent class, that is, "not suspicious". - **NM2.** The feature set is taken as the Data Preparation and Feature Engineering course left it and is **held fixed throughout the course**: five numeric columns, the per-person consumption ratio, the zero-reading indicator and five region columns, twelve in total. Columns derived from text are left out because they are left to the Natural Language Processing and Computer Vision course. The only thing that changes is the model. - **NM3.** The measure is accuracy and the test set is 252 subscribers; one subscriber is worth 0.0040. - **NM4.** Scaling is applied in the form set up in the scaling lesson of the Data Preparation and Feature Engineering course: the standard score is learned from the training set alone and applied to the other two sets. In a procedure that sums distances, scale is a column weight; that course measured that the sign of this decision changes with the family, and those numbers are not repeated here. - **NM5.** A **hyperparameter** is a number that the model does not learn from the data and that is given from outside. This family has two: the neighbor count and the distance metric. - **NM6.** The choice is read from the **validation set**. The number on the test set is only reported; what would come out if the choice were made on the test set from the same table is printed separately. - **NM7.** Neighbor counts are chosen odd so that no tie arises in the vote. - **NM8.** The columns added in the curse of dimensionality measurement are **fictional**, come from the same generator and carry no relation to the label. #### A Model Without Training The setup reproduces the set, learns the scale from the training set and writes the procedure. ```python # neighbor.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the feature set comes from the Data Preparation and Feature Engineering course and is fixed throughout the course. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "region_name": k["region"][0], "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) x["per_person"] = round(x["mean_consumption"] / x["household"], 3) DATA.append(x) NUMERIC = ["mean_consumption", "volatility", "household", "satisfaction", "period", "per_person"] FIELDS = NUMERIC + ["zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] BASELINE_TEST = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) BASELINE_VAL = sum(x["suspicious"] == 0 for x in VAL) / len(VAL) SCALE = {a: (statistics.fmean(x[a] for x in TRAIN), statistics.pstdev([x[a] for x in TRAIN])) for a in NUMERIC} def scale_rows(s): return [dict(x, **{a: (x[a] - SCALE[a][0]) / SCALE[a][1] for a in SCALE}) for x in s] def vectorize(s, fields=None): return [([x[a] for a in (fields or FIELDS)], x["suspicious"]) for x in s] def abs_diff(a, b): return sum(abs(p - q) for p, q in zip(a, b)) def neighbor_accuracy(train, target, k, metric): # MODEL: instance-based procedure hits = 0 for v, y in target: u = sorted((metric(v, w), t) for w, t in train) hits += (sum(t for _, t in u[:k]) * 2 > k) == y return hits / len(target) S_TRAIN, S_VAL, S_TEST = scale_rows(TRAIN), scale_rows(VAL), scale_rows(TEST) V_TRAIN, V_VAL, V_TEST = vectorize(S_TRAIN), vectorize(S_VAL), vectorize(S_TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, columns {len(FIELDS)}") print(f"baseline: test {BASELINE_TEST:.4f}, validation {BASELINE_VAL:.4f}") print(f"unscaled columns (absolute difference, k=31) test " f"{neighbor_accuracy(vectorize(TRAIN), vectorize(TEST), 31, abs_diff):.4f}") print(f"scaled columns (absolute difference, k=31) test " f"{neighbor_accuracy(V_TRAIN, V_TEST, 31, abs_diff):.4f}") ``` ``` train 756, validation 252, test 252, columns 12 baseline: test 0.7579, validation 0.8135 unscaled columns (absolute difference, k=31) test 0.7778 scaled columns (absolute difference, k=31) test 0.7937 ``` The body of the `neighbor_accuracy` function has no model-building step. In the other families there is a stage that produces weights, coefficients or split thresholds; here that stage is empty and the model itself is **the 756-row training set**. One consequence shows on the prediction side: for every new subscriber, 756 distances are computed and sorted. The family hands the cost of training over to prediction. The second observation is in the scaling lines. The same procedure reads 0.7778 with unscaled columns and 0.7937 with scaled columns. The reason is that distance is a **sum**: in the sum of twelve differences, the raw range of mean consumption is forty times wider than the range of a region column, so unscaled distance is quietly a weighting. The Data Preparation and Feature Engineering course measured this decision separately in three families and showed that its sign changes with the family; here scaling stands not as a correction but as the condition for the notion of distance to be comparable across columns. One more number needs attention: the baseline is not the same on the two sets. It is 0.7579 on the test set and 0.8135 on the validation set. Two sets coming out of the same split have drifted apart by 0.0556 in the suspicious share. This means that a model's validation number cannot be compared directly with its test number; each number is read against the baseline of its own set. #### Twenty-Four Candidates The family has two hyperparameters and neither is learned from the data. The neighbor count determines how many examples a decision rests on. The distance metric is the definition of the word "near": **squared distance**, which sums the squares of the differences; **absolute difference**, which sums the absolute differences; and **largest difference**, which looks only at the single largest difference. The three do not give the same ordering on the same set. ```python def squared_distance(a, b): return sum((p - q) * (p - q) for p, q in zip(a, b)) def largest_difference(a, b): return max(abs(p - q) for p, q in zip(a, b)) NEIGHBOR_COUNTS = [1, 3, 5, 9, 15, 21, 31, 45] METRICS = [("squared distance", squared_distance), ("absolute difference", abs_diff), ("largest difference", largest_difference)] def neighbor_labels(train, target, metric): return [([t for _, t in sorted((metric(v, w), t) for w, t in train)][:max(NEIGHBOR_COUNTS)], y) for v, y in target] CANDIDATES = {} for name, metric in METRICS: lv, lt = neighbor_labels(V_TRAIN, V_VAL, metric), neighbor_labels(V_TRAIN, V_TEST, metric) for k in NEIGHBOR_COUNTS: CANDIDATES[(name, k)] = (sum((sum(t[:k]) * 2 > k) == y for t, y in lv) / len(lv), sum((sum(t[:k]) * 2 > k) == y for t, y in lt) / len(lt)) for i, split in enumerate(("validation", "test")): print(f"{split + ' k =':<20}" + "".join(f"{k:>8}" for k in NEIGHBOR_COUNTS)) for name, _ in METRICS: print(f"{name:<20}" + "".join(f"{CANDIDATES[(name, k)][i]:>8.4f}" for k in NEIGHBOR_COUNTS)) cv = max(CANDIDATES, key=lambda a: (CANDIDATES[a][0], -a[1])) ct = max(CANDIDATES, key=lambda a: (CANDIDATES[a][1], -a[1])) print(f"\ncandidates tried {len(CANDIDATES)}") print(f"chosen on validation: {cv[0]}, k={cv[1]} -> validation {CANDIDATES[cv][0]:.4f}, " f"test {CANDIDATES[cv][1]:.4f}") print(f"chosen on test: {ct[0]}, k={ct[1]} -> validation {CANDIDATES[ct][0]:.4f}, " f"test {CANDIDATES[ct][1]:.4f}") print(f"validation - test on the chosen candidate {CANDIDATES[cv][0] - CANDIDATES[cv][1]:+.4f}; " f"family number {CANDIDATES[cv][1] - BASELINE_TEST:+.4f} above the baseline") print(f"candidates above the validation baseline " f"{sum(1 for a in CANDIDATES if CANDIDATES[a][0] > BASELINE_VAL)}, " f"above the test baseline {sum(1 for a in CANDIDATES if CANDIDATES[a][1] > BASELINE_TEST)}") ``` ``` validation k = 1 3 5 9 15 21 31 45 squared distance 0.7540 0.7738 0.7698 0.8214 0.8175 0.8254 0.8214 0.8254 absolute difference 0.7778 0.7738 0.7937 0.8254 0.8214 0.8214 0.8294 0.8254 largest difference 0.7500 0.7897 0.8056 0.8214 0.8056 0.8095 0.8056 0.8135 test k = 1 3 5 9 15 21 31 45 squared distance 0.7262 0.7341 0.7341 0.7579 0.7619 0.7738 0.7778 0.7937 absolute difference 0.7063 0.7262 0.7302 0.7698 0.7698 0.7619 0.7937 0.7937 largest difference 0.7262 0.7460 0.7619 0.7817 0.7897 0.7738 0.7579 0.7778 candidates tried 24 chosen on validation: absolute difference, k=31 -> validation 0.8294, test 0.7937 chosen on test: absolute difference, k=31 -> validation 0.8294, test 0.7937 validation - test on the chosen candidate +0.0357; family number +0.0357 above the baseline candidates above the validation baseline 11, above the test baseline 14 ``` The first column shows the best-known flaw of the family. The procedure that looks at a single neighbor is **below** the baseline on the test set under all three metrics: 0.7262, 0.7063 and 0.7262. One neighbor means copying the label of a single row in the training set; if that row sits on the wrong side, the decision sits on the wrong side too. As the neighbor count grows, each decision rests on more rows and the number rises. The choice read from the validation set is absolute difference with 31 neighbors; that candidate's validation number is 0.8294 and its test number is 0.7937. **The family number is 0.7937**, and it beats the baseline by 0.0357, that is, nine subscribers. If the choice were made on the test set from the same table, the chosen candidate would not change; this is luck, and it shows in the table, because 0.7937 appears on three different candidates at once on the test set. What would change is not the candidate but **the reported number**: had the 0.8294 on the set the choice was made on been reported, the family would have seemed to gain eighteen subscribers instead of nine. The validation set is a set spent on choosing, and the highest number read on it is not an estimate of generalization. The number of candidates tried is **24**, and this number cannot be read without standing next to the family number. Of the twenty-four candidates, 11 beat their own baseline on the validation set, one comes out exactly equal to the baseline, and the remaining twelve are worse than a procedure that says nothing. The sentence "the family beat the baseline by 0.0357" is not the same thing as the sentence "the best of twenty-four candidates beat the baseline by 0.0357". #### The Curse of Dimensionality Distance stops being distinguishing as the column count grows. The mechanism is in the sum itself: every new column adds a term to the sum, and if the terms are independent of each other, the sum approaches a common mean as the column count grows. As a result the ratio between the farthest neighbor and the nearest neighbor falls to 1 and the word "nearest" loses its meaning. This is known as the **curse of dimensionality**, and it can be counted. ```python def noisy(s, d, seed): # FICTIONAL: columns that carry no information out = [] for i, x in enumerate(s): r = rng(seed + i) out.append(([x[a] for a in FIELDS] + [(r() + r() + r() - 1.5) * 1.15 for _ in range(d)], x["suspicious"])) return out print(f"{'columns':>7} {'farthest / nearest':>19} {'test':>8} {'vs baseline':>12}") for d in (0, 4, 12, 40, 120): NT, NE = noisy(S_TRAIN, d, SEED + 300000), noisy(S_TEST, d, SEED + 700000) ratio, correct = [], 0 for v, y in NE: u = sorted((abs_diff(v, w), t) for w, t in NT) ratio.append(u[-1][0] / u[0][0]) correct += (sum(t for _, t in u[:31]) * 2 > 31) == y print(f"{len(FIELDS) + d:>7} {statistics.fmean(ratio):>19.2f} " f"{correct / len(NE):>8.4f} {correct / len(NE) - BASELINE_TEST:>+12.4f}") ``` ``` columns farthest / nearest test vs baseline 12 26.91 0.7937 +0.0357 16 6.70 0.7817 +0.0238 24 3.46 0.7738 +0.0159 52 2.01 0.7698 +0.0119 132 1.49 0.7579 +0.0000 ``` The two columns are read together. At twelve columns the farthest training row is 26.91 times farther than the nearest one, and the word "neighbor" has a counterpart. When four noise columns are added the ratio falls to 6.70, and at forty columns to 2.01. At one hundred twenty noise columns the ratio is 1.49: less than half again separates the nearest training row from the farthest one, and the model settles exactly on **the baseline**, 0.7579. The family is no longer learning anything. None of the added columns carries wrong information; they carry no information at all. The difference comes from the family's inability to ignore these columns. A decision tree that picks a single column at each split can leave the uninformative column outside by not choosing it as a split candidate; a linear model that learns coefficients can shrink that column's coefficient. The sum of distances has no such means, because distance is not learned, it is defined. Feature selection and dimensionality reduction are not an improvement for this family but the condition for it to work. #### Summary - The training stage of an instance-based procedure is empty; the model is the 756-row training set itself, and the cost is on the prediction side: 756 distances are computed for every subscriber. - The candidate chosen on the validation set is absolute difference with 31 neighbors; the family number on the test set is 0.7937, 0.0357 above the baseline, and this number was found by trying **24 candidates**. - The same candidate's number on the validation set is 0.8294; had the number read on the set the choice was made on been reported, the family would have seemed to gain eighteen subscribers instead of nine. - The procedure that looks at a single neighbor falls below the baseline on the test set under all three distance metrics (0.7262, 0.7063 and 0.7262); thirteen of the twenty-four candidates fail to beat their own baseline on the validation set. - When the column count rises from 12 to 132, the ratio of the farthest neighbor to the nearest falls from 26.91 to 1.49, and the family number settles exactly on the baseline, 0.7579. #### Next Step In this family, twelve columns melt into a single sum of distances: the region indicator and the mean consumption meet in the same sum, and which of them carries the decision is not visible. For a categorical field, "distance" is already a forced notion; after the region columns are opened into five binary columns, the difference between two different regions is always the same number. The next lesson takes a family that does not sum the columns but **counts**: it counts how many times each feature value is seen under each class and multiplies those counts. For the multiplication to be valid, the columns must be independent of each other given the class; in this set they are not. The lesson counts the two separate consequences of the broken assumption separately, the probability output going to the extremes and the ranking staying standing, and measures how smoothing solves the problem of a never-seen count silencing the model entirely. ### Probabilistic Classifier with an Independence Assumption Source: https://academia.sh/en/courses/supervised-learning/independence-assumption-classifier Measuring a family that counts columns instead of summing them: instead of the 201,600 combinations of eight columns, a count table of 80 cells is kept, twenty candidates are tried and the candidate chosen on the validation set gives 0.7817 on the test set. A zero count eliminates a class entirely; with 60 training rows, 137 test rows are vetoed, and smoothing removes the veto but does not guarantee accuracy. When the column carrying the same information is added eight times, subscribers with extreme probabilities rise from 2 to 169 and the mean confidence from 0.842 to 0.970; the hits in the top 61 of the ranking go from 29 to 33. The previous lesson reduced twelve columns to a single sum of distances. In that sum, the region indicator and the mean consumption stand in the same list of terms and which of them carries the decision is not visible; for a categorical field, "distance" is already a notion attached after the fact. This lesson takes a family that does not sum the columns but **counts** them: it counts how many times each level of each column is seen under each class. There is an obstacle to classifying by counting. It is not possible to directly count how many times a particular column combination came out suspicious, because almost all of the combinations have never been seen in the training set. The family gets past this obstacle with a single assumption: it accepts that the columns are independent of each other given the class, and replaces the joint count with the **product** of the individual column counts. The assumption is not true in this set; the lesson counts the two separate consequences of the falseness separately. - **NM9.** The set, the split and the baseline do not change: a **fictional** table of 1,260 subscribers, seed 20260218, split 756/252/252, baseline 0.7579 on the test set and 61 suspicious subscribers. - **NM10.** The feature set is fixed; only its **representation** changes. The five binary columns of the region turn into a single five-level column, mean consumption is split into the 10, 25 and 40 m³ tariff bands from the Data Analytics curriculum, and volatility and per-person consumption are split into intervals that hold an equal number of rows. The interval boundaries are learned **from the training set alone**. - **NM11.** There are two hyperparameters and neither is learned from the data: the number of intervals and the **smoothing** coefficient. The product is computed as a sum of logarithms; the direct product of eight small probabilities falls to zero in floating point. - **NM12.** The **prior** is the share of a class in the training set; the **posterior** is its probability after the columns are seen. The decision threshold is 0.5; threshold selection is left to the evaluation topic of the course. - **NM13.** Text classification is not built and is left to the Natural Language Processing and Computer Vision course; all the fields used are categorical fields of the fictional subscriber table. - **NM14.** The choice is read from the validation set. In the ranking measurement the 61 subscribers with the highest probabilities are taken; 61 is the actual number of suspicious subscribers in the test set. #### From Counts to Probability The setup generates the set, builds the level representation and gives the reason for the assumption with two numbers. ```python # counting_model.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is generated with the same seed; # the feature set is fixed, only its representation is turned into levels. import collections import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "region_name": k["region"][0], "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] BASELINE_TEST = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def tariff_band(v): # tariff bands 10 / 25 / 40 m3 return 0 if v < 10 else (1 if v < 25 else (2 if v < 40 else 3)) def cutpoints(train, a, n): # LEARNING: interval boundaries from the training set v = sorted(x[a] for x in train) return [v[int(i * len(v) / n)] for i in range(1, n)] def level(v, cuts): return sum(1 for k in cuts if v >= k) def categorize(s, cuts): # APPLICATION return [({"tariff": tariff_band(x["mean_consumption"]), "region": x["region_name"], "household": x["household"], "satisfaction": x["satisfaction"], "period": x["period"], "zero_readings": x["zero_readings"], "volatility": level(x["volatility"], cuts["volatility"]), "per_person": level(x["per_person"], cuts["per_person"])}, x["suspicious"]) for x in s] COLUMNS = ["tariff", "region", "household", "satisfaction", "period", "zero_readings", "volatility", "per_person"] CUTS = {a: cutpoints(TRAIN, a, 8) for a in ("volatility", "per_person")} C_TRAIN, C_VAL, C_TEST = categorize(TRAIN, CUTS), categorize(VAL, CUTS), categorize(TEST, CUTS) print(f"train {len(C_TRAIN)}, test {len(C_TEST)}, suspicious in test {sum(y for _, y in C_TEST)}, " f"baseline {BASELINE_TEST:.4f}") LEVELS = {a: {x[a] for x, _ in C_TRAIN} for a in COLUMNS} combos, cells = 1, 0 for a in COLUMNS: combos, cells = combos * len(LEVELS[a]), cells + 2 * len(LEVELS[a]) print(f"columns {len(COLUMNS)}, combinations {combos}, training rows {len(C_TRAIN)}, " f"cells in the per-column count table {cells}") ``` ``` train 756, test 252, suspicious in test 61, baseline 0.7579 columns 8, combinations 201600, training rows 756, cells in the per-column count table 80 ``` The second line is the reason for the assumption. The levels of the eight columns produce 201,600 distinct combinations and there are 756 training rows; keeping the joint count directly is not an option. Counting the columns separately reduces the same table to **80 cells**. The price is in the multiplication itself: the probability that a subscriber is suspicious is the product of the class prior and the shares of the subscriber's level in each column under that class, and this product gives the right number only if the columns are independent **given the class**. #### The Model's Number and Twenty Candidates The model is written with two hyperparameters and a grid is scanned. The table printed belongs to the validation set; the test set numbers are given in the summary lines. ```python def fit(rows, columns): # MODEL: counts and prior shares totals = collections.Counter(y for _, y in rows) counts = {(a, c): collections.Counter(d[a] for d, y in rows if y == c) for a in columns for c in (0, 1)} return totals, counts, {a: len({d[a] for d, _ in rows}) for a in columns} def posterior(m, d, columns, alpha): # class 1 probability; the product is a sum of logs totals, counts, levels = m n, logs = sum(totals.values()), [] for c in (0, 1): t = math.log(totals[c] / n) for a in columns: numerator = counts[(a, c)][d[a]] + alpha t = -math.inf if numerator <= 0.0 else t + math.log( numerator / (totals[c] + alpha * levels[a])) logs.append(t) if logs[0] == -math.inf and logs[1] == -math.inf: return None # both classes eliminated: the model goes silent e = [math.exp(t - max(logs)) for t in logs] return e[1] / (e[0] + e[1]) def evaluate(train, target, columns, alpha): m, p, silent = fit(train, columns), [], 0 for d, y in target: o = posterior(m, d, columns, alpha) if o is None: silent, o = silent + 1, 0.0 p.append((o, y)) return sum((o > 0.5) == y for o, y in p) / len(p), p, silent GRID, ALPHAS = {}, (0.0, 0.25, 1.0, 4.0) for n in (3, 4, 6, 8, 12): CUTS_N = {a: cutpoints(TRAIN, a, n) for a in ("volatility", "per_person")} TR_N, VA_N, TE_N = (categorize(TRAIN, CUTS_N), categorize(VAL, CUTS_N), categorize(TEST, CUTS_N)) for alpha in ALPHAS: GRID[(n, alpha)] = (evaluate(TR_N, VA_N, COLUMNS, alpha)[0], evaluate(TR_N, TE_N, COLUMNS, alpha)[0]) print(f"{'bins':>6}" + "".join(f"{a:>9}" for a in ALPHAS) + " (validation)") for n in (3, 4, 6, 8, 12): print(f"{n:>6}" + "".join(f"{GRID[(n, a)][0]:>9.4f}" for a in ALPHAS)) cv = max(GRID, key=lambda a: (GRID[a][0], -a[0], -a[1])) ct = max(GRID, key=lambda a: (GRID[a][1], -a[0], -a[1])) print(f"\ncandidates tried {len(GRID)}") for label, a in (("on validation", cv), ("on test", ct)): print(f"{'chosen ' + label + ':':<22}bins {a[0]}, alpha {a[1]} -> validation " f"{GRID[a][0]:.4f}, test {GRID[a][1]:.4f}") print(f"family number {GRID[cv][1]:.4f}, {GRID[cv][1] - BASELINE_TEST:+.4f} above the " f"baseline; the two selection sets differ by {GRID[ct][1] - GRID[cv][1]:+.4f}") ``` ``` bins 0.0 0.25 1.0 4.0 (validation) 3 0.7540 0.7540 0.7540 0.7579 4 0.7540 0.7540 0.7540 0.7579 6 0.7738 0.7738 0.7659 0.7698 8 0.7778 0.7778 0.7857 0.7857 12 0.7579 0.7579 0.7619 0.7698 candidates tried 20 chosen on validation: bins 8, alpha 1.0 -> validation 0.7857, test 0.7817 chosen on test: bins 4, alpha 4.0 -> validation 0.7579, test 0.7857 family number 0.7817, +0.0238 above the baseline; the two selection sets differ by +0.0040 ``` The family number is **0.7817**; it beats the baseline by 0.0238, six subscribers, and was found by trying **20 candidates**. The instance-based procedure of the previous lesson had read 0.7937 with the same feature set. This family, which produces probabilities and rests on an explicit assumption, stayed **behind** the procedure that sums distances; the difference is three subscribers. This is the first example of the course's second claim and it is not hidden. The two selection sets give different candidates. From validation, eight intervals with 1.0 emerge; had the test set been consulted, four intervals with 4.0 would be chosen and the reported number would be 0.7857. That candidate's validation number is 0.7579, which puts it among the worst of the twenty candidates. #### Zero Counts and Smoothing The sparsest of the eighty cells is the intersection of the top band of the tariff column with the suspicious class, and its count is zero: a subscriber with consumption above forty cubic meters has never been seen as suspicious in training. In a product, a single zero erases everything the other seven columns say. **Smoothing** adds a small constant to every count; its reason is not a performance setting but this structural flaw. ```python def veto_count(train, target, columns, all_levels): seen = {(a, c): {d[a] for d, y in train if y == c} for a in columns for c in (0, 1)} empty = sum(len(all_levels[a] - seen[(a, c)]) for a in columns for c in (0, 1)) veto = wrong = 0 for d, y in target: eliminated = [c for c in (0, 1) if any(d[a] not in seen[(a, c)] for a in columns)] if eliminated: veto, wrong = veto + 1, wrong + (y in eliminated) return empty, veto, wrong print(f"{'train':>6} {'empty cells':>12} {'vetoed':>10} {'wrong veto':>12} " f"{'alpha=0':>8} {'alpha=1':>8}") for m in (60, 120, 250, 500, 756): b, v, y = veto_count(C_TRAIN[:m], C_TEST, COLUMNS, LEVELS) print(f"{m:>6} {b:>12} {v:>10} {y:>12} " f"{evaluate(C_TRAIN[:m], C_TEST, COLUMNS, 0.0)[0]:>8.4f} " f"{evaluate(C_TRAIN[:m], C_TEST, COLUMNS, 1.0)[0]:>8.4f}") ``` ``` train empty cells vetoed wrong veto alpha=0 alpha=1 60 8 137 24 0.7341 0.7302 120 4 56 9 0.7460 0.7579 250 2 1 1 0.7381 0.7540 500 1 1 1 0.7659 0.7698 756 1 1 1 0.7778 0.7817 ``` With sixty training rows, eight of the eighty cells are empty, and these eight gaps veto **137** of the 252 test rows; in 24 of them the vetoed class is the subscriber's actual class, that is, the decision rests not on the evidence of eight columns but on a single missing count. When the training grows to 756 rows, the empty cells fall to one and the veto to a single row; the moment the smoothing coefficient is greater than zero, the veto is lifted entirely. The last two columns show the limit of the fix. With sixty training rows, smoothing **lowers** the number from 0.7341 to 0.7302; at 120 rows it raises it from 0.7460 to 0.7579. What it pays for is not accuracy but the model not going silent. #### When the Assumption Breaks Conditional independence is a measurable claim: how far the joint count of two columns within a class deviates from the product of their individual shares. If the deviation is zero, the assumption holds for that pair. ```python def dependence(rows, a, b): # deviation of the joint count from the product, given the class t = 0.0 for c in (0, 1): sub = [d for d, y in rows if y == c] n = len(sub) pa = collections.Counter(d[a] for d in sub) pb = collections.Counter(d[b] for d in sub) pab = collections.Counter((d[a], d[b]) for d in sub) t += n / len(rows) * sum(abs(pab[(u, v)] / n - pa[u] * pb[v] / n / n) for u in pa for v in pb) / 2 return t for a, b in (("household", "per_person"), ("period", "volatility"), ("period", "zero_readings")): print(f"{a:<10} {b:<14} deviation {dependence(C_TRAIN, a, b):.3f}") def duplicate(rows, k): # the column carrying the same information re-enters the table return [(dict(d, **{f"tariff_{i}": d["tariff"] for i in range(k)}), y) for d, y in rows] POS = sum(y for _, y in C_TEST) print(f"\n{'copies':>6} {'columns':>7} {'accuracy':>9} {'extreme prob':>12} " f"{'mean confidence':>15} {'top-' + str(POS) + ' hits':>14}") for k in (0, 1, 2, 4, 8): cols = COLUMNS + [f"tariff_{i}" for i in range(k)] d, p, _ = evaluate(duplicate(C_TRAIN, k), duplicate(C_TEST, k), cols, 1.0) top = sorted(p, key=lambda t: -t[0])[:POS] print(f"{k:>6} {len(cols):>7} {d:>9.4f} " f"{sum(1 for o, _ in p if o > 0.99 or o < 0.01):>12} " f"{statistics.fmean(max(o, 1 - o) for o, _ in p):>15.3f} " f"{sum(y for _, y in top):>14}") ``` ``` household per_person deviation 0.407 period volatility deviation 0.235 period zero_readings deviation 0.011 copies columns accuracy extreme prob mean confidence top-61 hits 0 8 0.7817 2 0.842 29 1 9 0.7778 5 0.862 32 2 10 0.7778 17 0.889 33 4 12 0.7817 72 0.932 32 8 16 0.7857 169 0.970 33 ``` The assumption is already broken in this table. The deviation between household size and per-person consumption is 0.407, and this is the definition itself: per-person consumption is mean consumption divided by household size. Period count and volatility deviate by 0.235, because the volatility of a subscriber with a single period is zero. The most independent pair is the period count and zero readings, at 0.011. The lower table separates two consequences. **The probability output goes to the extremes:** the number of subscribers with extreme probabilities rises from 2 to 169 and the mean confidence from 0.842 to 0.970; when the same evidence is multiplied eight times, the model behaves as if it had seen eight independent pieces of evidence. **The ranking, on the other hand, stays standing:** the number of actual suspicious subscribers in the top 61 rises from 29 to 33 and accuracy goes from 0.7817 to 0.7857. Because the copied column pushes the scores of all subscribers in the same direction, it does not break the ranking; what it breaks is **the magnitude of the number**. The output of this family can be used as a ranking score, but not as a measure of confidence. #### Summary - Against the 201,600 combinations of eight columns there are 756 training rows; the conditional independence assumption reduces the count table to 80 cells, and this is the family's only justification. - The family number is 0.7817, 0.0238 above the baseline, and was found by trying **20 candidates**; the instance-based procedure of the previous lesson had read 0.7937 with the same feature set. - Had the choice been made on the test set, a different candidate would be chosen and the reported number would be 0.7857. - A zero count eliminates a class entirely: with 60 training rows, 137 test rows are vetoed and in 24 of them the vetoed class is the right class. Smoothing removes the veto but does not guarantee accuracy; at 60 rows the number falls from 0.7341 to 0.7302. - When the assumption breaks, two consequences separate: subscribers with extreme probabilities rise from 2 to 169 while the hits in the top 61 go from 29 to 33. #### Next Step This family's decision rests on the counts of all the training rows: a subscriber standing far from the boundary and a subscriber standing exactly on the boundary both grow the same cell by one unit. In the neighbor procedure of the previous lesson too, each of the 31 neighbors cast an equal vote. The next lesson takes a family that does the opposite. The **margin-based classifier** is concerned only with the examples near the boundary; it places the boundary in the middle of the gap between the two classes and counts openly the examples that carry the decision as **support vectors**. The lesson sweeps the coefficient that tunes the margin width, prints how many examples carry the decision at each width, and measures with explicitly computed polynomial features the **kernel transform** that produces a curved boundary without opening columns. ### Margin-Based Classifier Source: https://academia.sh/en/courses/supervised-learning/margin-based-classifier Measuring a family that bases its decision only on the examples near the boundary: as the penalty coefficient is swept, the margin widens from 0.047 to 93.488, the candidate chosen on the validation set gives 0.7857 on the test set and the rule that resolves the tie alone is worth four subscribers. The decision is carried by the 324 rows inside the margin; dropping 300 rows from outside the margin changes 4 predictions, dropping 300 rows from inside changes 21. Nine candidates are tried with three kernels and the curved boundary cannot beat the straight one; the polynomial kernel gives the same prediction on 252 of 252 test rows as 90 explicitly computed columns, without opening those 90 columns at all. The previous two families based their decision on the whole training set with equal weight: in the neighbor procedure each of the 31 neighbors cast a vote, in the counting family each row grew a cell by one unit. This lesson takes a family that does the opposite. The **margin-based classifier** places the boundary in the middle of the gap between the two classes and concerns itself only with the examples that fall into that gap; the width of the gap is called the **margin**. The family also carries a construct that can produce a curved boundary without opening columns: the **kernel transform**. The lesson counts all three: the relation between the margin width and the held-out numbers, the number of examples that carry the decision, and what the kernel gains against explicit columns. - **NM17.** The set, the split and the baseline do not change: a **fictional** table of 1,260 subscribers, seed 20260218, split 756/252/252, baseline 0.7579 on the test set and 0.8135 on validation. The feature set is the twelve columns that come from the Data Preparation and Feature Engineering course; the scale is learned from the training set alone. - **NM18.** The labels are written as $+1$ and $-1$. The model minimizes the sum of the margin loss and the squared penalty on the weights; the **penalty coefficient** is the tuning variable of this lesson and the margin width is $2 / \lVert w \rVert$. - **NM19.** A **support vector** is a training row that stays inside the margin or on the wrong side of it; those outside the margin contribute nothing to the loss. - **NM20.** The solver is a subgradient descent whose step shrinks as $1/(\lambda t)$ and runs 300 rounds in the primal form and 40 rounds in the dual form; the numbers printed are the numbers of this **fixed budget**. - **NM21.** A **kernel** is a function that gives the inner product in an expanded space without building that space. The polynomial kernel is $(x \cdot z + 1)^2/10$ and the radial kernel is $e^{-0.05 \lVert x - z \rVert^2}$; the decimal coefficients are fixed so that the same penalty range can be used with all three kernels. The dual form keeps as many kernel values as the square of the number of training rows. - **NM22.** The choice is read from the validation set; in a tie the **wider margin** is chosen. #### The Width of the Margin The setup generates the set, writes the model and sweeps the penalty coefficient over eight values. ```python # margin.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the feature set comes from the Data Preparation and Feature Engineering course and is fixed throughout the course. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "region_name": k["region"][0], "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) x["per_person"] = round(x["mean_consumption"] / x["household"], 3) DATA.append(x) NUMERIC = ["mean_consumption", "volatility", "household", "satisfaction", "period", "per_person"] FIELDS = NUMERIC + ["zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] BASELINE_TEST = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) SCALE = {a: (statistics.fmean(x[a] for x in TRAIN), statistics.pstdev([x[a] for x in TRAIN])) for a in NUMERIC} def vectorize(s): # labels +1 / -1 return [([(x[a] - SCALE[a][0]) / SCALE[a][1] if a in SCALE else x[a] for a in FIELDS], 1 if x["suspicious"] else -1) for x in s] V_TRAIN, V_VAL, V_TEST = vectorize(TRAIN), vectorize(VAL), vectorize(TEST) def score(w, b, v): return sum(w[i] * v[i] for i in range(len(w))) + b def train_margin(train, lam, rounds=300, n=None): # MODEL: margin-based classifier d, n = len(train[0][0]), n or len(train) w, b = [0.0] * d, 0.0 for t in range(1, rounds + 1): step, gw, gb = 1.0 / (lam * t), [0.0] * d, 0.0 for v, y in train: if y * score(w, b, v) < 1: # inside the margin or on the wrong side for i in range(d): gw[i] -= y * v[i] gb -= y for i in range(d): w[i] -= step * (gw[i] / n + lam * w[i]) b -= step * gb / n return w, b def accuracy(m, target): return sum((score(m[0], m[1], v) > 0) == (y > 0) for v, y in target) / len(target) CANDIDATES = {} print(f"baseline on test {BASELINE_TEST:.4f}; training {len(V_TRAIN)} rows, {len(FIELDS)} columns") print(f"{'penalty':>8} {'|w|':>7} {'margin':>8} {'support':>7} {'validation':>10} {'test':>8}") for lam in (0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0): m = train_margin(V_TRAIN, lam) nw = math.sqrt(sum(x * x for x in m[0])) CANDIDATES[lam] = (accuracy(m, V_VAL), accuracy(m, V_TEST)) inside = sum(1 for v, y in V_TRAIN if y * score(m[0], m[1], v) < 1) print(f"{lam:>8} {nw:>7.3f} {2 / nw:>8.3f} {inside:>7} " f"{CANDIDATES[lam][0]:>10.4f} {CANDIDATES[lam][1]:>8.4f}") best = max(CANDIDATES, key=lambda a: (CANDIDATES[a][0], a)) smallest = max(CANDIDATES, key=lambda a: (CANDIDATES[a][0], -a)) print(f"candidates tried {len(CANDIDATES)}; penalty chosen on validation {best} -> test " f"{CANDIDATES[best][1]:.4f}, {CANDIDATES[best][1] - BASELINE_TEST:+.4f} above the baseline") print(f"had the tie been resolved in favor of the smallest penalty, penalty {smallest} -> test " f"{CANDIDATES[smallest][1]:.4f}") ``` ``` baseline on test 0.7579; training 756 rows, 12 columns penalty |w| margin support validation test 0.0003 42.768 0.047 252 0.7421 0.7619 0.001 12.742 0.157 338 0.8294 0.8016 0.003 4.289 0.466 320 0.8294 0.7976 0.01 1.587 1.260 324 0.8294 0.7857 0.03 0.833 2.401 333 0.8175 0.7857 0.1 0.266 7.523 334 0.8135 0.7579 0.3 0.075 26.789 335 0.8135 0.7579 1.0 0.021 93.488 335 0.8135 0.7579 candidates tried 8; penalty chosen on validation 0.01 -> test 0.7857, +0.0278 above the baseline had the tie been resolved in favor of the smallest penalty, penalty 0.001 -> test 0.8016 ``` The margin widens from 0.047 to 93.488 and both ends of the table are bad. At the narrowest margin the length of the weight vector climbs to 42.768 and the validation number falls **0.0714 below** its own baseline; at the widest margin the boundary flattens and on the test set the model reads exactly the baseline. The three candidates in between are tied on the validation set at 0.8294. Resolving the tie is a decision and the last two lines give its cost. When it is resolved in favor of the wide margin, penalty 0.01 is chosen and the **family number is 0.7857**, 0.0278 above the baseline; had it been resolved in favor of the narrow margin, 0.8016 would have been reported. The 0.0159 between them is four subscribers and comes from no measurement, only from the tie rule. #### How Many Examples Carry the Decision The number of support vectors is in the table but it is not a claim on its own; whether they carry the decision is measured by dropping those rows and retraining the model. ```python CHOSEN = 0.01 M = train_margin(V_TRAIN, CHOSEN) P0 = [score(M[0], M[1], v) > 0 for v, _ in V_TEST] INSIDE = [i for i, (v, y) in enumerate(V_TRAIN) if y * score(M[0], M[1], v) < 1] OUTSIDE = [i for i, (v, y) in enumerate(V_TRAIN) if y * score(M[0], M[1], v) >= 1] r = rng(SEED + 610000) def random_pick(pool, k): h = list(pool) for i in range(len(h) - 1, 0, -1): j = int(r() * (i + 1)) h[i], h[j] = h[j], h[i] return set(h[:k]) print(f"penalty {CHOSEN}: {len(INSIDE)} rows inside the margin, {len(OUTSIDE)} rows outside the margin") print(f"{'dropped':>7} {'pool':>14} {'changed predictions':>19} {'test':>8}") for k in (200, 300): for name, pool in (("outside margin", OUTSIDE), ("inside margin", INSIDE)): dropped = random_pick(pool, k) m2 = train_margin([p for i, p in enumerate(V_TRAIN) if i not in dropped], CHOSEN, n=len(V_TRAIN)) p2 = [score(m2[0], m2[1], v) > 0 for v, _ in V_TEST] print(f"{k:>7} {name:>14} {sum(a != b for a, b in zip(P0, p2)):>19} " f"{accuracy(m2, V_TEST):>8.4f}") ``` ``` penalty 0.01: 324 rows inside the margin, 432 rows outside the margin dropped pool changed predictions test 200 outside margin 0 0.7857 200 inside margin 15 0.7817 300 outside margin 4 0.7937 300 inside margin 21 0.7579 ``` In the chosen candidate, 324 of the 756 training rows are inside the margin; this is a measure of how intertwined the classes are. When 300 rows are dropped from **outside** the margin, the prediction changes on only four of the 252 test rows, and when 200 rows are dropped none changes. When the same number of rows is dropped from **inside** the margin, the number of changed predictions rises to 21. The number of changed predictions is not a loss measure either: when 300 rows are dropped from outside the margin, the test number **rises** to 0.7937. What is measured is the movement of the boundary. #### The Kernel Transform A curved boundary is obtained in two ways. The first is opening columns: squares and products are computed and added to the table. The second is the kernel transform: since the model uses nothing but the inner product of two rows in the expanded space, writing a function that gives that inner product directly makes building the space unnecessary. For this the model is written in the **dual form**: instead of a weight vector, the coefficients of the training rows are kept. ```python def gram(A, B): # inner product matrix; kernels derive from it return [[sum(p * q for p, q in zip(a, b)) for b, _ in B] for a, _ in A] def kernel(name, G, n1, n2): if name == "linear": return G if name == "polynomial": # (x . z + 1)^2 / 10 return [[(g + 1.0) ** 2 / 10.0 for g in s] for s in G] return [[math.exp(-0.05 * (n1[i] + n2[j] - 2 * g)) for j, g in enumerate(s)] for i, s in enumerate(G)] # radial kernel def train_dual(K, y, lam, rounds=40): # MODEL: the same criterion, in dual form n, a, b = len(y), [0.0] * len(y), 0.0 for t in range(1, rounds + 1): step = 1.0 / (lam * t) f = [sum(a[j] * K[i][j] for j in range(n)) + b for i in range(n)] a, gb = [x * (1.0 - step * lam) for x in a], 0.0 for i in range(n): if y[i] * f[i] < 1: a[i] += step * y[i] / n gb += y[i] b += step * gb / n return a, b def dual_accuracy(a, b, KH, target): return sum((sum(a[j] * KH[i][j] for j in range(len(a))) + b > 0) == (y > 0) for i, (v, y) in enumerate(target)) / len(target) Y = [y for _, y in V_TRAIN] G_TRAIN, G_VAL, G_TEST = gram(V_TRAIN, V_TRAIN), gram(V_VAL, V_TRAIN), gram(V_TEST, V_TRAIN) N_TRAIN = [sum(p * p for p in v) for v, _ in V_TRAIN] N_VAL = [sum(p * p for p in v) for v, _ in V_VAL] N_TEST = [sum(p * p for p in v) for v, _ in V_TEST] KERNELS = {} print(f"{'kernel':<12}{'penalty':>8}{'validation':>11}{'test':>8}") for name in ("linear", "polynomial", "radial"): K = kernel(name, G_TRAIN, N_TRAIN, N_TRAIN) KV, KT = kernel(name, G_VAL, N_VAL, N_TRAIN), kernel(name, G_TEST, N_TEST, N_TRAIN) for lam in (0.003, 0.01, 0.03): a, b = train_dual(K, Y, lam) KERNELS[(name, lam)] = (dual_accuracy(a, b, KV, V_VAL), dual_accuracy(a, b, KT, V_TEST)) print(f"{name:<12}{lam:>8}{KERNELS[(name, lam)][0]:>11.4f}{KERNELS[(name, lam)][1]:>8.4f}") chosen = max(KERNELS, key=lambda a: (KERNELS[a][0], -a[1])) print(f"candidates tried {len(CANDIDATES) + len(KERNELS)} (linear {len(CANDIDATES)} + kernel " f"{len(KERNELS)}); chosen on validation {chosen[0]}, penalty {chosen[1]} -> test " f"{KERNELS[chosen][1]:.4f}") print(f"kernel matrix {len(V_TRAIN)} x {len(V_TRAIN)} = {len(V_TRAIN) ** 2} entries") ``` ``` kernel penalty validation test linear 0.003 0.8016 0.7817 linear 0.01 0.8095 0.7897 linear 0.03 0.8294 0.7857 polynomial 0.003 0.7778 0.7857 polynomial 0.01 0.7897 0.7778 polynomial 0.03 0.8175 0.7937 radial 0.003 0.7222 0.6627 radial 0.01 0.8135 0.7579 radial 0.03 0.8135 0.7579 candidates tried 17 (linear 8 + kernel 9); chosen on validation linear, penalty 0.03 -> test 0.7857 kernel matrix 756 x 756 = 571536 entries ``` None of the nine candidates can beat the straight boundary. The choice read from validation is the **linear kernel** and the test number is 0.7857, the same number the primal form gave. The polynomial kernel's highest test number is 0.7937 and it exceeds the linear kernel's, but its validation number stays at 0.8175; choosing that candidate would be looking at the test set. The radial kernel at the smallest penalty falls 0.0952 below the baseline. The difference the curved boundary makes on this set is **zero** when the choice is read from the validation set; its cost is visible, though: the dual form keeps 756 coefficients and computes 571,536 kernel entries. What remains is the kernel's real claim: the same model should give the same decision when the columns are computed explicitly. ```python def explicit_polynomial(v): # explicit columns equivalent to the (x . z + 1)^2 / 10 kernel d = len(v) o = [math.sqrt(2) * v[i] for i in range(d)] + [v[i] * v[i] for i in range(d)] o += [math.sqrt(2) * v[i] * v[j] for i in range(d) for j in range(i + 1, d)] return [x / math.sqrt(10.0) for x in o] X_TRAIN = [(explicit_polynomial(v), y) for v, y in V_TRAIN] X_TEST = [(explicit_polynomial(v), y) for v, y in V_TEST] LAM = 0.03 ma = train_margin(X_TRAIN, LAM, rounds=40) KT_, KS_ = kernel("polynomial", G_TRAIN, N_TRAIN, N_TRAIN), kernel("polynomial", G_TEST, N_TEST, N_TRAIN) a, b = train_dual(KT_, Y, LAM) tk = [sum(a[j] * KS_[i][j] for j in range(len(a))) + b > 0 for i in range(len(V_TEST))] ta = [score(ma[0], ma[1], v) > 0 for v, _ in X_TEST] print(f"explicitly computed columns {len(X_TRAIN[0][0])}, columns opened by the kernel 0") print(f"test with explicit columns {accuracy(ma, X_TEST):.4f}") print(f"test with the kernel {KERNELS[('polynomial', LAM)][1]:.4f}") print(f"test rows on which the two forms give the same prediction " f"{sum(p == q for p, q in zip(tk, ta))}/{len(V_TEST)}") ``` ``` explicitly computed columns 90, columns opened by the kernel 0 test with explicit columns 0.7937 test with the kernel 0.7937 test rows on which the two forms give the same prediction 252/252 ``` All the second-degree terms of twelve columns open **90 columns** and the two forms give the same prediction on **252 of the 252** test rows. The gain is not in the number but in the columns that are not opened: had the degree gone up to three, the number of columns to open would go to 454 while the kernel function would remain a single change of exponent. **Seventeen candidates** were tried, the family number is 0.7857, and the curved boundary added nothing to the difference above the baseline. #### Summary - The margin width goes from 0.047 to 93.488 with the penalty coefficient; at the narrowest margin the validation number falls 0.0714 below its own baseline. - The family number is 0.7857 and it beats the baseline by 0.0278; because three candidates tied on validation, 0.0159 of this number comes from the tie rule. - The decision is carried by the 324 rows inside the margin: when 300 rows are dropped from outside the margin 4 predictions change, from inside 21. - Nine candidates were tried with three kernels; the chosen kernel is **linear** and the curved boundary added nothing. - The polynomial kernel gives the same prediction on 252 of 252 test rows as 90 explicitly computed columns; the cost of the dual form is 571,536 kernel entries. #### Next Step The three families shared one constraint: the sum of distances, the product of probabilities and the inner product all use every column at once, and which column carries the decision can be read only indirectly. The next lesson turns to a family that handles the columns one at a time: the **decision tree** chooses a single column and a single threshold at each step and makes the choice according to a **split criterion**. The tree itself was used and its depth measured throughout the Introduction to Machine Learning and Data Preparation and Feature Engineering courses; what has not been asked is which criterion chooses the split and what **pruning** a grown tree takes back, instead of limiting the depth in advance. ### Decision Trees Source: https://academia.sh/en/courses/supervised-learning/decision-trees Measuring the split chosen by the split criterion and how pruning differs from a depth limit: impurity and entropy choose the same root split but agree on only eighteen of forty-seven splits, while the misclassification rate chooses an entirely different 695/61 split. The unpruned depth-12 tree with 127 leaves gives 0.7341 on the test set, below the baseline; pruning brings the same tree down to 20 leaves and up to 0.7857. The depth limit gives 0.7897 with 48 leaves while pruning gives 0.7857 with 11 leaves, and four of those are deeper than the sixth level. The trees of eight 90% subsamples share not a single split. The previous three families used the columns together: the sum of distances, the product of probabilities and the inner product all brought all twelve columns into every decision. The **decision tree** does the opposite; at each step it chooses a single column and a single threshold, splits the set in two and repeats the same work on the two parts. The tree itself is not new in this curriculum: a tree with a fixed depth of 6 was used throughout the Introduction to Machine Learning and Data Preparation and Feature Engineering courses, and the divergence of the two numbers as capacity grows and the effect of the leaf minimum were measured in the generalization topic of the Introduction to Machine Learning course. Those measurements are not repeated here. Two things have not been asked: **which criterion** chooses the split, and what **pruning** a grown tree takes back, instead of limiting the depth in advance. - **NM24.** The set, the split and the baseline do not change: a **fictional** table of 1,260 subscribers, seed 20260218, split 756/252/252, baseline 0.7579 on the test set and 0.8135 on validation. The feature set is the twelve columns that come from the Data Preparation and Feature Engineering course. Since the tree looks at the ordering, no scaling is applied; the Data Preparation and Feature Engineering course measured the contribution of that step to the tree to be exactly zero. - **NM25.** The **split criterion** is the function that measures how much a split gains. Three are compared: **impurity**, **entropy** and the **misclassification rate**. The gain is the value before the split minus the value of the two parts weighted by their row counts. - **NM26.** At most ten candidate thresholds are tried for each column and at least two rows stay in each leaf. The depth sweep, the learning curve and the leaf minimum **were measured in the Introduction to Machine Learning course and are not repeated**; the depth 6 here is the constant from that course and the depth 12 is the equivalent of "let it grow without limit". - **NM27.** **Pruning** is turning a node of the grown tree into a leaf. The decision is read from the **validation set**: a node is turned into a leaf if, once turned, its error on the validation set does not increase. The **pruning threshold** is how much of an increase in error is tolerated and is the tuning variable. - **NM28.** The number of leaves and the depth of the leaves are the size measures of the tree; the two are printed separately. - **NM29.** The choice is read from the validation set; in a tie the tree with fewer leaves is chosen. - **NM30.** In the fragility measurement, 90% of the training set is drawn eight times with separate seeds; the validation and test sets do not change. #### What the Split Criterion Chooses The setup generates the set, writes the three criteria and prints the root split and the depth 6 trees. ```python # tree.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the feature set comes from the Data Preparation and Feature Engineering course and is fixed throughout the course. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) x["per_person"] = round(x["mean_consumption"] / x["household"], 3) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "per_person", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] BASELINE_TEST = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def share(s): return sum(x["suspicious"] for x in s) / len(s) if s else 0.0 def impurity(s): p = share(s) return 2 * p * (1 - p) def entropy(s): p = share(s) return 0.0 if p <= 0.0 or p >= 1.0 else -(p * math.log2(p) + (1 - p) * math.log2(1 - p)) def misclass_rate(s): # misclassification rate p = share(s) return min(p, 1 - p) CRITERIA = [("impurity", impurity), ("entropy", entropy), ("misclass rate", misclass_rate)] def best_split(s, f, least=2): best = None for a in FIELDS: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < least: continue k = f(s) - (len(left) * f(left) + len(right) * f(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) return best def tree(s, depth, f, least=2): # MODEL: decision tree p, pos = share(s), sum(x["suspicious"] for x in s) best = best_split(s, f, least) if depth and len(s) >= 2 * least and 0.0 < p < 1.0 else None if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5), "n": len(s), "pos": pos} return {"field": best[1], "threshold": best[2], "n": len(s), "pos": pos, "left": tree(best[3], depth - 1, f, least), "right": tree(best[4], depth - 1, f, least)} def predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def accuracy(d, s): return sum(predict(d, x) == x["suspicious"] for x in s) / len(s) def splits(d, path=()): # (position, column, threshold) triples if "prediction" in d: return [] return ([(path, d["field"], round(d["threshold"], 3))] + splits(d["left"], path + ("l",)) + splits(d["right"], path + ("r",))) print(f"baseline on test {BASELINE_TEST:.4f}; training {len(TRAIN)} rows") print(f"{'criterion':<14}{'root column':>11}{'threshold':>10}{'gain':>9}{'left/right':>12}") for name, f in CRITERIA: best = best_split(TRAIN, f) print(f"{name:<14}{best[1]:>11}{best[2]:>10.2f}{best[0]:>9.5f}" f"{str(len(best[3])) + '/' + str(len(best[4])):>12}") TREE6 = {} print(f"\n{'criterion (depth 6)':<20}{'leaves':>7}{'validation':>11}{'test':>8}") for name, f in CRITERIA: TREE6[name] = tree(TRAIN, 6, f) print(f"{name:<20}{len(splits(TREE6[name])) + 1:>7}" f"{accuracy(TREE6[name], VAL):>11.4f}{accuracy(TREE6[name], TEST):>8.4f}") B = {name: set(splits(t)) for name, t in TREE6.items()} print(f"common splits of the impurity and entropy trees " f"{len(B['impurity'] & B['entropy'])}, " f"common splits with misclassification rate {len(B['impurity'] & B['misclass rate'])}") ``` ``` baseline on test 0.7579; training 756 rows criterion root column threshold gain left/right impurity volatility 11.69 0.03923 511/245 entropy volatility 11.69 0.07761 511/245 misclass rate volatility 23.15 0.01984 695/61 criterion (depth 6) leaves validation test impurity 48 0.8016 0.7897 entropy 38 0.7778 0.7897 misclass rate 8 0.7937 0.7698 common splits of the impurity and entropy trees 18, common splits with misclassification rate 0 ``` Impurity and entropy agree on the root split: both choose the volatility column and the threshold 11.69 and divide the set 511/245. The gain numbers differ — 0.03923 and 0.07761 — because the two criteria do not measure the same magnitude; the split they choose is the same. The misclassification rate, on the same set, chooses an entirely different 695/61 split. The difference comes from the shape of the criterion. The misclassification rate is a function of the class share made of linear pieces: if the majority class of both parts stays the same, the gain is exactly zero, however much the split purifies the parts. Impurity and entropy are strictly concave and reward purification even when the majority class does not change. This is why the misclassification rate criterion sees only the splits that flip a majority and stops the tree at eight leaves; its test number also stays at 0.7698, below the 0.7897 of the others. The same root does not mean the same tree. The impurity tree carries forty-seven splits, the entropy tree thirty-seven, and only **eighteen** of them are shared as both column and threshold. Even when the criteria do not diverge at the root split, they diverge in the lower floors of the tree; while the two trees give the same number on the test set, they ask different questions. #### How Pruning Differs from a Depth Limit A depth limit cuts the tree at the same level in every branch. Pruning first lets the tree go free and then takes back **only the branches that do not pay**; whether they pay is measured on the held-out set. ```python def error(d, s, as_leaf): if as_leaf: t = int(d["pos"] / d["n"] > 0.5) return sum(1 for x in s if x["suspicious"] != t) return sum(1 for x in s if predict(d, x) != x["suspicious"]) def prune(d, s, threshold, n): # pruning read from the held-out set if "prediction" in d: return d d = dict(d, left=prune(d["left"], [x for x in s if x[d["field"]] < d["threshold"]], threshold, n), right=prune(d["right"], [x for x in s if x[d["field"]] >= d["threshold"]], threshold, n)) if error(d, s, True) <= error(d, s, False) + threshold * n: return {"prediction": int(d["pos"] / d["n"] > 0.5), "n": d["n"], "pos": d["pos"]} return d def leaf_depths(d, k=0): if "prediction" in d: return [k] return leaf_depths(d["left"], k + 1) + leaf_depths(d["right"], k + 1) PRUNED, THRESHOLDS = {}, (0.0, 0.004, 0.012) print(f"{'criterion':<14}{'pruning':>8}{'leaves':>8}{'depth':>9}{'validation':>11}{'test':>8}") for name, f in CRITERIA: grown = tree(TRAIN, 12, f) ld = leaf_depths(grown) print(f"{name:<14}{'none':>8}{len(ld):>8}{max(ld):>9}" f"{accuracy(grown, VAL):>11.4f}{accuracy(grown, TEST):>8.4f}") for threshold in THRESHOLDS: b = prune(grown, VAL, threshold, len(VAL)) ld = leaf_depths(b) PRUNED[(name, threshold)] = (accuracy(b, VAL), accuracy(b, TEST), len(ld), max(ld)) print(f"{name:<14}{threshold:>8}{len(ld):>8}{max(ld):>9}" f"{PRUNED[(name, threshold)][0]:>11.4f}{PRUNED[(name, threshold)][1]:>8.4f}") pick_ = max(PRUNED, key=lambda a: (PRUNED[a][0], PRUNED[a][2])) print(f"\ncandidates tried {len(CRITERIA) + len(PRUNED)} (criterion {len(CRITERIA)} + pruning {len(PRUNED)})") print(f"chosen on validation: {pick_[0]}, pruning threshold {pick_[1]} -> test " f"{PRUNED[pick_][1]:.4f}, {PRUNED[pick_][1] - BASELINE_TEST:+.4f} above the baseline") b = prune(tree(TRAIN, 12, impurity), VAL, 0.004, len(VAL)) print(f"depth limit 6: {len(splits(TREE6['impurity'])) + 1} leaves, test " f"{accuracy(TREE6['impurity'], TEST):.4f}") print(f"pruning threshold 0.004: {len(leaf_depths(b))} leaves, test " f"{accuracy(b, TEST):.4f}, leaves deeper than 6 " f"{sum(1 for k in leaf_depths(b) if k > 6)}") ``` ``` criterion pruning leaves depth validation test impurity none 127 12 0.7619 0.7341 impurity 0.0 20 8 0.8413 0.7857 impurity 0.004 11 8 0.8294 0.7857 impurity 0.012 1 0 0.8135 0.7579 entropy none 122 12 0.7381 0.7460 entropy 0.0 22 12 0.8373 0.7817 entropy 0.004 3 2 0.8214 0.7857 entropy 0.012 1 0 0.8135 0.7579 misclass rate none 8 4 0.7937 0.7698 misclass rate 0.0 1 0 0.8135 0.7579 misclass rate 0.004 1 0 0.8135 0.7579 misclass rate 0.012 1 0 0.8135 0.7579 candidates tried 12 (criterion 3 + pruning 9) chosen on validation: impurity, pruning threshold 0.0 -> test 0.7857, +0.0278 above the baseline depth limit 6: 48 leaves, test 0.7897 pruning threshold 0.004: 11 leaves, test 0.7857, leaves deeper than 6 4 ``` The unpruned tree is bad across the board: with 127 leaves it reads 0.7341 on the test set, that is, 0.0238 **below** the baseline. When the same tree is pruned it goes down to 20 leaves and up to 0.7857; the validation number goes from 0.7619 to 0.8413. **The family number is 0.7857**, it beats the baseline by 0.0278 and was found by trying **12 candidates**. The last three lines give how pruning differs from a depth limit. The depth 6 tree reads 0.7897 with 48 leaves, the pruned tree 0.7857 with 11 leaves: the difference in the number is one subscriber, the difference in leaves is more than fourfold. More important is that the pruned tree's **depth is 8** and four of its leaves are deeper than the sixth level. The depth limit would have cut those four leaves without looking at whether they pay; pruning kept them and threw away thirty-seven leaves elsewhere. What the two steps take back is not the same: one looks at the **level**, the other at the **contribution**. The rows of the misclassification rate criterion say something separate: at all three pruning thresholds the tree goes down to a single leaf and the model falls to the baseline. Not one split of its eight-leaf tree pays for itself on the validation set. #### How Sturdy a Single Tree's Decision Is The selected tree gave a number. How much of that number comes from the training set being exactly these 756 rows is measured separately. ```python def subsample(s, seed, share=0.9): # 90% of the training set r, k = rng(seed), list(s) for i in range(len(k) - 1, 0, -1): j = int(r() * (i + 1)) k[i], k[j] = k[j], k[i] return k[:int(len(k) * share)] COMMON, NUMBERS = None, [] print(f"{'sample':>9}{'root split':>26}{'leaves':>8}{'test':>8}") for i in range(8): sub = subsample(TRAIN, SEED + 820000 + i * 13) t = prune(tree(sub, 12, impurity), VAL, 0.004, len(VAL)) best = best_split(sub, impurity) b = set(splits(t)) COMMON = b if COMMON is None else COMMON & b NUMBERS.append(accuracy(t, TEST)) print(f"{i + 1:>9}{best[1] + ' < ' + format(best[2], '.2f'):>26}" f"{len(leaf_depths(t)):>8}{NUMBERS[-1]:>8.4f}") print(f"splits found in all eight trees (column and threshold together) {len(COMMON)}") print(f"test numbers between {min(NUMBERS):.4f} and {max(NUMBERS):.4f}, " f"range {max(NUMBERS) - min(NUMBERS):.4f}") ``` ``` sample root split leaves test 1 volatility < 13.99 3 0.7897 2 volatility < 13.88 10 0.7897 3 volatility < 13.80 13 0.7817 4 volatility < 14.12 3 0.7897 5 volatility < 11.41 8 0.7817 6 volatility < 13.60 5 0.7619 7 volatility < 13.91 6 0.7897 8 volatility < 14.12 10 0.7857 splits found in all eight trees (column and threshold together) 0 test numbers between 0.7619 and 0.7897, range 0.0278 ``` The root column stays the same in all eight samples: volatility. The threshold wanders between 11.41 and 14.12 and the number of leaves varies between 3 and 13 — same procedure, same settings, only a tenth of the training set changed. The number of splits found in **all** eight trees is **zero**: taking the column and the threshold together, there is not a single common split. The test numbers are between 0.7619 and 0.7897, a range of 0.0278. This number is equal to the whole of the family's difference above the baseline. The measured difference can be erased entirely, depending on which 90% of the training set is drawn. #### Summary - Impurity and entropy choose the same root split (volatility, 11.69, 511/245); the misclassification rate chooses a different 695/61 split, because it gives exactly zero gain to splits that do not flip the majority. - Two trees growing from the same root carry forty-seven and thirty-seven splits and agree on only eighteen of them. - The unpruned depth-12 tree reads 0.7341 with 127 leaves, 0.0238 below the baseline; pruning takes the same tree down to 20 leaves and up to 0.7857. - The family number is 0.7857, it beats the baseline by 0.0278 and was found by trying **12 candidates**. - Unlike a depth limit, pruning looks at the **contribution**: four leaves of the 11-leaf pruned tree are deeper than the sixth level and a depth limit would have cut them without measuring. - The trees of eight 90% subsamples share not a single split; the test numbers wander within a range of 0.0278. #### Next Step A single tree was pruned, the number left over beat the baseline by 0.0278 and twelve candidates were tried. But the last table undermines this number: the same procedure chooses entirely different splits when a tenth of the training set is changed, and the number it gives wanders as much as the whole of the difference above the baseline. The decision of a single tree is **fragile** and fragility is not a defect of the tree but a consequence of its definition: a single threshold is chosen at each node, and if that threshold is separated from its nearest rival by only a few rows, the choice changes when a few rows change. The next lesson, instead of correcting this fragility as a defect, uses it: it runs the same procedure many times on **different training samples** and combines the decisions. What is to be measured is the held-out number settling and the variability falling as the number of trees grows. ### Bagged Tree Ensembles Source: https://academia.sh/en/courses/supervised-learning/bagged-tree-ensembles Counting the fragility of a single tree and measuring what bagging adds: across 41 bag samples the same procedure chooses nine different column-threshold pairs for the root split, two trees classify on average 46.2 of the 252 subscribers in the validation set differently, and the single tree's test number wanders between 0.6905 and 0.7897. As the number of trees goes from 1 to 41, the spread on the validation set falls from 0.0193 to 0.0075. When feature randomness is added, the ensemble chosen on the validation set among 31 candidates gives 0.7897 on the test set; the out-of-bag estimate, 0.7884, stands 0.0013 from this number, while the validation set's own number stands 0.0516 away. The previous lesson separated the split criterion of the decision tree from pruning; the pruned tree beat the baseline and became the most readable model among the families tried in the four lessons. One of its properties was left unmeasured: when the training set changes a little, the tree chooses entirely different splits. When the root moves, everything under it moves; the number a single tree gives is not a measurement but the result of a draw. This lesson first counts that draw, then builds the way to narrow it. **Bagging** runs the same procedure separately on many samples drawn with replacement and takes the **majority vote** of the trees. **Feature randomness** adds a second split: each tree chooses its splits from a randomly chosen subset of columns. What is measured is not one number but two: the number of the ensemble and the spread of that number. - **NM31.** The set is the **fictional** subscriber table of the Introduction to Machine Learning course: 1,260 subscribers, 756/252/252, seed 20260218. The baseline is 0.7579 on the test set and 0.8135 on validation; the number the course carries is the one on the test set. - **NM32.** The feature set comes from the Data Preparation and Feature Engineering course and is eleven columns; the only thing that changes is the model. - **NM33.** The comparison point is the tree of the previous lessons: depth 6, leaf minimum 2, split criterion impurity reduction. The thresholds are extracted once from the training set and are at most twenty-four per column; this is a speed decision and does not change the splits that can be chosen. - **NM34.** The bag sample is 756 rows and is drawn with replacement. Sampling with replacement was established in the Applied Statistics topic of the Statistical Analysis and Exploratory Data Analysis course; its method is not repeated. - **NM35.** The decision is a majority vote: if the share of trees that say positive exceeds 0.5, the subscriber is counted as suspicious. The threshold is fixed from the Data Preparation and Feature Engineering course; its selection is left to model evaluation. - **NM36.** The number of candidates tried is 31: seven tree counts and twenty-four column-subset–depth pairs. The choice is read from the **validation set**; the test set appears only in the last line. #### The Spread of a Single Tree The setup generates the table, bins the columns by thresholds and prints the single tree that is the comparison point. ```python # bagging.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the feature set comes from the Data Preparation and Feature Engineering course and is kept fixed throughout the course. import bisect import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] THRESHOLDS = {} for a in FIELDS: d = sorted({x[a] for x in TRAIN}) THRESHOLDS[a] = d[1:] if len(d) <= 24 else [d[int(i * len(d) / 25)] for i in range(1, 25)] BINS = [len(THRESHOLDS[a]) + 1 for a in FIELDS] def binarize(s): # each row: bin numbers and label return [(tuple(bisect.bisect_right(THRESHOLDS[a], x[a]) for a in FIELDS), x["suspicious"]) for x in s] B_TRAIN, B_VAL, B_TEST = binarize(TRAIN), binarize(VAL), binarize(TEST) ALL = list(range(len(FIELDS))) def tree(sample, depth, columns, least=2): # MODEL: decision tree n, p, best = len(sample), sum(y for _, y in sample), None if depth and n >= 2 * least and 0 < p < n: for k in columns: count, pos = [0] * BINS[k], [0] * BINS[k] for b, y in sample: count[b[k]] += 1 pos[b[k]] += y sc = sp = 0 for j in range(BINS[k] - 1): sc, sp = sc + count[j], sp + pos[j] if sc < least or n - sc < least: continue gain = sp * sp / sc + (p - sp) ** 2 / (n - sc) - p * p / n if best is None or gain > best[0]: best = (gain, k, j) if best is None or best[0] <= 1e-9: return p / n _, k, j = best return (k, j, tree([o for o in sample if o[0][k] <= j], depth - 1, columns, least), tree([o for o in sample if o[0][k] > j], depth - 1, columns, least)) def ratio(d, b): while type(d) is tuple: d = d[2] if b[d[0]] <= d[1] else d[3] return d def accuracy(d, target): return sum((ratio(d, b) > 0.5) == y for b, y in target) / len(target) def baseline(target): return sum(y == 0 for _, y in target) / len(target) SINGLE = tree(B_TRAIN, 6, ALL) print(f"baseline validation {baseline(B_VAL):.4f} test {baseline(B_TEST):.4f}") print(f"single tree depth 6 validation {accuracy(SINGLE, B_VAL):.4f} " f"test {accuracy(SINGLE, B_TEST):.4f} train {accuracy(SINGLE, B_TRAIN):.4f}") ``` ``` baseline validation 0.8135 test 0.7579 single tree depth 6 validation 0.8056 test 0.7778 train 0.8426 ``` Two held-out sets give two different baselines with the same procedure: 0.7579 on the test set, 0.8135 on validation. The tree reads 0.7778 on the test set, beating the baseline by 0.0199, and stays at 0.8056 on validation, falling 0.0079 below its own baseline. The fragility is counted as follows: 41 samples are drawn from the training set with replacement and a tree is built on each with the same procedure. The only thing that changes is which row appears how many times. ```python def spread(v): o = sum(v) / len(v) return (sum((q - o) ** 2 for q in v) / len(v)) ** 0.5 def forest(seed, count, m, depth): # MODEL: bagged tree ensemble r, trees, bags = rng(seed), [], [] for _ in range(count): chosen = [int(r() * len(B_TRAIN)) for _ in range(len(B_TRAIN))] pool, columns = list(ALL), [] for _ in range(min(m, len(ALL))): columns.append(pool.pop(int(r() * len(pool)))) trees.append(tree([B_TRAIN[i] for i in chosen], depth, sorted(columns))) bags.append(set(chosen)) return trees, bags def vote(trees, b): # majority vote: share of trees that say positive return sum(ratio(d, b) > 0.5 for d in trees) / len(trees) def forest_accuracy(trees, target): return sum((vote(trees, b) > 0.5) == y for b, y in target) / len(target) FORESTS = [forest(SEED + 13 * i, 41, 11, 6) for i in range(5)] TREES = FORESTS[0][0] roots = {} for d in TREES: roots[(d[0], d[1])] = roots.get((d[0], d[1]), 0) + 1 labels = [[ratio(d, b) > 0.5 for b, _ in B_VAL] for d in TREES] differ = [sum(u != v for u, v in zip(labels[i], labels[j])) for i in range(len(TREES)) for j in range(i + 1, len(TREES))] print(f"{len(roots)} distinct (column, threshold) pairs for the root split, " f"the most frequent in {max(roots.values())} trees") print(f"{'single tree':<12} {'mean':>9} {'spread':>9} {'lowest':>9} {'highest':>10}") for name, target in (("validation", B_VAL), ("test", B_TEST)): v = [accuracy(d, target) for d in TREES] print(f"{' ' + name:<12} {sum(v) / len(v):>9.4f} {spread(v):>9.4f} " f"{min(v):>9.4f} {max(v):>10.4f}") print(f"subscribers two trees classify differently (out of 252): mean " f"{sum(differ) / len(differ):.1f}, at least {min(differ)}, at most {max(differ)}") print(f"vote of the same 41 trees validation {forest_accuracy(TREES, B_VAL):.4f} " f"test {forest_accuracy(TREES, B_TEST):.4f}") ``` ``` 9 distinct (column, threshold) pairs for the root split, the most frequent in 16 trees single tree mean spread lowest highest validation 0.7744 0.0218 0.7222 0.8135 test 0.7489 0.0246 0.6905 0.7897 subscribers two trees classify differently (out of 252): mean 46.2, at least 24, at most 77 vote of the same 41 trees validation 0.8095 test 0.7698 ``` The root split is not the same in all 41 trees: nine different column-threshold pairs become the root, and the most frequent appears in only 16 trees. The split that determines everything beneath it moves somewhere else in 60% of the trees when the sample changes. The second number is more concrete: two random trees classify on average 46.2 of the 252 subscribers differently, 24 in the closest pair and 77 in the farthest pair. The third is how this reflects on the held-out set. The single trees' test number has a mean of 0.7489, a spread of 0.0246, and a range from 0.6905 to 0.7897; the distance between the two ends is 25 subscribers. Reading a model's number from a single sample is picking a random point from this range. **Spread** is the square root of the mean of the squared distances from the mean. The last line is the vote of the same 41 trees: validation 0.8095, test 0.7698. This number is 0.0209 above the single trees' mean of 0.7489, but **below** the single tree's 0.7778: with all the columns open to every tree, bagging pulls the mean up but does not beat the single tree. #### Number of Trees and Variability What bagging really adds is not in the mean but in the spread. Five ensembles are built with five seeds, and the first 1, 3, 5, 11, 21, 31 and 41 trees of each are voted separately. ```python print(f"{'trees':>5} {'validation':>10} {'spread':>9} {'test':>9} {'spread':>9}") for count in (1, 3, 5, 11, 21, 31, 41): vv = [forest_accuracy(a[:count], B_VAL) for a, _ in FORESTS] vt = [forest_accuracy(a[:count], B_TEST) for a, _ in FORESTS] print(f"{count:>5} {sum(vv) / 5:>10.4f} {spread(vv):>9.4f} " f"{sum(vt) / 5:>9.4f} {spread(vt):>9.4f}") ``` ``` trees validation spread test spread 1 0.7786 0.0193 0.7627 0.0088 3 0.7944 0.0136 0.7762 0.0102 5 0.8032 0.0096 0.7778 0.0148 11 0.7984 0.0095 0.7722 0.0122 21 0.8103 0.0105 0.7762 0.0082 31 0.8183 0.0119 0.7754 0.0124 41 0.8214 0.0075 0.7778 0.0066 ``` The spread columns are the lesson's measure. For a single tree the validation spread is 0.0193, for 41 trees 0.0075: the difference among the five runs narrows 2.6-fold. The mean rises from 0.7786 to 0.8214 but not smoothly; 0.8032 at 5 trees, 0.7984 at 11. Adding trees does not give a better number at every step; what it gives is a number that moves less on repetition. On the test set it is the same: from 0.7627 to 0.7778, a gain of four subscribers, and the spread from 0.0088 to 0.0066. The cost is 41 times the computation. #### Feature Randomness and the Out-of-Bag Estimate Bagged trees resemble one another, because all of them look at the same eleven columns and find the same strong column at the root. A second source of randomness breaks this: each tree chooses its splits from a randomly chosen subset of `m` columns. The grid takes `m` from two to eleven and the depth from three to eight. A third number is also produced: the **out-of-bag** estimate has each training row voted on only by the trees that did not take that row into their bag, and gives a number without touching the held-out set. ```python def out_of_bag(trees, bags): # each row is voted on by the trees that did not see it out = [(b, y, [d for d, c in zip(trees, bags) if i not in c]) for i, (b, y) in enumerate(B_TRAIN)] return sum((vote(v, b) > 0.5) == y for b, y, v in out if v) / sum(1 for *_, v in out if v) CANDIDATES, GRID = 7, {} print(f"{'m':>3}" + "".join(f"{'depth ' + str(d):>16}" for d in (3, 4, 6, 8))) for m in (2, 3, 4, 6, 8, 11): line = [] for depth in (3, 4, 6, 8): a, t = forest(SEED, 41, m, depth) GRID[(m, depth)] = (forest_accuracy(a, B_VAL), forest_accuracy(a, B_TEST), out_of_bag(a, t)) CANDIDATES += 1 line.append(f"{GRID[(m, depth)][0]:.4f}/{GRID[(m, depth)][1]:.4f}") print(f"{m:>3}" + "".join(f"{q:>16}" for q in line)) ORDER = [(m, d) for m in (2, 3, 4, 6, 8, 11) for d in (3, 4, 6, 8)] def top(column): e = max(GRID[k][column] for k in ORDER) return next(k for k in ORDER if GRID[k][column] == e) for name, column in (("from validation", 0), ("from test", 1), ("from out-of-bag", 2)): m, depth = top(column) d, s, o = GRID[(m, depth)] print(f"choice {name:<15} m={m:<3} depth={depth:<3} " f"validation {d:.4f} test {s:.4f} out-of-bag {o:.4f}") m, depth = top(0) print(f"candidates tried {CANDIDATES}, family number {GRID[(m, depth)][1]:.4f}, " f"above baseline {GRID[(m, depth)][1] - baseline(B_TEST):+.4f}") ``` ``` m depth 3 depth 4 depth 6 depth 8 2 0.8135/0.7579 0.8135/0.7579 0.8135/0.7579 0.8135/0.7579 3 0.8095/0.7540 0.8135/0.7579 0.8175/0.7619 0.8095/0.7579 4 0.8175/0.7698 0.8214/0.7817 0.8175/0.7817 0.8254/0.7817 6 0.8214/0.7778 0.8413/0.7897 0.8373/0.7937 0.8413/0.7857 8 0.8254/0.7857 0.8214/0.7817 0.8254/0.7698 0.8294/0.7738 11 0.8175/0.7857 0.8214/0.7778 0.8095/0.7698 0.8016/0.7698 choice from validation m=6 depth=4 validation 0.8413 test 0.7897 out-of-bag 0.7884 choice from test m=6 depth=6 validation 0.8373 test 0.7937 out-of-bag 0.7950 choice from out-of-bag m=6 depth=8 validation 0.8413 test 0.7857 out-of-bag 0.7989 candidates tried 31, family number 0.7897, above baseline +0.0317 ``` The top row shows the limit of randomness: when `m` is two, the ensemble gives exactly the baseline at all four depths, because the vote of two-column trees moves no subscriber to the suspicious side. When `m` is six, 0.8413 on validation and 0.7897 on the test set come out; at `m` eleven, that is, with no randomness, the number falls back to 0.7778. The difference randomness adds is this column, three to four subscribers. The last three lines pay off the course's rule. The choice read from validation says `m` six and depth four; that candidate's test number is 0.7897. Had it been read from the test set, depth six would be chosen and the number would be 0.7937; the difference is 0.0040, a single subscriber. Small, but its direction is fixed: if the choice and the measurement are made from the same 252 rows, the number grows. The out-of-bag column is sharper. The chosen candidate's validation number is 0.8413, its test number 0.7897 and its out-of-bag estimate 0.7884: 0.0013 from the test set and 0.0516 from validation. The out-of-bag is read from 756 rows and spends no held-out set; validation is read from 252 rows and is used up with every look. The family number is 0.7897, the candidates tried 31; 0.0317 above the baseline. The real difference is here: the single tree's number was a draw between 0.6905 and 0.7897, the ensemble's is not. #### Summary - The same procedure chooses nine different column-threshold pairs for the root across 41 bag samples; two trees classify on average 46.2 of the 252 subscribers differently. - The single tree's test number wanders between 0.6905 and 0.7897 with a spread of 0.0246; a number read from a single sample is a randomly chosen point of this range. - As the number of trees goes from 1 to 41, the validation spread goes from 0.0193 to 0.0075 and the mean from 0.7786 to 0.8214; the rise is not smooth, the fall is. - Without feature randomness, bagging does not beat the single tree: with eleven columns open, the test number is 0.7698 against the single tree's 0.7778; when `m` is six it is 0.7897. - The out-of-bag estimate is 0.7884, 0.0013 from the test set and 0.0516 from validation; it is read without spending a held-out set. #### Next Step Bagging builds the trees **independently of one another**: not one of the 41 trees knows where the others err, all of them solve the same problem from scratch and vote at the end. Independence lowered the spread and moved the mean by only four subscribers. What if the trees could see one another: if the second tree looked only where the first erred, and the third at the remaining error of the first two. The next lesson builds this sequential arrangement and sets two tuning variables against each other: how many rounds to run and with what size of step each round is added. The two substitute for each other; the same number can come out with many rounds of a small step or few rounds of a large step, and the budgets of the two paths are not the same. ### Sequential Boosting Source: https://academia.sh/en/courses/supervised-learning/sequential-boosting Measuring what happens when trees are built on the remaining error instead of independently of one another: over two hundred rounds the training number climbs from 0.7778 to 0.8638, while the validation number peaks at 0.8373 in round ten and drops to 0.8056. The learning rate and the number of rounds substitute for each other; the same target is reached in 79 rounds with a rate of 0.02, in 30 with 0.05, in 10 with 0.15 and in 4 with 0.45, and the product of rate and rounds stays between 1.50 and 1.80. On the seventy-two-cell grid, the candidate chosen from validation gives 0.7738 on the test set and the one chosen from the test set gives 0.8056: a difference of 0.0318. In total 1,279 candidates were tried, and boosting with 30 splits fell 0.0159 behind bagging with 568 splits. The previous lesson built forty-one trees independently of one another; none of them knew where the others erred. Independence lowered the spread from 0.0193 to 0.0075 but moved the test mean up by only four subscribers. This lesson tries the opposite direction: each new model is built by looking at the **remaining error** of the previous ones. **Sequential boosting** builds a sum. The start is the logit of the suspicious rate in the training set; each round fits a **weak learner** to the difference the sum leaves at each subscriber and adds its contribution to the sum multiplied by a **learning rate**. The weak learner here is a small tree with one or two splits. Three things are measured: the separation of two numbers as the number of rounds grows, the substitution of the learning rate and the number of rounds for each other, and the bill these write to the tuning budget. - **NM37.** The set, the split and the baseline are the same as in the previous lesson: 1,260 subscribers, 756/252/252, seed 20260218, the baseline is 0.7579 on the test set and 0.8135 on validation. The features do not change. - **NM38.** In this lesson the tree is a **regression** tree: each example carries a gradient and a curvature, and the value of a leaf is the ratio of the two. Giving the label as the gradient and one as the curvature turns the same function into the previous lesson's classification tree. - **NM39.** The gradient of a round is taken as `label - probability` and the curvature as `probability * (1 - probability)`, and the curvature does not fall below `1e-6`; this is an overflow guard, not a tuning choice. - **NM40.** The decision threshold is the zero of the sum, that is, 0.5 of the probability; it is fixed from the Data Preparation and Feature Engineering course. - **NM41.** The depth of the weak learner is 1, 2 or 3; the learning rate is 0.02 / 0.05 / 0.15 / 0.45; the number of rounds is at most 300. - **NM42.** The target of the trade-off measurement is 208/252 on the validation set, that is, 0.8254, and it was fixed before the measurement. - **NM43.** The cost measure is the **number of splits**: the total of the internal nodes. Time is not written. - **NM44.** The tuning budget counts every `(rate, depth, rounds)` triple measured on the validation set; the choice is made only from the seventy-two-cell grid. #### Building on the Remaining Error The setup generates the table, writes the tree and the boosting loop, and sweeps the number of rounds with a fixed setting. ```python # boosting.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the feature set comes from the Data Preparation and Feature Engineering course and is kept fixed throughout the course. import bisect import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] THRESHOLDS = {} for a in FIELDS: d = sorted({x[a] for x in TRAIN}) THRESHOLDS[a] = d[1:] if len(d) <= 24 else [d[int(i * len(d) / 25)] for i in range(1, 25)] BINS = [len(THRESHOLDS[a]) + 1 for a in FIELDS] def binarize(s): # each row: bin numbers and label return [(tuple(bisect.bisect_right(THRESHOLDS[a], x[a]) for a in FIELDS), x["suspicious"]) for x in s] B_TRAIN, B_VAL, B_TEST = binarize(TRAIN), binarize(VAL), binarize(TEST) SETS, ALL = (B_TRAIN, B_VAL, B_TEST), list(range(len(FIELDS))) def tree(sample, depth, columns, least=2): # MODEL: each example is (bins, gradient, curvature) n, best = len(sample), None G, H = sum(o[1] for o in sample), sum(o[2] for o in sample) if depth and n >= 2 * least: for k in columns: sg, sh, sn = [0.0] * BINS[k], [0.0] * BINS[k], [0] * BINS[k] for b, g, h in sample: sg[b[k]] += g sh[b[k]] += h sn[b[k]] += 1 cg = ch = cn = 0 for j in range(BINS[k] - 1): cg, ch, cn = cg + sg[j], ch + sh[j], cn + sn[j] if cn < least or n - cn < least: continue gain = cg * cg / ch + (G - cg) ** 2 / (H - ch) - G * G / H if best is None or gain > best[0]: best = (gain, k, j) if best is None or best[0] <= 1e-9: return G / H _, k, j = best return (k, j, tree([o for o in sample if o[0][k] <= j], depth - 1, columns, least), tree([o for o in sample if o[0][k] > j], depth - 1, columns, least)) def value(d, b): while type(d) is tuple: d = d[2] if b[d[0]] <= d[1] else d[3] return d def squash(z): return 1.0 / (1.0 + math.exp(-z)) if z > -30 else 0.0 def boost(rate, rounds, depth, stops): # MODEL: sequential boosting p = sum(y for _, y in B_TRAIN) / len(B_TRAIN) F = [[math.log(p / (1 - p))] * len(k) for k in SETS] trace = {} for t in range(1, rounds + 1): sample = [] for i, (b, y) in enumerate(B_TRAIN): q = squash(F[0][i]) sample.append((b, y - q, max(q * (1 - q), 1e-6))) d = tree(sample, depth, ALL) for u, subset in enumerate(SETS): for i, (b, _) in enumerate(subset): F[u][i] += rate * value(d, b) if t in stops: trace[t] = tuple(sum((F[u][i] > 0) == y for i, (_, y) in enumerate(subset)) / len(subset) for u, subset in enumerate(SETS)) return trace def baseline(subset): return sum(y == 0 for _, y in subset) / len(subset) STOPS = (1, 5, 10, 25, 50, 100, 200) TRACE = boost(0.15, 200, 2, set(STOPS)) print(f"baseline validation {baseline(B_VAL):.4f} test {baseline(B_TEST):.4f}") print(f"{'rounds':>6} {'train':>9} {'validation':>10} {'test':>9}") for t in STOPS: print(f"{t:>6} {TRACE[t][0]:>9.4f} {TRACE[t][1]:>10.4f} {TRACE[t][2]:>9.4f}") ``` ``` baseline validation 0.8135 test 0.7579 rounds train validation test 1 0.7778 0.8135 0.7579 5 0.7804 0.8095 0.7579 10 0.8082 0.8373 0.7738 25 0.8280 0.8214 0.7937 50 0.8439 0.8135 0.7897 100 0.8532 0.8016 0.7778 200 0.8638 0.8056 0.7817 ``` The table is a separation. In the first round the training number is 0.7778 and validation 0.8135; the held-out set is **ahead**. In round ten validation peaks at 0.8373. After round twenty-five the directions part: the training number climbs without stopping to 0.8638 while validation falls to 0.8056. The gap between them is 0.0357 in the first round, changes sign in round two hundred and becomes 0.0582. The source is visible: each round writes a larger gradient to the subscribers who remain on the wrong side, and the new weak learner is built on them; after a hundred rounds the sum is chasing individual subscribers. The number of rounds is a capacity knob and **its limit is read from the held-out set**: what comes after round ten is a gain for training and a loss for validation. #### The Trade-off Between Learning Rate and Number of Rounds The second knob is the learning rate. In how many rounds the same target is reached is counted separately for four rates; in this section the weak learner is a one-split tree. ```python TARGET, ROUNDS = 208 / 252, 300 print(f"{'rate':>6} {'to target':>11} {'best round':>11} {'best val':>11} " f"{'test there':>12} {'train at 300':>13}") TRADEOFF = {} for rate in (0.02, 0.05, 0.15, 0.45): trace = boost(rate, ROUNDS, 1, set(range(1, ROUNDS + 1))) first = next((t for t in range(1, ROUNDS + 1) if trace[t][1] >= TARGET), 0) best = max(range(1, ROUNDS + 1), key=lambda t: trace[t][1]) TRADEOFF[rate] = (first, best, trace[best][1], trace[best][2]) print(f"{rate:>6.2f} {first:>11} {best:>11} {trace[best][1]:>11.4f} " f"{trace[best][2]:>12.4f} {trace[ROUNDS][0]:>13.4f}") print(f"target {TARGET:.4f} (208 of 252 subscribers), weak learner is a one-split tree") ``` ``` rate to target best round best val test there train at 300 0.02 79 204 0.8294 0.7897 0.8135 0.05 30 84 0.8333 0.8016 0.8148 0.15 10 36 0.8333 0.7976 0.8280 0.45 4 8 0.8294 0.7976 0.8307 target 0.8254 (208 of 252 subscribers), weak learner is a one-split tree ``` The second column is the lesson's measure. The 208 subscribers in the validation set are reached in 79 rounds with a rate of 0.02, in 30 with 0.05, in 10 with 0.15 and in 4 with 0.45. The product of rate and rounds is 1.58, 1.50, 1.50 and 1.80 respectively: about the same in all four. The two tuning variables are two faces of one knob and must be read **together**; "two hundred rounds" alone is not a tuning recipe. Even though the product is constant, the result is not. The best validation number is 0.8333 at the middle two rates and 0.8294 at the ends; the test numbers of the same candidates wander between 0.7897 and 0.8016. The training number at round three hundred is 0.8135 at rate 0.02 and 0.8307 at 0.45: a large rate goes to the same place faster and memorizes more. #### Budget, Cost and the Difference from Bagging The last section combines the two knobs and the number of rounds in a single grid, reads the choice from validation, then compares the cost of boosting with the bagged ensemble of the previous lesson. ```python def split_count(d): return 0 if type(d) is not tuple else 1 + split_count(d[2]) + split_count(d[3]) def forest(seed, count, m, depth): # the bagged ensemble of the previous lesson r, trees = rng(seed), [] for _ in range(count): chosen = [int(r() * len(B_TRAIN)) for _ in range(len(B_TRAIN))] pool, columns = list(ALL), [] for _ in range(min(m, len(ALL))): columns.append(pool.pop(int(r() * len(pool)))) trees.append(tree([(B_TRAIN[i][0], B_TRAIN[i][1], 1.0) for i in chosen], depth, sorted(columns))) return trees FOREST = forest(SEED, 41, 6, 4) GRID, STOPS3 = {}, (5, 10, 25, 50, 100, 200) print(f"{'rate':>6} {'depth':>9}" + "".join(f"{'round ' + str(t):>14}" for t in STOPS3)) for rate in (0.02, 0.05, 0.15, 0.45): for depth in (1, 2, 3): trace = boost(rate, 200, depth, set(STOPS3)) for t in STOPS3: GRID[(rate, depth, t)] = trace[t][1:] print(f"{rate:>6.2f} {depth:>9}" + "".join(f"{trace[t][1]:>7.4f}/{trace[t][2]:.4f}" for t in STOPS3)) ORDER = [(a, d, t) for a in (0.02, 0.05, 0.15, 0.45) for d in (1, 2, 3) for t in STOPS3] for name, column in (("from validation", 0), ("from test", 1)): e = max(GRID[k][column] for k in ORDER) a, d, t = next(k for k in ORDER if GRID[k][column] == e) print(f"choice {name:<15} rate={a:<5} depth={d} rounds={t:<4} " f"validation {GRID[(a, d, t)][0]:.4f} test {GRID[(a, d, t)][1]:.4f}") e = max(GRID[k][0] for k in ORDER) a, d, t = next(k for k in ORDER if GRID[k][0] == e) print(f"boosting {t} rounds x depth {d} = {t * (2 ** d - 1)} splits, " f"test {GRID[(a, d, t)][1]:.4f}") print(f"bagging 41 trees x depth 4 = {sum(map(split_count, FOREST))} splits, " f"test {sum((sum(value(x, b) > 0.5 for x in FOREST) * 2 > 41) == y for b, y in B_TEST) / len(B_TEST):.4f}") print(f"candidates tried {7 + 4 * ROUNDS + len(ORDER)}, family number {GRID[(a, d, t)][1]:.4f}, " f"above baseline {GRID[(a, d, t)][1] - baseline(B_TEST):+.4f}") ``` ``` rate depth round 5 round 10 round 25 round 50 round 100 round 200 0.02 1 0.8135/0.7579 0.8135/0.7579 0.8135/0.7579 0.8135/0.7579 0.8254/0.7817 0.8254/0.7857 0.02 2 0.8135/0.7579 0.8135/0.7579 0.8135/0.7579 0.8135/0.7659 0.8294/0.7817 0.8214/0.7937 0.02 3 0.8135/0.7579 0.8135/0.7579 0.8135/0.7579 0.8214/0.7857 0.8135/0.7778 0.8254/0.7817 0.05 1 0.8135/0.7579 0.8135/0.7579 0.8135/0.7579 0.8175/0.7857 0.8294/0.7976 0.8294/0.7937 0.05 2 0.8135/0.7579 0.8135/0.7579 0.8294/0.7738 0.8175/0.7897 0.8294/0.7937 0.8175/0.7857 0.05 3 0.8135/0.7579 0.8135/0.7579 0.8175/0.7817 0.8254/0.7778 0.8254/0.7817 0.8095/0.7857 0.15 1 0.8135/0.7579 0.8254/0.7817 0.8254/0.7897 0.8333/0.7937 0.8175/0.8016 0.8135/0.8056 0.15 2 0.8095/0.7579 0.8373/0.7738 0.8214/0.7937 0.8135/0.7897 0.8016/0.7778 0.8056/0.7817 0.15 3 0.8254/0.7817 0.8056/0.7817 0.8095/0.7817 0.8056/0.7778 0.8016/0.7778 0.8016/0.7817 0.45 1 0.8254/0.7857 0.8294/0.7897 0.8135/0.7897 0.8175/0.7937 0.8175/0.8016 0.8214/0.7897 0.45 2 0.8135/0.7857 0.8214/0.7778 0.8254/0.7778 0.7976/0.7698 0.7857/0.7659 0.7976/0.7659 0.45 3 0.8175/0.7778 0.7937/0.7738 0.7817/0.7540 0.7897/0.7659 0.7698/0.7619 0.7778/0.7738 choice from validation rate=0.15 depth=2 rounds=10 validation 0.8373 test 0.7738 choice from test rate=0.15 depth=1 rounds=200 validation 0.8135 test 0.8056 boosting 10 rounds x depth 2 = 30 splits, test 0.7738 bagging 41 trees x depth 4 = 568 splits, test 0.7897 candidates tried 1279, family number 0.7738, above baseline +0.0159 ``` The highest validation number the grid gives is 0.8373, and in a single cell: rate 0.15, depth 2, round 10. That candidate's test number is **0.7738**. Had it been read from the test set, rate 0.15, depth 1, round 200 would have been chosen and the number would be 0.8056. The difference is 0.0318, that is, eight subscribers. In the previous lesson the same difference was 0.0040, one subscriber. The cause is the grid itself. In this lesson the number of triples measured on the validation set is 1,279; in the previous lesson it was 31. When this many candidates are tried on a set of 252 rows, the highest cell is the cell not of the best setting but of the **luckiest setting**. The course's rule finds its full counterpart here: a model's number cannot be read without writing how many candidates were tried to find that number. The last two lines give the difference between the families as a number. Boosting chosen from validation builds 30 splits and gives 0.7738; the previous lesson's bagged ensemble builds 568 splits and gives 0.7897. Boosting is nineteen times cheaper and four subscribers behind. The difference changing the family adds is −0.0159; the difference the budget inflates within the same grid is +0.0318. **The budget's difference is twice the family's difference.** The family's number is 0.7738, 0.0159 above the baseline. It is also below the single tree's 0.7778 and is not corrected: this is the grid's best validation cell. #### Summary - Over two hundred rounds the training number climbs from 0.7778 to 0.8638 while validation peaks at 0.8373 and falls to 0.8056; the gap between the two numbers changes sign and becomes 0.0582. - The same validation target is reached in 79 rounds with a rate of 0.02, in 30 with 0.05, in 10 with 0.15 and in 4 with 0.45; the product of rate and rounds stays between 1.50 and 1.80. - Even though the product is constant, the result is not: the best validation number of the end rates is 0.8294, of the middle ones 0.8333; a large rate memorizes training up to 0.8307 at round three hundred. - The choice read from validation gives 0.7738 on the test set, the one read from the test set 0.8056; the difference is 0.0318 and 1,279 candidates were tried. In the previous lesson the same difference with 31 candidates was 0.0040. - Boosting gives 0.7738 with 30 splits, bagging 0.7897 with 568 splits: the family difference is −0.0159, the budget difference +0.0318. #### Next Step Two combinations were tried and both were fed from the same family, the decision tree: bagging had independent trees vote, boosting stacked them into a sum. There is a third form: a second layer that takes the outputs of models from **different families** as its input. This immediately raises a question: from which set will the second layer be learned. If it is learned from the first layer's outputs on the training set, the layer sees memorized outputs and the number inflates. The next lesson puts the three combinations side by side, counts the inflation and prints the number of subscribers on whom the ensembles err **together**. ### Ensemble Methods Source: https://academia.sh/en/courses/supervised-learning/ensemble-methods Counting side by side what bagging, boosting and stacking add and what they cost: the three base models give 0.7778, 0.7897 and 0.7738 on the test set at costs of 43, 568 and 30 splits. On 45 of the 252 subscribers all three are wrong, bagging and boosting classify only 6 subscribers differently, and no combination that sees nothing but the three labels can pass 0.7897. When the second layer of stacking is learned from the validation set it reports 0.8254 and gives 0.7778; when it is learned from the training set it reports 0.8532 and again gives 0.7778: the inflation goes from 0.0476 to 0.0754 and the weight shifts to the model that memorizes most. The topic's total tuning budget is 1,313 candidates. The previous two lessons multiplied the same family in two different ways. Bagging built the trees in parallel and had them vote and gave 0.7897 on the test set; boosting stacked them into a sequential sum and gave 0.7738. Both were multiplying a single type of learner. The third form is to build **a second layer that takes the outputs of different models as its input**, and it immediately raises a question: from which set will that layer be learned. This lesson puts the three combinations side by side. The measure is three numbers: the difference on the held-out set, the cost as the **number of splits**, and the candidates tried. A fourth measure is added, the **similar errors** of the ensembles: if three models err on the same subscribers, the combination has nothing to gain, and this is not an intuition but a ceiling that can be computed. - **NM45.** The set, the split, the baseline and the features are the same as in the previous two lessons: 1,260 subscribers, 756/252/252, seed 20260218, the baseline is 0.7579 on the test set and 0.8135 on validation. - **NM46.** The three base models are carried over **as they are** from the previous lessons and are not retuned here: a depth-6 single tree, a bagged ensemble of 41 trees (`m` six, depth 4) and boosting with rate 0.15 / depth 2 / 10 rounds. - **NM47.** The output of each base model is in the range [0, 1]: the leaf ratio, the vote share, the boosting probability. The decision threshold is 0.5 and is fixed from the Data Preparation and Feature Engineering course. - **NM48.** The second layer of stacking (the meta-learner) is a gradient-based linear model that takes the three outputs as input: three weights and a constant, 400 iterations, rate 2.0. Its settings were fixed before the measurement. - **NM49.** The cost measure is the **number of splits**; the cost of stacking is the splits of the three base models plus the four learned numbers. - **NM50.** The shared-error count is done on the test set and at the **label level**. - **NM51.** The **ceiling** of a combination that looks at the label pattern is the accuracy of the rule that, in each pattern, says that pattern's majority label; no combination that sees nothing but the three labels can pass it. - **NM52.** No new tuning is searched in this lesson; the candidates tried are three. The topic's total budget, with the carried numbers, is 1,313 candidates. #### Three Combinations and Three Base Models The setup generates the table, rebuilds two of the combinations and prints the numbers of the three base models together with their cost. ```python # stacking.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the feature set comes from the Data Preparation and Feature Engineering course and is kept fixed throughout the course. import bisect import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] THRESHOLDS = {} for a in FIELDS: d = sorted({x[a] for x in TRAIN}) THRESHOLDS[a] = d[1:] if len(d) <= 24 else [d[int(i * len(d) / 25)] for i in range(1, 25)] BINS = [len(THRESHOLDS[a]) + 1 for a in FIELDS] def binarize(s): # each row: bin numbers and label return [(tuple(bisect.bisect_right(THRESHOLDS[a], x[a]) for a in FIELDS), x["suspicious"]) for x in s] B_TRAIN, B_VAL, B_TEST = binarize(TRAIN), binarize(VAL), binarize(TEST) SETS, ALL = (B_TRAIN, B_VAL, B_TEST), list(range(len(FIELDS))) def tree(sample, depth, columns, least=2): # MODEL: each example is (bins, gradient, curvature) n, best = len(sample), None G, H = sum(o[1] for o in sample), sum(o[2] for o in sample) if depth and n >= 2 * least: for k in columns: sg, sh, sn = [0.0] * BINS[k], [0.0] * BINS[k], [0] * BINS[k] for b, g, h in sample: sg[b[k]] += g sh[b[k]] += h sn[b[k]] += 1 cg = ch = cn = 0 for j in range(BINS[k] - 1): cg, ch, cn = cg + sg[j], ch + sh[j], cn + sn[j] if cn < least or n - cn < least: continue gain = cg * cg / ch + (G - cg) ** 2 / (H - ch) - G * G / H if best is None or gain > best[0]: best = (gain, k, j) if best is None or best[0] <= 1e-9: return G / H _, k, j = best return (k, j, tree([o for o in sample if o[0][k] <= j], depth - 1, columns, least), tree([o for o in sample if o[0][k] > j], depth - 1, columns, least)) def value(d, b): while type(d) is tuple: d = d[2] if b[d[0]] <= d[1] else d[3] return d def squash(z): return 1.0 / (1.0 + math.exp(-z)) if z > -30 else 0.0 def split_count(d): return 0 if type(d) is not tuple else 1 + split_count(d[2]) + split_count(d[3]) def forest(seed, count, m, depth): # COMBINATION 1: bagging, parallel vote r, trees = rng(seed), [] for _ in range(count): chosen = [int(r() * len(B_TRAIN)) for _ in range(len(B_TRAIN))] pool, columns = list(ALL), [] for _ in range(min(m, len(ALL))): columns.append(pool.pop(int(r() * len(pool)))) trees.append(tree([(B_TRAIN[i][0], B_TRAIN[i][1], 1.0) for i in chosen], depth, sorted(columns))) return trees def boost(rate, rounds, depth): # COMBINATION 2: boosting, sequential sum p = sum(y for _, y in B_TRAIN) / len(B_TRAIN) F, trees = [[math.log(p / (1 - p))] * len(k) for k in SETS], [] for _ in range(rounds): q = [squash(z) for z in F[0]] d = tree([(b, y - q[i], max(q[i] * (1 - q[i]), 1e-6)) for i, (b, y) in enumerate(B_TRAIN)], depth, ALL) trees.append(d) for u, subset in enumerate(SETS): for i, (b, _) in enumerate(subset): F[u][i] += rate * value(d, b) return trees, [[squash(z) for z in f] for f in F] SINGLE = tree([(b, y, 1.0) for b, y in B_TRAIN], 6, ALL) FOREST = forest(SEED, 41, 6, 4) BOOSTED, BOOSTED_P = boost(0.15, 10, 2) BASE = [("single tree", [[value(SINGLE, b) for b, _ in k] for k in SETS], split_count(SINGLE)), ("bagging", [[sum(value(x, b) > 0.5 for x in FOREST) / 41 for b, _ in k] for k in SETS], sum(map(split_count, FOREST))), ("boosting", BOOSTED_P, sum(map(split_count, BOOSTED)))] def accuracy(output, subset): return sum((c > 0.5) == y for c, (_, y) in zip(output, subset)) / len(subset) print(f"{'model':<12} {'validation':>10} {'test':>8} {'splits':>7}") print(f"{'baseline':<12} {sum(y == 0 for _, y in B_VAL) / len(B_VAL):>10.4f} " f"{sum(y == 0 for _, y in B_TEST) / len(B_TEST):>8.4f} {0:>7}") for name, output, cost in BASE: print(f"{name:<12} {accuracy(output[1], B_VAL):>10.4f} " f"{accuracy(output[2], B_TEST):>8.4f} {cost:>7}") ``` ``` model validation test splits baseline 0.8135 0.7579 0 single tree 0.8056 0.7778 43 bagging 0.8413 0.7897 568 boosting 0.8373 0.7738 30 ``` The three rows show three different trade-offs. Bagging gives the highest number, 0.7897, and is the most expensive: 568 splits. Boosting gives 0.7738 with 30 splits, that is, nineteen times cheaper. The single tree stands at 0.7778 with 43 splits and is **above boosting**. The real question is this: how differently do these three models err. ```python def labels(u): return [tuple(c[u][i] > 0.5 for _, c, _ in BASE) for i in range(len(SETS[u]))] E = labels(2) ERRORS = [sum(e[j] != y for j in range(3)) for e, (_, y) in zip(E, B_TEST)] PATTERNS = {} for e, (_, y) in zip(E, B_TEST): PATTERNS.setdefault(e, [0, 0])[y] += 1 print("models wrong on test " + " ".join(f"{k}: {ERRORS.count(k):>3}" for k in range(4))) print(f"ceiling of a combination that sees the label pattern " f"{sum(max(v) for v in PATTERNS.values()) / len(B_TEST):.4f}, distinct patterns {len(PATTERNS)}") for i in range(3): for j in range(i + 1, 3): differ = sum(e[i] != e[j] for e in E) both = sum(e[i] != y and e[j] != y for e, (_, y) in zip(E, B_TEST)) print(f"{BASE[i][0]:>11} - {BASE[j][0]:<11} classified differently {differ:>3}, " f"both wrong {both:>3}") ``` ``` models wrong on test 0: 185 1: 13 2: 9 3: 45 ceiling of a combination that sees the label pattern 0.7897, distinct patterns 7 single tree - bagging classified differently 17, both wrong 46 single tree - boosting classified differently 21, both wrong 46 bagging - boosting classified differently 6, both wrong 52 ``` The first line fixes the ceiling. On 185 of the 252 subscribers all three are right, on 45 all three are wrong; the remaining 22 subscribers are the only region where a combination can work. Those 45 subscribers cannot be rescued by any vote, any sum or any second layer, because all three inputs point in the same wrong direction. The second line turns this into an exact number. Three labels could form eight patterns, only seven are seen on the test set, and the rule that says the majority label in each pattern gives 0.7897. **No combination that sees nothing but the three labels can pass this**, and this number is equal to bagging's own number. The last three lines say why. Bagging and boosting classify only **6** of the 252 subscribers differently. One is a parallel vote and the other a sequential sum; their structures are opposites, their decisions nearly the same. Bagging errs on 53 subscribers, boosting on 57, and the two err **together** on 52: nearly every subscriber boosting misses, bagging misses too. The single tree is a little further away (17 and 21 subscribers), but it too is fed from the same family. #### From Which Set the Second Layer Is Learned Stacking builds a model that takes the three outputs as input. The data this model will learn from is the outputs of the base models; on which set those outputs were produced is the lesson's question. ```python def second_layer(u, rate=2.0, rounds=400): # COMBINATION 3: stacking subset, w, bias = SETS[u], [0.0] * len(BASE), 0.0 for _ in range(rounds): gw, gs = [0.0] * len(BASE), 0.0 for i, (_, y) in enumerate(subset): h = squash(sum(w[j] * BASE[j][1][u][i] for j in range(len(BASE))) + bias) - y for j in range(len(BASE)): gw[j] += h * BASE[j][1][u][i] gs += h for j in range(len(BASE)): w[j] -= rate * gw[j] / len(subset) bias -= rate * gs / len(subset) return w, bias def layer_accuracy(model, u): w, bias = model return sum((sum(w[j] * BASE[j][1][u][i] for j in range(len(BASE))) + bias > 0) == y for i, (_, y) in enumerate(SETS[u])) / len(SETS[u]) RIGHT, WRONG = second_layer(1), second_layer(0) print(f"{'second layer':<22} {'train':>8} {'validation':>10} {'test':>8} weights") for name, model, u in (("fit on validation", RIGHT, 1), ("fit on training", WRONG, 0)): print(f"{name:<22}" + "".join(f"{layer_accuracy(model, v):>{(8, 10, 8)[v]}.4f}" for v in range(3)) + " " + " ".join(f"{q:+.2f}" for q in model[0])) FINAL = [("bagging", accuracy(BASE[1][1][1], B_VAL), accuracy(BASE[1][1][2], B_TEST), BASE[1][2], 31), ("boosting", accuracy(BASE[2][1][1], B_VAL), accuracy(BASE[2][1][2], B_TEST), BASE[2][2], 1279), ("stacking", layer_accuracy(RIGHT, 1), layer_accuracy(RIGHT, 2), sum(t[2] for t in BASE) + 4, 3)] print(f"\n{'combination':<12} {'reported':>13} {'test':>8} {'inflation':>9} " f"{'splits':>7} {'candidates':>10}") for name, d, s, cost, candidates in FINAL: print(f"{name:<12} {d:>13.4f} {s:>8.4f} {d - s:>9.4f} {cost:>7} {candidates:>10}") print(f"total tuning budget of the topic {sum(t[4] for t in FINAL)} candidates") ``` ``` second layer train validation test weights fit on validation 0.8333 0.8254 0.7778 +1.99 +1.54 +1.45 fit on training 0.8532 0.7937 0.7778 +5.04 +2.38 +0.54 combination reported test inflation splits candidates bagging 0.8413 0.7897 0.0516 568 31 boosting 0.8373 0.7738 0.0635 30 1279 stacking 0.8254 0.7778 0.0476 645 3 total tuning budget of the topic 1313 candidates ``` The last column of the two rows is the same: 0.7778. Learning from the wrong set did not **change the delivered number** in this table. There are two things it did change, and both are visible. The first is the reported number. When the layer is learned from the training set it reads 0.8532 on that set; the real number is 0.7778, so the inflation is 0.0754. The layer learned from validation reads 0.8254 on its own set and its inflation is 0.0476. **Both numbers are inflated**, because both were read from the set the layer learned from; the wrong set only enlarged the inflation by 0.0278, that is, seven subscribers. The second is the weights. The layer learned from validation gives balanced weights: +1.99, +1.54, +1.45. The layer learned from training writes +5.04 to the single tree and lowers boosting to +0.54. The cause is in the training column: the single tree reads 0.8426 on the training set, the bagged ensemble 0.8175, boosting 0.8082. The second layer takes the model that **memorized** most to be the best model. Because the memorization here is mild, the harm stayed mild; had one of the base models learned the training set exactly, the layer would have given all the weight to it and the delivered number would have dropped too. Learning from validation has a cost as well: that set is now spent. The way that spends nothing is to learn the second layer from the base models' outputs **on rows that did not train them**; the out-of-bag prediction is exactly this form. The last table is the topic's bill. The inflation of the three combinations is 0.0516, 0.0635 and 0.0476; the smallest budget gives the highest number. Bagging reaches 0.7897 with 31 candidates, boosting stays at 0.7738 with 1,279. The topic's total budget is 1,313 candidates and its return is a number 0.0318 above the baseline. #### Summary - The three base models give 0.7778, 0.7897 and 0.7738 on the test set; their costs are 43, 568 and 30 splits, that is, the cheapest combination is one nineteenth of the most expensive. - On 185 of the 252 subscribers all three are right, on 45 all three are wrong; the region where a combination can work is 22 subscribers. - The ceiling of a combination that looks at the three labels is 0.7897 and equals bagging's number; bagging and boosting classify only 6 subscribers differently. - When the second layer is learned from training it reports 0.8532 and gives 0.7778 (inflation 0.0754); when it is learned from validation it reports 0.8254 and again 0.7778 (inflation 0.0476). - The wrong set shifts the weight to the model that memorizes most: +5.04 to the single tree against +1.99 in the correct setup; the smallest tuning budget of the three combinations gives the highest number. #### Next Step This topic tried seven structures in seven lessons: an example-based procedure, a probabilistic classifier built on a conditional independence assumption, a margin-based classifier, a decision tree, a bagged tree ensemble, sequential boosting and stacking. All were compared with the same table, the same split and the same baseline, and next to every number was written how many candidates were tried. But the comparison itself was made from start to finish with **a single number**: accuracy. What that number hides was never asked. Of the 45 subscribers on whom all three were wrong, how many were really suspicious and how many meant a team sent to a door for nothing was never counted. It is possible for two models that give the same accuracy to make entirely different errors, and the costs of the two kinds of error are not the same. The next topic, model evaluation, opens with this question. ## Evaluation and Tuning Separating out which measure, which threshold and how many candidates a model's number came from: the different error distribution of two models giving the same accuracy and the different cost of the two error types, the ranking changing across accuracy, precision, recall and F1, the opposite movement of precision and recall as the threshold is swept and which curve is misleading on an imbalanced set, the difference between the stated probability and the realized rate and the existence of a model whose ranking is good but whose calibration is bad, mean absolute error, mean squared error and the explained variation ratio not giving the same ranking, the estimate shifting as the number of folds grows and the number of trainings rising, the best candidate three searches find with the same budget and the validation number that inflates as the budget grows, the slice where the error concentrates and the number of bad slices that grows by chance as the slices multiply. ### Confusion Matrix Source: https://academia.sh/en/courses/supervised-learning/confusion-matrix Separating the error distributions of two models with the same accuracy: among forty-nine candidates, fourteen accuracy values are shared by more than one candidate, two candidates giving 0.7817 catch 16 of the 61 suspicious subscribers in one and 32 in the other, and the number of flagged subscribers ranges from 26 to 58. When a field visit counts as 1 and a missed illicit case as 6, the cost is 296 and 232, the baseline costs 366, and calling everyone suspicious at 252 is cheaper than the baseline. The best candidate by accuracy gives 263 in cost, while the best candidate by cost is twentieth among forty-nine candidates in accuracy. The previous topic tried seven model families on the same held-out set and wrote next to every number how many candidates were tried to find it. The whole comparison ran on a single number: **accuracy**. The difference a family adds and the difference the tuning budget inflates were both read from this one number. **Which errors it hides** was never asked. The test set has 61 suspicious subscribers out of 252. A model with an accuracy of 0.80 is wrong on fifty subscribers, but the direction of those fifty errors is not written: calling a non-suspicious subscriber suspicious and missing a suspicious one fall into the same cell. The two cannot stand in for each other — the cost of one is a wasted field visit, the cost of the other is an unnoticed illicit case. - **ET1.** The set, the split and the features stay fixed throughout the course: the fictional subscriber table, the 756/252/252 split, the twelve features the Data Preparation and Feature Engineering course left behind. The only thing that changes is the model. - **ET2.** The baseline does not change: telling every subscriber the most frequent class of the training set. This procedure gives 0.7579 on the test set. - **ET3.** The positive class is **the suspicious subscriber**. The words positive and negative are not a value judgment; they are the name of the class being counted. - **ET4.** The model produces a probability and in this lesson the decision threshold is **fixed at 0.5**. The threshold itself is left to the third lesson of this topic. - **ET5.** Forty-nine candidates are tried: forty-two decision trees as the product of six depths and seven class weights, and a gradient-based linear model at seven class weights. The class weight is used in the form it had in the Data Preparation and Feature Engineering course. - **ET6.** This lesson **does not choose a model**; the test set number of all forty-nine candidates is printed. Choosing from the validation set is the work of the third lesson. - **ET7.** The cost has two items: one **field visit** for every flagged subscriber, one **missed illicit case** for every suspicious subscriber that is missed. The visit cost is written to all flagged subscribers; the outcome of the visit does not change the cost. The unit values are fictional and are not read from the data. #### The Four Cells A **confusion matrix** holds four numbers in a two-class decision. When the model says suspicious and the subscriber really is suspicious, it is a **true positive**; if not, a **false positive**. When the model does not say suspicious and the subscriber is suspicious, it is a **false negative**; if not, a **true negative**. Accuracy adds two of these four numbers and divides the sum by all four; the rest of the information is thrown away. ```python # evaluation.py — MODEL: the FICTIONAL subscriber table is rebuilt with the same seed. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] def impurity(rows): n = sum(x.get("w", 1.0) for x in rows) p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): # MODEL: weighted decision tree n = sum(x.get("w", 1.0) for x in rows) p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n best = None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue gain = impurity(rows) - (sum(x.get("w", 1.0) for x in left) * impurity(left) + sum(x.get("w", 1.0) for x in right) * impurity(right)) / n if best is None or gain > best[0]: best = (gain, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def probability(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def weighted(train, w): return [dict(x, w=(w if x["suspicious"] else 1.0)) for x in train] def matrix(score, data, threshold=0.5): # tp, fp, fn, tn m = [0, 0, 0, 0] for x in data: i = (0 if x["suspicious"] else 1) + (0 if score(x) >= threshold else 2) m[i] += 1 return m N_SUSP = sum(x["suspicious"] for x in TEST) print(f"test set {len(TEST)} subscribers, suspicious {N_SUSP}, " f"baseline {1 - N_SUSP / len(TEST):.4f}") t6 = tree(TRAIN, 6, FIELDS) m6 = matrix(lambda x: probability(t6, x), TEST) print(f"depth 6 decision tree: accuracy {(m6[0] + m6[3]) / len(TEST):.4f}") print(f"{'':>24}{'actual suspicious':>19}{'actual not suspicious':>23}") print(f"{'called suspicious':>24}{m6[0]:>19}{m6[1]:>23}") print(f"{'not called suspicious':>24}{m6[2]:>19}{m6[3]:>23}") ``` ``` test set 252 subscribers, suspicious 61, baseline 0.7579 depth 6 decision tree: accuracy 0.7937 actual suspicious actual not suspicious called suspicious 24 15 not called suspicious 37 176 ``` Accuracy is 0.7937 and above the baseline. The matrix makes the same model read differently: the model catches 24 of the 61 suspicious subscribers, misses 37, and flags 15 subscribers for nothing. Most of the accuracy comes from the bottom right cell — the majority in the set is already them. The whole 0.7579 of the baseline comes from that cell: the procedure that calls nobody suspicious collects 191 true negatives and accumulates 61 false negatives. #### The Same Number, a Different Matrix The way accuracy throws information away shows most clearly when two models give the same number. Forty-nine candidates are tried and the candidates that share the same accuracy value are searched for. ```python def scaler(train, fields): o = {a: sum(x[a] for x in train) / len(train) for a in fields} s = {a: (sum((x[a] - o[a]) ** 2 for x in train) / len(train)) ** 0.5 or 1.0 for a in fields} return lambda x: [(x[a] - o[a]) / s[a] for a in fields] def linear(train, fields, w, epochs=300, step=0.5): # MODEL: gradient-based linear model f, coef, b = scaler(train, fields), [0.0] * len(fields), 0.0 X = [f(x) for x in train] y = [x["suspicious"] for x in train] sw = [(w if x["suspicious"] else 1.0) for x in train] n = sum(sw) for _ in range(epochs): gc, gb = [0.0] * len(fields), 0.0 for i, xi in enumerate(X): z = b + sum(coef[j] * xi[j] for j in range(len(coef))) err = (1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - y[i]) * sw[i] gb += err for j in range(len(coef)): gc[j] += err * xi[j] b -= step * gb / n for j in range(len(coef)): coef[j] -= step * gc[j] / n def score(x): v = f(x) z = b + sum(coef[j] * v[j] for j in range(len(coef))) return 1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) return score CANDIDATES = {} for d in (3, 4, 5, 6, 7, 8): for w in (1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0): t = tree(weighted(TRAIN, w), d, FIELDS) CANDIDATES[f"tree d{d} w{w}"] = matrix(lambda x, t=t: probability(t, x), TEST) for w in (1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0): s = linear(TRAIN, FIELDS, w) CANDIDATES[f"linear w{w}"] = matrix(s, TEST) TIES = {} for name, m in CANDIDATES.items(): TIES.setdefault(round((m[0] + m[3]) / len(TEST), 4), []).append(name) shared = {d: v for d, v in TIES.items() if len(v) > 1} print(f"candidates tried {len(CANDIDATES)}, distinct accuracy values {len(TIES)}, " f"shared by more than one candidate {len(shared)}") widest = max(shared.items(), key=lambda kv: max(CANDIDATES[a][0] for a in kv[1]) - min(CANDIDATES[a][0] for a in kv[1])) print(f"\n{'candidate':<14}{'accuracy':>9}{'tp':>5}{'fp':>5}{'fn':>5}{'tn':>5}{'flagged':>10}") for d in sorted(shared, reverse=True)[:3]: for name in shared[d]: m = CANDIDATES[name] print(f"{name:<14}{d:>9.4f}{m[0]:>5}{m[1]:>5}{m[2]:>5}{m[3]:>5}{m[0] + m[1]:>10}") print() print(f"widest divergence at accuracy {widest[0]:.4f}: {' / '.join(widest[1])}") ``` ``` candidates tried 49, distinct accuracy values 26, shared by more than one candidate 14 candidate accuracy tp fp fn tn flagged tree d5 w1.0 0.8016 23 12 38 179 35 linear w1.0 0.8016 22 11 39 180 33 linear w1.5 0.8016 29 18 32 173 47 tree d3 w1.0 0.7937 17 8 44 183 25 tree d6 w1.0 0.7937 24 15 37 176 39 tree d4 w1.0 0.7817 16 10 45 181 26 tree d8 w1.0 0.7817 32 26 29 165 58 widest divergence at accuracy 0.7817: tree d4 w1.0 / tree d8 w1.0 ``` The forty-nine candidates produce only twenty-six distinct accuracy values and fourteen of these are shared by more than one candidate. The first block of the table shows the highest accuracy in the set: three candidates give the value 0.8016 at once, and from two different model families at that. The numbers of suspicious subscribers they catch are 22, 23 and 29; the numbers of subscribers they flag are 33, 35 and 47. To someone looking by accuracy the three are the same model. The last block is the widest divergence. Two candidates with an accuracy of 0.7817 catch 16 of the 61 suspicious subscribers in one and 32 in the other — exactly twice as many — and flag 58 subscribers against 26. The total number of correct predictions equalizes at 197 because the 16 true positives the second model gains swap exactly with the 16 true negatives it loses. Accuracy is indifferent to this swap; the matrix is not. #### The Two Errors Do Not Cost the Same The swap stops being a matter of indifference where the price of the two error types is counted. In the fictional set, one **field visit** is written for every flagged subscriber and one **missed illicit case** for every suspicious subscriber that is missed. The unit of the visit is taken as 1 and that of the missed illicit case as 6; these two numbers are not read from the data, they are given from outside. ```python VISIT_COST, MISS_COST = 1.0, 6.0 def cost(m): return VISIT_COST * (m[0] + m[1]) + MISS_COST * m[2] print(f"{'procedure':<28}{'accuracy':>9}{'visits':>9}{'missed':>11}{'cost':>8}") for name, m in (("baseline (none suspicious)", [0, 0, N_SUSP, len(TEST) - N_SUSP]), ("everyone suspicious", [N_SUSP, len(TEST) - N_SUSP, 0, 0]), ("tree d4 w1.0", CANDIDATES["tree d4 w1.0"]), ("tree d8 w1.0", CANDIDATES["tree d8 w1.0"]), ("tree d5 w1.0", CANDIDATES["tree d5 w1.0"]), ("linear w1.5", CANDIDATES["linear w1.5"])): print(f"{name:<28}{(m[0] + m[3]) / len(TEST):>9.4f}{m[0] + m[1]:>9}" f"{m[2]:>11}{cost(m):>8.0f}") a, b = CANDIDATES["tree d4 w1.0"], CANDIDATES["tree d8 w1.0"] ratio = ((a[0] + a[1]) - (b[0] + b[1])) / (b[2] - a[2]) print(f"\nbreak-even cost ratio {ratio:.2f}: the ranking flips once the cost of a missed " f"illicit case exceeds {ratio:.2f} times the cost of one field visit") best_acc = max(CANDIDATES, key=lambda k: (CANDIDATES[k][0] + CANDIDATES[k][3])) best_cost = min(CANDIDATES, key=lambda k: cost(CANDIDATES[k])) print(f"best candidate by accuracy {best_acc}, cost {cost(CANDIDATES[best_acc]):.0f}") print(f"best candidate by cost {best_cost}, accuracy " f"{(CANDIDATES[best_cost][0] + CANDIDATES[best_cost][3]) / len(TEST):.4f}") print(f"accuracy rank of the best candidate by cost " f"{sorted(CANDIDATES, key=lambda k: -(CANDIDATES[k][0] + CANDIDATES[k][3])).index(best_cost) + 1}" f" / {len(CANDIDATES)}") ``` ``` procedure accuracy visits missed cost baseline (none suspicious) 0.7579 0 61 366 everyone suspicious 0.2421 252 0 252 tree d4 w1.0 0.7817 26 45 296 tree d8 w1.0 0.7817 58 29 232 tree d5 w1.0 0.8016 35 38 263 linear w1.5 0.8016 47 32 239 break-even cost ratio 2.00: the ranking flips once the cost of a missed illicit case exceeds 2.00 times the cost of one field visit best candidate by accuracy tree d5 w1.0, cost 263 best candidate by cost linear w3.5, accuracy 0.7460 accuracy rank of the best candidate by cost 20 / 49 ``` The second row of the table shows on its own that accuracy is a one-directional measure. Calling everyone suspicious gives 0.2421 in accuracy — a number that looks unusable — but costs 252 and is cheaper than the baseline's 366. Neither procedure has learned anything; one looks good on accuracy, the other on cost. The cost of the two candidates with the same accuracy is 296 and 232. The 64 units between them are the price of the trade-off that accuracy hides. The break-even cost ratio comes out as 2.00: if the cost of a missed illicit case is below twice the cost of a field visit, the model that flags less is cheaper; if it is above, the model that flags more. This ratio cannot be read from the data; it comes from the operator of the measurement network — the same two numbers, two different decisions depending on which side of 2 the ratio falls. The last two lines look at all forty-nine candidates. The best candidate by accuracy gives 263 in cost; the best candidate by cost is in twentieth place in the accuracy ranking. Being at the top on one metric does not bring being at the top on the other, and the difference is not at the level of chance. Accuracy is a cost calculation that counts the two errors equally; every calculation that does not count them equally picks a different candidate. #### Summary - The confusion matrix splits the two-class decision into four numbers — true positive, false positive, false negative, true negative — and accuracy adds two of the four and divides by the total number of observations. - The forty-nine candidates produce only twenty-six distinct accuracy values; fourteen values are shared by more than one candidate, so the same number often corresponds to more than one error distribution. - Two candidates with an accuracy of 0.7817 catch 16 and 32 of the 61 suspicious subscribers and flag 26 and 58 subscribers; because the true positives gained swap exactly with the true negatives lost, accuracy stays the same. - When a field visit counts as 1 and a missed illicit case as 6, the cost of these two candidates is 296 and 232; the break-even cost ratio at which the ranking flips is 2.00, and this number comes not from the data but from the operation. - Calling everyone suspicious stays far below the baseline in accuracy with 0.2421, while in cost it beats the baseline's 366 with 252; the best candidate by accuracy and the best candidate by cost are not the same. #### Next Step This lesson counted the four cells but did not name the metrics derived from the cells. The imbalanced classes lesson of the Data Preparation and Feature Engineering course wrote that the share of the correctly flagged minority is called **precision** and the share of the caught minority is called **recall**, and left their definitions and how they read against each other to this course. The next lesson pays that debt: it derives four metrics from the four cells, ranks five models separately on these metrics and counts on how many metrics the leader of the ranking changes. Why the F1 score is the **harmonic mean** of two numbers and not their average is shown with numbers, together with the case in which the average gives a misleadingly high value. ### Classification Metrics Source: https://academia.sh/en/courses/supervised-learning/classification-metrics Ranking five models by four metrics derived from the confusion matrix: the model that is first in accuracy is second in precision and fourth in recall and the F1 score, three different models take the lead on the four metrics, and seven of the ten model pairs swap places between accuracy and F1. Because the baseline flags no subscriber, its precision is undefined and its recall and F1 score are zero. For a procedure with precision 1.0000 and recall 0.0164, the arithmetic mean is 0.5082 while the F1 score is 0.0323, and the difference rises to 0.4759. The previous lesson split accuracy into four cells but did not name any of the metrics derived from the cells. The imbalanced classes lesson of the Data Preparation and Feature Engineering course had written two names — **precision** for the share of the minority that was flagged correctly, **recall** for the share of the minority that was caught — printed raw numbers, and left the definitions and how the two read against each other to this course. This lesson pays that debt. The measure is this: five models are ranked separately on four metrics and it is counted on how many metrics the **leader** of the ranking changes. If the ranking is preserved when the metric changes, which one is chosen does not matter; if the order turns, the choice of metric is a decision that comes before the choice of model, and that decision cannot be read from the data. - **ET8.** All four metrics are derived from a single confusion matrix; no new measurement is made, the same four numbers are combined in another way. - **ET9.** The five models are taken from the forty-nine-candidate grid of the previous lesson. The number of candidates trained in this lesson is **five**; the selection was made for metric variety, not according to the number on the test set. - **ET10.** The denominator of precision is the number of subscribers the model flags and it can be zero. When it is zero, precision is **undefined**, not zero; the baseline is in this situation. - **ET11.** The denominator of recall is the 61 suspicious subscribers in the test set and is independent of the model. - **ET12.** The F1 score is the harmonic mean of precision and recall; if the number of true positives is zero, it is written as zero. - **ET13.** All four metrics are read at the decision threshold of 0.5. The number the threshold moves is measured in the next lesson. - **ET14.** The rankings are built on the five models only; the baseline is a comparison row and does not enter the ranking. - **ET15.** The numbers of the imbalanced classes lesson are not repeated. The set there had eleven features, the one here has twelve; the numbers cannot be compared one to one, the metric definitions are compared. #### Four Metrics from Four Cells Accuracy is the ratio of the sum of true positives and true negatives to all observations. **Precision** is how many of the subscribers called suspicious really are suspicious: the ratio of true positives to the sum of true positives and false positives. **Recall** is how many of the really suspicious subscribers were caught: the ratio of true positives to the sum of true positives and false negatives. Precision looks at the column the model flagged, recall at the suspicious row of the truth. The **F1 score** combines the two in a single number and the combination is not the arithmetic mean but the **harmonic mean**: $F_1 = \dfrac{2pr}{p+r}$, where $p$ is precision and $r$ is recall. ```python # evaluation.py — MODEL: the FICTIONAL subscriber table is rebuilt with the same seed. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] def impurity(rows): n = sum(x.get("w", 1.0) for x in rows) p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): # MODEL: weighted decision tree n = sum(x.get("w", 1.0) for x in rows) p = sum(x.get("w", 1.0) * x["suspicious"] for x in rows) / n best = None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue gain = impurity(rows) - (sum(x.get("w", 1.0) for x in left) * impurity(left) + sum(x.get("w", 1.0) for x in right) * impurity(right)) / n if best is None or gain > best[0]: best = (gain, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def probability(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def weighted(train, w): return [dict(x, w=(w if x["suspicious"] else 1.0)) for x in train] def matrix(score, data, threshold=0.5): # tp, fp, fn, tn m = [0, 0, 0, 0] for x in data: i = (0 if x["suspicious"] else 1) + (0 if score(x) >= threshold else 2) m[i] += 1 return m def scaler(train, fields): o = {a: sum(x[a] for x in train) / len(train) for a in fields} s = {a: (sum((x[a] - o[a]) ** 2 for x in train) / len(train)) ** 0.5 or 1.0 for a in fields} return lambda x: [(x[a] - o[a]) / s[a] for a in fields] def linear(train, fields, w, epochs=300, step=0.5): # MODEL: gradient-based linear model f, coef, b = scaler(train, fields), [0.0] * len(fields), 0.0 X = [f(x) for x in train] y = [x["suspicious"] for x in train] sw = [(w if x["suspicious"] else 1.0) for x in train] n = sum(sw) for _ in range(epochs): gc, gb = [0.0] * len(fields), 0.0 for i, xi in enumerate(X): z = b + sum(coef[j] * xi[j] for j in range(len(coef))) err = (1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - y[i]) * sw[i] gb += err for j in range(len(coef)): gc[j] += err * xi[j] b -= step * gb / n for j in range(len(coef)): coef[j] -= step * gc[j] / n def score(x): v = f(x) z = b + sum(coef[j] * v[j] for j in range(len(coef))) return 1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) return score N_SUSP = sum(x["suspicious"] for x in TEST) MODEL = {} for name, d, w in (("depth 5 tree", 5, 1.0), ("depth 3 tree", 3, 1.0), ("depth 8 tree", 8, 1.0), ("depth 6 tree w3", 6, 3.0)): t = tree(weighted(TRAIN, w), d, FIELDS) MODEL[name] = matrix(lambda x, t=t: probability(t, x), TEST) MODEL["linear w2.5"] = matrix(linear(TRAIN, FIELDS, 2.5), TEST) MODEL["baseline"] = [0, 0, N_SUSP, len(TEST) - N_SUSP] def metrics(m): tp, fp, fn, tn = m p = tp / (tp + fp) if tp + fp else float("nan") r = tp / (tp + fn) return (tp + tn) / sum(m), p, r, (2 * p * r / (p + r) if tp else 0.0) print(f"{'model':<18}{'accuracy':>9}{'precision':>10}{'recall':>11}{'F1':>8}" f"{'flagged':>9}") for name, m in MODEL.items(): a, p, r, f = metrics(m) print(f"{name:<18}{a:>9.4f}{('undefined' if p != p else f'{p:.4f}'):>10}" f"{r:>11.4f}{f:>8.4f}{m[0] + m[1]:>9}") ``` ``` model accuracy precision recall F1 flagged depth 5 tree 0.8016 0.6571 0.3770 0.4792 35 depth 3 tree 0.7937 0.6800 0.2787 0.3953 25 depth 8 tree 0.7817 0.5517 0.5246 0.5378 58 depth 6 tree w3 0.7302 0.4545 0.5738 0.5072 77 linear w2.5 0.7659 0.5135 0.6230 0.5630 74 baseline 0.7579 undefined 0.0000 0.0000 0 ``` The baseline row breaks three metrics at once. The precision of a procedure that flags no subscriber is undefined — there is nothing to divide — and its recall and F1 score are zero. The same procedure that reads 0.7579 on accuracy gives an unusable result on the other three metrics. This is the defined form of the table the imbalanced classes lesson showed with raw numbers. In the five-model rows two columns move in opposite directions. As the number of flagged subscribers rises from 25 to 77, precision falls from 0.6800 to 0.4545 and recall rises from 0.2787 to 0.5738. The model that flags less is right more often when it flags, but misses most; the model that flags more misses fewer, but more than half of what it flags is wasted. The accuracy column is almost indifferent to this trade-off: there are 0.0714 points between the highest and the lowest, 0.2255 points in precision, 0.3443 points in recall. #### The Ranking Turns with the Metric The same five models are ranked separately on four metrics and the rankings are printed side by side. ```python FIVE = [a for a in MODEL if a != "baseline"] METRICS = ("accuracy", "precision", "recall", "F1") RANK = {o: {a: j + 1 for j, a in enumerate( sorted(FIVE, key=lambda a: -metrics(MODEL[a])[i]))} for i, o in enumerate(METRICS)} print(f"{'model':<18}" + "".join(f"{o:>11}" for o in METRICS)) for a in FIVE: print(f"{a:<18}" + "".join(f"{RANK[o][a]:>11}" for o in METRICS)) leaders = [min(RANK[o], key=RANK[o].get) for o in METRICS] print(f"\nmodel leading each metric: {' | '.join(leaders)}") print(f"{len(set(leaders))} distinct models lead on the four metrics; the model chosen by " f"accuracy is not in the lead on {sum(1 for b in leaders[1:] if b != leaders[0])} metrics") flips = sum(1 for i in range(len(FIVE)) for j in range(i + 1, len(FIVE)) if (RANK["accuracy"][FIVE[i]] - RANK["accuracy"][FIVE[j]]) * (RANK["F1"][FIVE[i]] - RANK["F1"][FIVE[j]]) < 0) print(f"model pairs that swap places between accuracy and F1 {flips} / " f"{len(FIVE) * (len(FIVE) - 1) // 2}") ``` ``` model accuracy precision recall F1 depth 5 tree 1 2 4 4 depth 3 tree 2 1 5 5 depth 8 tree 3 3 3 2 depth 6 tree w3 5 5 2 3 linear w2.5 4 4 1 1 model leading each metric: depth 5 tree | depth 3 tree | linear w2.5 | linear w2.5 3 distinct models lead on the four metrics; the model chosen by accuracy is not in the lead on 3 metrics model pairs that swap places between accuracy and F1 7 / 10 ``` The accuracy column descends smoothly from 1 to 5 because the models were listed by accuracy. The other three columns do not recognize that order. The depth 5 tree, first in accuracy, is second in precision and fourth in recall and the F1 score. The depth 3 tree, second in accuracy, is first in precision but last in recall and the F1 score. The linear model, fourth in accuracy, is first in recall and the F1 score. The numbers gather in three places. **Three different models** take the lead on the four metrics; the model chosen by accuracy is not in the lead on **three** of the four metrics; **seven** of the ten model pairs swap places between accuracy and F1. The ranking is built not on the measure but on the metric. Writing which budget a model's number was found with is not enough — it is also necessary to write which metric it was ranked by, because five models have entered four separate races on four metrics. #### Why the Harmonic Mean Is Not the Mean If the F1 score combined precision and recall with the arithmetic mean, a procedure with one side high and the other near zero would look to be in the middle. The harmonic mean gives weight to the smaller one. The difference grows as the two numbers move apart, and the growth can be counted. ```python print(f"{'precision':>9}{'recall':>11}{'mean':>10}{'F1':>8}{'diff':>8}") for p, r in ((1.00, 1 / 61), (0.90, 0.10), (0.70, 0.30), (0.60, 0.40), (0.50, 0.50)): o, f = (p + r) / 2, 2 * p * r / (p + r) print(f"{p:>9.4f}{r:>11.4f}{o:>10.4f}{f:>8.4f}{o - f:>8.4f}") MEAN = {a: sum(metrics(MODEL[a])[1:3]) / 2 for a in FIVE} print(f"\n{'model':<18}{'mean':>10}{'F1':>8}{'diff':>8}{'mean/F1 rank':>15}") for j, a in enumerate(sorted(FIVE, key=lambda a: -MEAN[a])): f = metrics(MODEL[a])[3] ranks = str(j + 1) + " / " + str(RANK["F1"][a]) print(f"{a:<18}{MEAN[a]:>10.4f}{f:>8.4f}{MEAN[a] - f:>8.4f}{ranks:>15}") ``` ``` precision recall mean F1 diff 1.0000 0.0164 0.5082 0.0323 0.4759 0.9000 0.1000 0.5000 0.1800 0.3200 0.7000 0.3000 0.5000 0.4200 0.0800 0.6000 0.4000 0.5000 0.4800 0.0200 0.5000 0.5000 0.5000 0.5000 0.0000 model mean F1 diff mean/F1 rank linear w2.5 0.5682 0.5630 0.0053 1 / 1 depth 8 tree 0.5382 0.5378 0.0003 2 / 2 depth 5 tree 0.5171 0.4792 0.0379 3 / 4 depth 6 tree w3 0.5142 0.5072 0.0069 4 / 3 depth 3 tree 0.4793 0.3953 0.0840 5 / 5 ``` The first row of the first table shows the extreme case: a procedure that flags a single subscriber and is right on that subscriber has a precision of 1.0000 and a recall of 0.0164. The arithmetic mean gives 0.5082 — a number higher than all five real models — and the F1 score gives 0.0323. The difference is 0.4759. The three middle rows meet the same arithmetic mean, 0.5000, with three different F1 values: 0.1800, 0.4200 and 0.4800. When the two numbers are equal the difference goes to zero. The only thing the harmonic mean does is make the combined number collapse too when one of the two metrics collapses. The second table shows the same calculation on real models. In the depth 8 tree, where precision and recall are close to each other, the difference between the two means is 0.0003; in the depth 3 tree, where the two diverge most, it is 0.0840. The ranking also breaks somewhere: the depth 5 tree, third by the arithmetic mean, is fourth on the F1 score, and the weighted tree, fourth, moves to third place. The sum of precision and recall of the two models is almost the same (0.5171 and 0.5142) but its distribution is different, and the harmonic mean sees the distribution. #### Summary - Precision is read from the column the model flagged, recall from the suspicious row of the truth; accuracy uses neither of them, it adds the two diagonals of the four cells. - The precision of the baseline is undefined because no subscriber was flagged; the same procedure reads 0.7579 on accuracy while giving zero on recall and the F1 score. - The five models are squeezed between 0.7302 and 0.8016 on accuracy while they spread 0.2255 points on precision and 0.3443 points on recall; accuracy is almost indifferent to this trade-off. - Three different models take the lead on the four metrics, the model chosen by accuracy is not in the lead on three metrics, and seven of the ten model pairs swap places between accuracy and F1. - The F1 score is the harmonic mean: for a procedure with a precision of 1.0000 and a recall of 0.0164 the arithmetic mean gives 0.5082 and the F1 score 0.0323; when the two numbers are equal, the two means are equal too. #### Next Step All four metrics of this lesson were read at the decision threshold of 0.5 and that threshold was never questioned. Yet the number the models produce is a probability; 0.5 is a choice that comes from outside, and it changes the table from start to finish. The next lesson sweeps the threshold from 0 to 1, prints precision and recall side by side at every threshold and chooses the **operating point** with the cost calculation of the first lesson. The same sweep produces two curves — the **ROC curve** and the **precision–recall curve** — and which of them misleads on an imbalanced set, and how far apart the areas under the two curves diverge on the same model, is shown by counting. ### Threshold Selection and Curves Source: https://academia.sh/en/courses/supervised-learning/threshold-selection-and-curves Separating the two numbers the threshold moves from the area under the curve: as the threshold goes from 0.10 to 0.90, precision rises from 0.3608 to 1.0000, recall falls from 0.9344 to 0.0328 and the cost rises from 182 to 356. The 0.16 chosen with the validation set among ninety-nine threshold candidates gives a cost of 181 on the test set, is 86 units cheaper than the default threshold of 0.50, and would have looked 7 units better if it had been read from the test set. When the suspicious share is lowered from 0.2421 to 0.0354, the area under the ROC curve rises from 0.8058 to 0.8624 while that of the precision–recall curve falls from 0.6212 to 0.4124. All four metrics of the previous lesson were read at the decision threshold of 0.5. That threshold does not come from the model: the model produces a probability, the threshold turns that probability into a decision, and it is given from outside. The only justification for 0.5 is that it sits in the middle of the two classes; in a set where the suspicious share is 0.2421, nothing says why that middle is the right place. This lesson sweeps the threshold: the same model, the same test set, only the threshold changes. Precision and recall are recomputed at every threshold, the **operating point** is chosen with a cost calculation, and the two curves the sweep produces are printed as numbers. - **ET16.** The model produces a probability; the threshold turns this probability into a decision and is independent of the model. Ninety-nine thresholds are tried, from 0.01 to 0.99; this is the tuning budget of this lesson. - **ET17.** The cost is the one from the first lesson: 1 unit of field visit for every flagged subscriber, 6 units for every suspicious subscriber that is missed. The operating point is chosen from the **validation set**; the number on the test set is only reported. - **ET18.** The training set, the features and the model stay fixed throughout the sweep. - **ET19.** On the ROC curve the horizontal axis is the false positive rate and the vertical axis is recall; on the precision–recall curve the horizontal axis is recall and the vertical axis is precision. The curves are **not drawn**, they are printed as points. - **ET20.** The area under the curve is computed with the trapezoid rule and observations with equal scores count as a single point. - **ET21.** The thinned set is built by keeping one tenth of the suspicious subscribers in the test set; the training and the model do not change. The estimate made with the remaining seven suspicious subscribers is volatile and is read against its baseline. - **ET22.** Choosing a threshold requires not that the probability be right but only that the **ranking** be right. #### Sweeping the Threshold As the threshold rises, the model flags fewer subscribers. It is right more often on the ones it flags, but the number of suspicious subscribers it misses grows. ```python # evaluation.py — MODEL: the FICTIONAL subscriber table is rebuilt with the same seed. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): # MODEL: decision tree n = len(rows) p = sum(x["suspicious"] for x in rows) / n best = None if depth and n >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue gain = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / n if best is None or gain > best[0]: best = (gain, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def probability(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def matrix(score, data, threshold=0.5): # tp, fp, fn, tn m = [0, 0, 0, 0] for x in data: m[(0 if x["suspicious"] else 1) + (0 if score(x) >= threshold else 2)] += 1 return m def linear(train, fields, w, epochs=300, step=0.5): # MODEL: gradient-based linear model mu = {f: sum(x[f] for x in train) / len(train) for f in fields} sd = {f: (sum((x[f] - mu[f]) ** 2 for x in train) / len(train)) ** 0.5 or 1.0 for f in fields} scale = lambda x: [(x[f] - mu[f]) / sd[f] for f in fields] coef, b = [0.0] * len(fields), 0.0 X = [scale(x) for x in train] y = [x["suspicious"] for x in train] sw = [(w if x["suspicious"] else 1.0) for x in train] total = sum(sw) for _ in range(epochs): gc, gb = [0.0] * len(fields), 0.0 for i, xi in enumerate(X): z = b + sum(coef[j] * xi[j] for j in range(len(coef))) e = (1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - y[i]) * sw[i] gb += e for j in range(len(coef)): gc[j] += e * xi[j] b -= step * gb / total for j in range(len(coef)): coef[j] -= step * gc[j] / total def score(x): v = scale(x) z = b + sum(coef[j] * v[j] for j in range(len(coef))) return 1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) return score N_SUSP = sum(x["suspicious"] for x in TEST) SCORE_FN = linear(TRAIN, FIELDS, 1.0) # MODEL: probability-producing gradient-based linear model VISIT_COST, MISS_COST = 1.0, 6.0 def point(score, rows, e): tp, fp, fn, tn = matrix(score, rows, e) return {"flagged": tp + fp, "tp": tp, "fpr": fp / (fp + tn), "recall": tp / (tp + fn), "precision": tp / (tp + fp) if tp + fp else float("nan"), "cost": VISIT_COST * (tp + fp) + MISS_COST * fn} print(f"{'threshold':>9}{'flagged':>9}{'caught':>10}{'fpr':>9}" f"{'recall':>11}{'precision':>10}{'cost':>7}") for e in (0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80, 0.90): n = point(SCORE_FN, TEST, e) print(f"{e:>9.2f}{n['flagged']:>9}{n['tp']:>10}{n['fpr']:>9.4f}" f"{n['recall']:>11.4f}{n['precision']:>10.4f}{n['cost']:>7.0f}") ``` ``` threshold flagged caught fpr recall precision cost 0.10 158 57 0.5288 0.9344 0.3608 182 0.20 100 45 0.2880 0.7377 0.4500 196 0.30 66 34 0.1675 0.5574 0.5152 228 0.40 47 29 0.0942 0.4754 0.6170 239 0.50 33 22 0.0576 0.3607 0.6667 267 0.60 24 18 0.0314 0.2951 0.7500 282 0.70 17 12 0.0262 0.1967 0.7059 311 0.80 8 8 0.0000 0.1311 1.0000 326 0.90 2 2 0.0000 0.0328 1.0000 356 ``` The two columns move in opposite directions. At a threshold of 0.10 the model flags 158 subscribers, catches 57 of the 61 suspicious ones, and its precision is 0.3608. At a threshold of 0.90 it flags only 2 subscribers and is right on both, but recall is 0.0328. Precision does not rise smoothly: it is 0.7500 at 0.60 and falls to 0.7059 at 0.70, because its denominator shrinks along with the model. Recall, on the other hand, is a function that does not decrease with the threshold; its denominator is fixed. The last column is the one that decides: the cost is 182 at a threshold of 0.10 and 356 at 0.90. The 174 units in between are entirely a product of the threshold; the model, the features and the training set did not change at all. #### The Operating Point Is Chosen with a Cost Calculation The threshold is a hyperparameter and the rule of the course applies here too: a choice does not count as made until it is written which set it was read from. ```python THRESHOLDS = [i / 100 for i in range(1, 100)] best_val = min(THRESHOLDS, key=lambda e: point(SCORE_FN, VAL, e)["cost"]) best_test = min(THRESHOLDS, key=lambda e: point(SCORE_FN, TEST, e)["cost"]) print(f"threshold candidates tried {len(THRESHOLDS)}, validation set {len(VAL)} subscribers") print(f"{'operating point':<22}{'threshold':>9}{'flagged':>9}{'caught':>10}" f"{'validation':>11}{'test':>8}") for name, e in (("default", 0.50), ("chosen on validation", best_val), ("chosen on test", best_test)): n = point(SCORE_FN, TEST, e) print(f"{name:<22}{e:>9.2f}{n['flagged']:>9}{n['tp']:>10}" f"{point(SCORE_FN, VAL, e)['cost']:>11.0f}{n['cost']:>8.0f}") ``` ``` threshold candidates tried 99, validation set 252 subscribers operating point threshold flagged caught validation test default 0.50 33 22 205 267 chosen on validation 0.16 121 51 160 181 chosen on test 0.13 138 55 173 174 ``` The cheapest of the ninety-nine candidates on the validation set is the threshold of 0.16, and it costs 160 units on that set; the same threshold gives 181 units on the test set. The default threshold of 0.50 gives 267 units: choosing the threshold earns 86 units, a gain close to a third, and this gain comes not from the model but from how its output is read. The third row measures the source of the choice. The smallest cost on the test set is at a threshold of 0.13, 174 units. The 7 units in between are the extra share by which someone who reads the choice off the test set looks better — a small number because the cost curve is flat around the threshold, but its sign is always in the same direction. #### Two Curves and Imbalance The sweep produces two curves. The **ROC curve** shows recall against the false positive rate; the **precision–recall curve** shows precision against recall. They differ when the suspicious share changes. ```python def curve(score, rows): # ROC and precision-recall points p = sorted(((score(x), x["suspicious"]) for x in rows), key=lambda t: -t[0]) P = sum(y for _, y in p) N = len(p) - P tp = fp = i = 0 pts = [(0.0, 0.0, 1.0)] while i < len(p): j = i while j < len(p) and p[j][0] == p[i][0]: j += 1 d = sum(y for _, y in p[i:j]) tp, fp, i = tp + d, fp + (j - i) - d, j pts.append((fp / N, tp / P, tp / (tp + fp))) return pts def area_roc(pts, limit=1.0): # trapezoid rule; limit clips the false positive rate return sum((min(b[0], limit) - a[0]) * (a[1] + b[1]) / 2 for a, b in zip(pts, pts[1:]) if a[0] < limit) / limit def area_pr(pts): # sum of steps on the recall axis return sum((b[1] - a[1]) * b[2] for a, b in zip(pts, pts[1:])) THIN = [x for i, x in enumerate(TEST) if not x["suspicious"] or i % 10 == 0] constant = lambda x: 0.5 print(f"{'test set':<18}{'rows':>6}{'suspicious':>11}{'ratio':>7}{'ROC':>8}{'PR':>8}" f"{'constant ROC':>13}{'constant PR':>12}") for name, v in (("real distribution", TEST), ("thinned", THIN)): n, s, ns = curve(SCORE_FN, v), curve(constant, v), sum(x["suspicious"] for x in v) print(f"{name:<18}{len(v):>6}{ns:>11}{ns / len(v):>7.4f}{area_roc(n):>8.4f}" f"{area_pr(n):>8.4f}{area_roc(s):>13.4f}{area_pr(s):>12.4f}") for h in (0.50, 0.70): r = [point(SCORE_FN, v, max(e for e in THRESHOLDS if point(SCORE_FN, v, e)["recall"] >= h))["precision"] for v in (TEST, THIN)] print(f"precision at recall above {h:.2f}: real distribution {r[0]:.4f}, " f"thinned {r[1]:.4f}") ``` ``` test set rows suspicious ratio ROC PR constant ROC constant PR real distribution 252 61 0.2421 0.8058 0.6212 0.5000 0.2421 thinned 198 7 0.0354 0.8624 0.4124 0.5000 0.0354 precision at recall above 0.50: real distribution 0.5161, thinned 0.2500 precision at recall above 0.70: real distribution 0.4624, thinned 0.1667 ``` The two curves move in opposite directions. When the suspicious share falls from 0.2421 to 0.0354, the area under the ROC curve **rises** from 0.8058 to 0.8624, while that of the precision–recall curve **falls** from 0.6212 to 0.4124. Part of the rise is the volatility of an estimate made with seven suspicious subscribers; the real matter is in the last two columns. The ROC curve of the constant-score procedure gives 0.5000 on both sets; the baseline of this curve does not look at the suspicious share. The baseline of the other curve, however, is directly the suspicious share. The last two lines are the counterpart of this in operation. When recall is held above 0.50, precision is 0.5161 on the real distribution and 0.2500 on the thinned set; when it is held above 0.70, 0.4624 against 0.1667. While the area reads 0.86, three of every four subscribers flagged to catch half of the suspicious ones go to waste. On an imbalanced set, the misleading curve is the one whose baseline does not move together with the imbalance. #### What the Single Number Hides The area under the curve collects all thresholds into a single number and drops two things while collecting: which threshold the work is done at and which region of the curve is useful. ```python TREE_D3 = tree(TRAIN, 3, FIELDS) # MODEL: depth 3 decision tree ONE_FIELD = linear(TRAIN, ["mean_consumption"], 1.0) # MODEL: single-feature gradient-based linear model print(f"{'model':<25}{'ROC':>8}{'fpr<0.10':>9}{'PR':>8}{'best cost':>14}" f"{'distinct scores':>16}") for name, s in (("linear, 12 features", SCORE_FN), ("tree d3, 12 features", lambda x: probability(TREE_D3, x)), ("linear, mean_consumption", ONE_FIELD)): n = curve(s, TEST) print(f"{name:<25}{area_roc(n):>8.4f}{area_roc(n, 0.10):>9.4f}{area_pr(n):>8.4f}" f"{min(point(s, TEST, e)['cost'] for e in THRESHOLDS):>14.0f}" f"{len({round(s(x), 12) for x in TEST}):>16}") ``` ``` model ROC fpr<0.10 PR best cost distinct scores linear, 12 features 0.8058 0.3364 0.6212 174 252 tree d3, 12 features 0.7222 0.2553 0.4485 197 8 linear, mean_consumption 0.7138 0.3008 0.5589 213 241 ``` The real reading is in the last two rows. The areas under the ROC curve of the two models are 0.7222 and 0.7138, with 0.0084 between them — by this measure they are almost equal. The other three columns do not agree: the single-feature model is 0.1104 points ahead on the area under the precision–recall curve, 0.0455 points ahead in the region where the false positive rate stays below 0.10, and 16 units behind on the best cost. Three readings, three different winners. The depth 3 tree, which produces eight distinct scores, and the single-feature model, which produces 241 distinct scores, move on the curve in entirely different ways: one in eight large steps, the other in hundreds of small ones. The area squeezes both into the same number. One more item is added to the rule of the course here: without writing which threshold will be used, the area under the curve cannot be turned into an operating decision. #### Summary - As the threshold rises, recall falls in a non-decreasing way, but precision does not rise smoothly: precision of 0.7500 at a threshold of 0.60 falls to 0.7059 at 0.70. Ninety-nine threshold candidates move the cost between 182 and 356; the model and the training set never change. - The threshold of 0.16 chosen on validation gives 181 units on the test set, 86 units cheaper than the default threshold of 0.50; if the same choice had been read from the test set, it would have looked 7 units better. - When the suspicious share falls from 0.2421 to 0.0354, the area under the ROC curve rises from 0.8058 to 0.8624 and the area under the precision–recall curve falls from 0.6212 to 0.4124; the baseline of the first curve is 0.5000 on both sets, that of the second is the suspicious share itself. - Two models counted as equal by the area under the ROC curve with a difference of 0.0084 diverge by 0.1104 on the area under the precision–recall curve and by 16 units on the best cost. #### Next Step This whole lesson rested on the **ranking**. The threshold was swept, the curves were printed, the operating point was chosen; at no point did the number the model says have to be right itself. If all the scores were passed through an increasing function the ranking would not change, the two curves would stay the same, and the operating point would flag the same subscribers. Yet when the model says 0.80 for a subscriber, that number should have a counterpart: about eighty percent of the subscribers given 0.80 should turn out to be suspicious. The next lesson prints the stated probability and the realized share side by side, bin by bin, shows that a model whose ranking is good and whose calibration is bad exists, and writes the correction as a single-variable mapping learned only from the validation set. ### Probability Calibration Source: https://academia.sh/en/courses/supervised-learning/probability-calibration Measuring the difference between the stated probability and the realized share: the calibration error of the linear model is 0.0242, that of the sharpened score, whose ranking is exactly equal to it, is 0.1512, and the area under the curve of the two models is also 0.8058. The calibration error of the depth 8 tree is 0.0000 on the training set by definition and 0.1435 on the test set; the calibration mapping learned from training converges to the identity mapping and corrects nothing. The single-variable mapping learned from the validation set lowers the error of the sharpened score from 0.1512 to 0.0463 and leaves the area under the curve at 0.8058. The whole of the previous lesson rested on the ranking. The threshold was swept, two curves were printed, the operating point was chosen with the cost calculation; at no step did the number the model says have to have a counterpart of its own. If the scores were passed through a strictly increasing transform, the ranking, the curves and the chosen set of subscribers would have stayed unchanged. But when the model says 0.80 for a subscriber, that number claims something: about eighty percent of the subscribers given 0.80 should turn out to be suspicious. This claim is independent of the ranking and is measured separately. **Calibration** is the difference between the stated probability and the realized share. - **ET23.** Calibration is a measure independent of the ranking; two models can give the same ranking and produce different calibration errors. - **ET24.** The probabilities are split into five bins of equal width. The number of bins is a choice: as the number grows, the number of subscribers in each bin falls and the realized share becomes volatile. - **ET25.** The calibration error is computed by summing, in each bin, the absolute difference between the sum of the stated probabilities and the number of suspicious subscribers realized, and dividing by the number of observations. - **ET26.** The sharpened score is a strictly increasing transform; it preserves the ranking, and therefore the area under the curve, **exactly**. - **ET27.** The calibration mapping sees no feature other than the score of the model; its only input is the score and its two parameters are the slope and the shift. As long as the slope stays positive, the ranking does not change. - **ET28.** The mapping is learned from the **validation set**; the other two sets are printed for comparison. - **ET29.** The leaf probability of a tree is the share of suspicious subscribers among the training rows that fall into that leaf; for this reason the calibration error of the tree on the training set is zero by definition. - **ET30.** The baseline gives a number here too: the procedure that tells every subscriber the suspicious share of the training set is measured on calibration, and its area under the curve is 0.5000. #### Stated and Realized To measure calibration the probabilities are split into bins, and in each bin the mean probability the model states and the realized suspicious share of the subscribers in that bin are written side by side. This table is called the **reliability curve**, and it need not be drawn; it is read as numbers. ```python # evaluation.py — MODEL: the FICTIONAL subscriber table is rebuilt with the same seed. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): # MODEL: decision tree n = len(rows) p = sum(x["suspicious"] for x in rows) / n best = None if depth and n >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue gain = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / n if best is None or gain > best[0]: best = (gain, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def probability(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def linear(train, fields, w, epochs=300, step=0.5): # MODEL: gradient-based linear model mu = {f: sum(x[f] for x in train) / len(train) for f in fields} sd = {f: (sum((x[f] - mu[f]) ** 2 for x in train) / len(train)) ** 0.5 or 1.0 for f in fields} scale = lambda x: [(x[f] - mu[f]) / sd[f] for f in fields] coef, b = [0.0] * len(fields), 0.0 X = [scale(x) for x in train] y = [x["suspicious"] for x in train] sw = [(w if x["suspicious"] else 1.0) for x in train] total = sum(sw) for _ in range(epochs): gc, gb = [0.0] * len(fields), 0.0 for i, xi in enumerate(X): z = b + sum(coef[j] * xi[j] for j in range(len(coef))) e = (1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) - y[i]) * sw[i] gb += e for j in range(len(coef)): gc[j] += e * xi[j] b -= step * gb / total for j in range(len(coef)): coef[j] -= step * gc[j] / total def score(x): v = scale(x) z = b + sum(coef[j] * v[j] for j in range(len(coef))) return 1 / (1 + math.exp(-max(-30.0, min(30.0, z)))) return score SCORE_FN = linear(TRAIN, FIELDS, 1.0) # MODEL: probability-producing gradient-based linear model BINS = 5 def reliability(score, rows, n=BINS): # per bin: count, stated, realized bins = [[0, 0.0, 0] for _ in range(n)] for x in rows: p = score(x) k = bins[min(n - 1, int(p * n))] k[0], k[1], k[2] = k[0] + 1, k[1] + p, k[2] + x["suspicious"] return bins def calibration_error(score, rows, n=BINS): # bin-weighted mean absolute difference return sum(abs(k[1] - k[2]) for k in reliability(score, rows, n)) / len(rows) def print_reliability(name, score, rows): print(f"{name} (calibration error {calibration_error(score, rows):.4f})") for i, k in enumerate(reliability(score, rows)): if k[0]: print(f" {i / BINS:.1f}-{(i + 1) / BINS:.1f} subscribers {k[0]:>3} " f"said {k[1] / k[0]:.4f} realized {k[2] / k[0]:.4f} " f"difference {k[1] / k[0] - k[2] / k[0]:+.4f}") print_reliability("linear model, test set", SCORE_FN, TEST) ``` ``` linear model, test set (calibration error 0.0242) 0.0-0.2 subscribers 152 said 0.0891 realized 0.1053 difference -0.0161 0.2-0.4 subscribers 53 said 0.2837 realized 0.3019 difference -0.0182 0.4-0.6 subscribers 23 said 0.4918 realized 0.4783 difference +0.0136 0.6-0.8 subscribers 16 said 0.7079 realized 0.6250 difference +0.0829 0.8-1.0 subscribers 8 said 0.8698 realized 1.0000 difference -0.1302 ``` In the first three bins the difference is below the 0.02 level and the calibration error is 0.0242. The last two bins are more volatile, because there are a total of 24 subscribers there: in the 0.6-0.8 bin the model says 0.7079 while 0.6250 is realized, in the 0.8-1.0 bin it says 0.8698 while all eight of the eight subscribers turn out to be suspicious. A deviation of 0.13 in a share computed from eight subscribers is the volatility of the measure itself; as the number of bins grows this volatility grows, as it shrinks the table goes blind. #### Good Ranking, Bad Calibration If the scores are passed through a strictly increasing transform, no subscriber's position changes. Every measure that rests on the ranking stays constant; calibration does not. ```python def area_roc(score, rows): # rank sum averaging ties p = sorted(((score(x), x["suspicious"]) for x in rows)) P = sum(y for _, y in p) t, i = 0.0, 0 while i < len(p): j = i while j < len(p) and p[j][0] == p[i][0]: j += 1 t += (i + j + 1) / 2.0 * sum(y for _, y in p[i:j]) i = j return (t - P * (P + 1) / 2) / (P * (len(p) - P)) def sharpen(p, g=3.0): # strictly increasing, rank-preserving transform return p ** g / (p ** g + (1 - p) ** g) SHARP = lambda x: sharpen(SCORE_FN(x)) # MODEL: sharpened score TREE_D8 = tree(TRAIN, 8, FIELDS) # MODEL: depth 8 decision tree BASELINE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN) print(f"{'model':<25}{'ROC':>8}{'cal.error':>10}{'mean score':>11}{'suspicious share':>17}") for name, s in (("linear", SCORE_FN), ("sharpened", SHARP), ("tree d8", lambda x: probability(TREE_D8, x)), ("baseline (training rate)", lambda x: BASELINE)): print(f"{name:<25}{area_roc(s, TEST):>8.4f}{calibration_error(s, TEST):>10.4f}" f"{sum(s(x) for x in TEST) / len(TEST):>11.4f}" f"{sum(x['suspicious'] for x in TEST) / len(TEST):>17.4f}") print() print_reliability("sharpened score, test set", SHARP, TEST) ``` ``` model ROC cal.error mean score suspicious share linear 0.8058 0.0242 0.2309 0.2421 sharpened 0.8058 0.1512 0.1504 0.2421 tree d8 0.7391 0.1435 0.2586 0.2421 baseline (training rate) 0.5000 0.0198 0.2222 0.2421 sharpened score, test set (calibration error 0.1512) 0.0-0.2 subscribers 204 said 0.0199 realized 0.1569 difference -0.1369 0.2-0.4 subscribers 10 said 0.3033 realized 0.3000 difference +0.0033 0.4-0.6 subscribers 7 said 0.4756 realized 0.8571 difference -0.3816 0.6-0.8 subscribers 8 said 0.7001 realized 0.3750 difference +0.3251 0.8-1.0 subscribers 23 said 0.9509 realized 0.7391 difference +0.2117 ``` The first two rows of the first table are the proof of the claim: the area under the curve of the two models is the same, 0.8058, because both order the 252 subscribers in exactly the same sequence. The calibration error, on the other hand, rises from 0.0242 to 0.1512, to more than six times. The third row carries a separate warning. The mean score of the depth 8 tree is 0.2586 and the real suspicious share on the test set is 0.2421 — the difference between them is only 0.0165. A model that looks right in total gives an error of 0.1435 at the bin level. A mean that holds is not calibration; it hides overconfidence and underconfidence that cancel each other. The fourth row says what calibration alone does not measure. The baseline that tells every subscriber the suspicious share of the training set, 0.2222, has a calibration error of 0.0198, the smallest number in the table; its area under the curve is 0.5000, that is, its ranking is the same as random. A procedure that has learned nothing can come first in calibration. The two measures answer separate questions and both need to be written. The second table shows where the sharpened score breaks. In the most crowded bin the model says 0.0199 and 0.1569 is realized — more than seven times. At the top end it says 0.9509 and 0.7391 is realized. The model is overly timid on one side and overly confident on the other; the two errors cancel each other somewhat on average, at the bin level both stand. #### The Correction: A Single-Variable Mapping The breakage is not in the ranking but in the numeric value of the score. The correction should not touch the ranking either: a mapping that takes the score as input, is strictly increasing and has two parameters is enough. ```python def logit(p, e=1e-6): p = min(1 - e, max(e, p)) return math.log(p / (1 - p)) def mapping(score, rows, epochs=600, step=0.5): # MODEL: single-variable calibration mapping z = [logit(score(x)) for x in rows] y = [x["suspicious"] for x in rows] a, b = 1.0, 0.0 for _ in range(epochs): ga = gb = 0.0 for i, zi in enumerate(z): e = 1 / (1 + math.exp(-max(-30.0, min(30.0, a * zi + b)))) - y[i] ga, gb = ga + e * zi, gb + e a, b = a - step * ga / len(z), b - step * gb / len(z) return lambda x: 1 / (1 + math.exp(-max(-30.0, min(30.0, a * logit(score(x)) + b)))), a, b print(f"{'mapping learned on':<28}{'a':>8}{'b':>9}{'ROC':>8}{'cal.error':>10}") print(f"{'no correction':<28}{'-':>8}{'-':>9}{area_roc(SHARP, TEST):>8.4f}" f"{calibration_error(SHARP, TEST):>10.4f}") for name, rows in (("train", TRAIN), ("validation", VAL), ("test", TEST)): d, a, b = mapping(SHARP, rows) print(f"{name:<28}{a:>8.4f}{b:>9.4f}{area_roc(d, TEST):>8.4f}" f"{calibration_error(d, TEST):>10.4f}") TREE_FN = lambda x: probability(TREE_D8, x) d8, a8, b8 = mapping(TREE_FN, TRAIN) print(f"\ntree d8 calibration error: train {calibration_error(TREE_FN, TRAIN):.4f}, " f"validation {calibration_error(TREE_FN, VAL):.4f}, test {calibration_error(TREE_FN, TEST):.4f}") print(f"mapping learned on train: a {a8:.4f}, b {b8:.4f}, calibration error on test " f"{calibration_error(d8, TEST):.4f}") ``` ``` mapping learned on a b ROC cal.error no correction - - 0.8058 0.1512 train 0.3340 0.0015 0.8058 0.0244 validation 0.3013 -0.3695 0.8058 0.0463 test 0.3038 -0.0043 0.8058 0.0208 tree d8 calibration error: train 0.0000, validation 0.1624, test 0.1435 mapping learned on train: a 1.0002, b 0.0001, calibration error on test 0.1435 ``` The slope of the mapping is around 0.30 on all three sets: as far as the sharpening pushed the scores toward the extremes, the mapping takes that much back. The area under the curve is 0.8058 on all four rows — as long as the slope is positive the ranking does not change and the correction does not move any subscriber from its place. The calibration error falls from 0.1512 to 0.0463 with the validation set, that is, to a third. The mapping learned from the test set gives 0.0208; this number is not a performance but the result of the correction having been learned from the set it is measured on, and the 0.0255 difference between it and 0.0463 is not an estimate but an optimism. The training row comes out well on this set, 0.0244; because the linear model does not fit the training set tightly, its probabilities there are not overconfident either. The reason for the rule is in the last two lines. The calibration error of the depth 8 tree is 0.0000 on the training set and 0.1435 on the test set — the leaf probability of the tree is the training share of that leaf, so on the training set the stated and the realized are equal by definition. The mapping learned from training finds a slope of 1.0002 and a shift of 0.0001 and carries the error on the test set from 0.1435 to 0.1435: it has seen nothing to correct, because there is nothing to correct in the set it looked at. #### Summary - Calibration is the difference between the stated probability and the realized share and is measured by splitting the probabilities into bins; the calibration error of the linear model on the test set is 0.0242. - A strictly increasing transform preserves the ranking: the area under the curve of the sharpened score is also 0.8058, while its calibration error is 0.1512, more than six times. - It is not enough for the mean score to be close to the real share: the mean of the depth 8 tree is 0.2586, the real share is 0.2421, and its error at the bin level is 0.1435. The baseline that says a constant 0.2222, on the other hand, is the best in calibration with 0.0198 and the worst in the area under the curve with 0.5000. - The single-variable mapping learned from validation lowers the calibration error from 0.1512 to 0.0463 and leaves the area under the curve at 0.8058; because the slope is positive, no position in the ranking changes. - The calibration error of the tree on the training set is 0.0000 by definition; the mapping learned from there converges to the identity mapping and leaves the error of 0.1435 on the test set as it is. #### Next Step In four lessons each of the measures was for classification and all of them came out of a single operation: **counting**. The confusion matrix was four counts, precision and recall two ratios, the curves a pair of counts per threshold, calibration a comparison of counts per bin. Because the target was a class, right and wrong were two discrete boxes. In the same fictional set, when the target is the mean consumption of period 3, there is no box left to count: the model says 22.40, the truth is 26.15, and this estimate is neither right nor wrong. The next lesson turns the measure into **distance** and rebuilds the baseline as the training mean. ### Regression Metrics Source: https://academia.sh/en/courses/supervised-learning/regression-metrics The shift of the measure from counting to distance when the target is a number, and two members of the same family trading places on three measures: the linear model fitted to squared error leads on mean squared error with 22.3164, the one fitted to absolute error leads on mean absolute error with 3.6178. The baseline is the train mean (19.7665 m³) and the coefficient of determination is defined exactly relative to this baseline. 13 rows of the test set carry 37.59% of the squared error and 17.71% of the absolute error; when those 13 rows are trimmed, the ranking reverses on both measures. The number of candidates tried is three and no hyperparameter was swept. The previous four lessons built the measure for classification. The confusion matrix, precision, recall, the threshold and calibration all came out of a single operation: **counting**. How many examples fell into the right class, how many went to a false positive, how many were realized in a probability bin. There was something to count, because the prediction and the truth either coincided or did not. When the target is a number, there is no such thing as coinciding. For a subscriber who consumes 24.80 m³, saying 24.79 versus saying 11.20 is not a difference in count but a difference in **distance**, and how that distance is to be summed is a decision. This lesson measures that decision: two members of the same family, on the same held-out set, trade places on two measures. - **ET36.** The target is the subscriber's **mean consumption** (m³), computed from the readings spread over three periods. The set is the fictional subscriber table of the Introduction to Machine Learning course, the seed is 20260218, the split is 756/252/252. - **ET37.** The baseline is the **mean of the train set**: the same number is stated for every row. This is the regression counterpart of the "state the most frequent class" baseline in classification, and it does not learn. - **ET38.** The features stay fixed throughout the course: five region indicators, household size, satisfaction, the number of readings and the zero-reading indicator. The only thing that changes is the model. - **ET39.** The mean absolute error (MAE) is the mean of the absolute values of the errors, the mean squared error (MSE) is the mean of their squares. The MSE is in the square of the unit of the target; when its square root is taken it returns to the unit. - **ET40.** The coefficient of determination (R²) is computed as $1 - \text{MSE}_{\text{model}} / \text{MSE}_{\text{baseline}}$ and the denominator is **the baseline of ET37**. If the baseline changes, R² changes. - **ET41.** Both linear models use the same features and the same form; the only place they part is **which error they minimize during training**. Both are models. - **ET42.** The number of candidates tried in this lesson is **three** and no hyperparameter was swept; the difference inflated by the tuning budget is measured in `07`. #### Three Measures and the Baseline That Does Not Learn ```python # COMMON — MODEL: The FICTIONAL subscriber table of the Introduction to Machine Learning course # is rebuilt with the same seed; in this lesson the target is a number, not a class: # mean_consumption (m3). import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol = sum(v) / len(v), max(v) - min(v) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["area_north", "area_south", "area_east", "area_west", "area_central", "household", "satisfaction", "period", "zero_readings"] def mae(t, s): return sum(abs(t[i] - x["mean_consumption"]) for i, x in enumerate(s)) / len(s) def mse(t, s): return sum((t[i] - x["mean_consumption"]) ** 2 for i, x in enumerate(s)) / len(s) BASELINE = sum(x["mean_consumption"] for x in TRAIN) / len(TRAIN) BASELINE_MSE = mse([BASELINE] * len(TEST), TEST) def determination(t, s): return 1 - mse(t, s) / BASELINE_MSE print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") print(f"train mean {BASELINE:.4f} m3, test mean " f"{sum(x['mean_consumption'] for x in TEST) / len(TEST):.4f} m3") print(f"baseline: MAE {mae([BASELINE] * len(TEST), TEST):.4f} " f"MSE {BASELINE_MSE:.4f} R2 {determination([BASELINE] * len(TEST), TEST):.4f}") ``` ``` train 756, validation 252, test 252 train mean 19.7665 m3, test mean 19.9234 m3 baseline: MAE 5.0806 MSE 42.6718 R2 0.0000 ``` The baseline states 19.7665 m³ and is off by 5.0806 m³ on average. The same error summed as squares is 42.6718; its square root is 6.53 m³. The gap between the two numbers comes from the measure itself: the sum of squares counts large errors more heavily than small ones, so when the error distribution is skewed the squared measure gives a larger number than the absolute one. The R² of the baseline is exactly zero by definition — the baseline explains no variation relative to itself. That the baseline is the **train** mean is not a detail but a necessity. The test set's own mean is not a prediction but a number read from the rows to be measured; making it the baseline would mean counting a procedure that has looked at the test set as the baseline. A procedure that does not learn must also see only the train set. #### Two Members of the Same Family The model is a linear model: each feature is multiplied by a coefficient and the products are summed. The only difference between the two members is which error they minimize during training. The first minimizes the squared error and solves the coefficients by least squares. The second minimizes the absolute error; it does this by lowering, in every round, the weight of the rows with large errors and rebuilding the same solution. ```python def solve(A, b): n = len(A) M = [list(A[i]) + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda r: abs(M[r][i])) M[i], M[p] = M[p], M[i] for r in range(n): if r != i: f = M[r][i] / M[i][i] for c in range(i, n + 1): M[r][c] -= f * M[i][c] return [M[i][n] / M[i][i] for i in range(n)] def weighted_least_squares(s, weights, penalty=1e-6): d = len(FIELDS) + 1 A = [[0.0] * d for _ in range(d)] b = [0.0] * d for x, g in zip(s, weights): v = [1.0] + [float(x[a]) for a in FIELDS] for i in range(d): b[i] += g * v[i] * x["mean_consumption"] for j in range(d): A[i][j] += g * v[i] * v[j] for i in range(1, d): A[i][i] += penalty return solve(A, b) def predict(w, x): return w[0] + sum(w[i + 1] * x[a] for i, a in enumerate(FIELDS)) def fit_absolute(s, rounds=40, floor_weight=0.05): w = weighted_least_squares(s, [1.0] * len(s)) for _ in range(rounds): g = [1.0 / max(abs(predict(w, x) - x["mean_consumption"]), floor_weight) for x in s] w = weighted_least_squares(s, g) return w REGION_MEAN = {} for x in TRAIN: REGION_MEAN.setdefault(x["region_name"], []).append(x["mean_consumption"]) REGION_MEAN = {k: sum(v) / len(v) for k, v in REGION_MEAN.items()} W_SQUARED, W_ABSOLUTE = weighted_least_squares(TRAIN, [1.0] * len(TRAIN)), fit_absolute(TRAIN) CANDIDATES = [("baseline", [BASELINE] * len(TEST)), ("regional mean", [REGION_MEAN[x["region_name"]] for x in TEST]), ("fitted to squared error", [predict(W_SQUARED, x) for x in TEST]), ("fitted to absolute error", [predict(W_ABSOLUTE, x) for x in TEST])] print(f"{'model':<26} {'MAE':>8} {'MSE':>9} {'R2':>8}") for name, t in CANDIDATES: print(f"{name:<26} {mae(t, TEST):>8.4f} {mse(t, TEST):>9.4f} {determination(t, TEST):>8.4f}") for measure, fn, direction in (("MAE", mae, 1), ("MSE", mse, 1), ("R2", determination, -1)): s = sorted(CANDIDATES[1:], key=lambda p: direction * fn(p[1], TEST)) print(f"{measure} ranking: " + ", ".join(f"{i + 1}) {a}" for i, (a, _) in enumerate(s))) ``` ``` model MAE MSE R2 baseline 5.0806 42.6718 0.0000 regional mean 4.0374 28.7760 0.3256 fitted to squared error 3.6504 22.3164 0.4770 fitted to absolute error 3.6178 22.3987 0.4751 MAE ranking: 1) fitted to absolute error, 2) fitted to squared error, 3) regional mean MSE ranking: 1) fitted to squared error, 2) fitted to absolute error, 3) regional mean R2 ranking: 1) fitted to squared error, 2) fitted to absolute error, 3) regional mean ``` The last three lines are the answer of the lesson. The mean absolute error reads the model fitted to absolute error as first: 3.6178 against 3.6504, a gap of 0.0326 m³. The mean squared error reads **another model** as first: 22.3164 against 22.3987, a gap of 0.0823. **Three candidates, two measures, two separate firsts.** No calculation is wrong; the two measures sum the same error list in two separate ways and produce two separate rankings. The regional mean is third on both measures and passes the baseline by a clear margin on both. Most of the difference the family adds is there: from 42.6718 the baseline drops to 28.7760 with the regional mean, and the linear model brings it down to 22.3164. The region alone provides 0.3256 of the explained variation, the remaining eight features provide 0.1514. The MSE difference of the two linear models is 0.0823. When the square roots are taken, this means 4.7240 m³ against 4.7327 m³; the typical error of the two models parts by 0.0087 m³. Because the raw value of the measure is in the square of the unit of the target, 0.0823 looks large to the eye, yet when returned to the unit it is of the same size class as the 0.0326 difference in the mean absolute error. Taking the square root of a squared measure before interpreting it is the cheapest way to see the true scale of the difference. #### Which Measure Weights Which Error The source of the trade is the weight the two measures give to large errors. The way to measure it is to sort the errors from largest to smallest and count which share of the total comes from which rows. ```python ERRORS = {name: sorted((abs(t[i] - x["mean_consumption"]) for i, x in enumerate(TEST)), reverse=True) for name, t in CANDIDATES[2:]} print(f"{'model':<26} {'top13 MAE':>10} {'top13 MSE':>10} " f"{'half MAE':>9} {'half MSE':>9}") for name, h in ERRORS.items(): def half(fn): total, running, count = sum(fn(q) for q in h), 0.0, 0 for q in h: running, count = running + fn(q), count + 1 if 2 * running >= total: return count print(f"{name:<26} {sum(h[:13]) / sum(h):>10.4f} " f"{sum(q * q for q in h[:13]) / sum(q * q for q in h):>10.4f} " f"{half(lambda q: q):>9} {half(lambda q: q * q):>9}") largest = sorted(range(len(TEST)), key=lambda i: -abs(CANDIDATES[2][1][i] - TEST[i]["mean_consumption"]))[:13] rest = [i for i in range(len(TEST)) if i not in set(largest)] subset = [TEST[i] for i in rest] print(f"\nafter trimming 13 rows ({len(subset)} rows)") for name, t in CANDIDATES[2:]: rest_pred = [t[i] for i in rest] print(f"{name:<26} MAE {mae(rest_pred, subset):.4f} MSE {mse(rest_pred, subset):.4f}") print(f"true values of the 13 trimmed rows: " f"{[TEST[i]['mean_consumption'] for i in largest[:4]]}") ``` ``` model top13 MAE top13 MSE half MAE half MSE fitted to squared error 0.1771 0.3759 58 22 fitted to absolute error 0.1838 0.3962 57 21 after trimming 13 rows (239 rows) fitted to squared error MAE 3.1674 MSE 14.6846 fitted to absolute error MAE 3.1186 MSE 14.3617 true values of the 13 trimmed rows: [43.48, 39.49, 36.53, 32.18] ``` **13** of the 252 rows of the test set, that is, five percent, carry 37.59% of the squared error; the same 13 rows carry only 17.71% of the absolute error. The same distinction shows up at the half of the total as well: half of the absolute error is spread over 58 rows, while half of the squared error is gathered in **22 rows**. The squared measure is read effectively from twenty-odd rows in a set of 252 rows. When these 13 rows are trimmed, the ranking reverses on both measures: the model fitted to absolute error moves ahead on both the MAE (3.1186 against 3.1674) and the MSE (14.3617 against 14.6846). The whole MSE advantage of the model fitted to squared error is in those 13 trimmed rows. The true values of the trimmed rows are 43.48, 39.49, 36.53, 32.18 m³ — the highest consumptions of the set. The model that minimizes the squared error accepts being off a little more on the remaining 239 rows in order to chase these extremes; the model that minimizes the absolute error does the opposite. What follows from this is that choosing a measure is a decision about **cost**. If, in a set of suspected illicit cases, one subscriber being off by 18 m³ and ten subscribers being off by 1.8 m³ each do not produce the same field visit cost, a measure that sums the two situations into the same number makes you choose the wrong model. #### The Coefficient of Determination Does Not Give a Third Ranking The R2 column gives **exactly the same** ranking as the MSE column, and this is not a coincidence. The definition is $1 - \text{MSE}_{\text{model}} / \text{MSE}_{\text{baseline}}$; the denominator is the same constant for all models (42.6718). Dividing by a constant cannot change a ranking. R² is not a third measure but **the form of the squared error read relative to the baseline**: 1 − 22.3164 / 42.6718 = 0.4770. What it adds is not a ranking but a **unit**. The MSE 22.3164 by itself does not say whether it is large or small; R² reads the same number as 0.4770, "it erased nearly half of the baseline's error". For this reason, when R² is written, **relative to which baseline** it was computed must also be written. Here the baseline is the mean of the train set. Had the denominator been computed from the test set's own mean, it would have been 42.6471 and R² would have come out as 0.4767 instead of 0.4770; the difference is small, because the two means are close to each other (19.7665 against 19.9234). When the train and test means are far apart, this difference does not stay small and the two R² values become incomparable. The number of candidates tried in this lesson is **three**: the regional mean, the linear model fitted to squared error, the linear model fitted to absolute error. No hyperparameter was swept in any of them, so no number in the table is inflated by the search. How the same table changes under a budget is the subject of `07`. #### Summary - When the target is a number, the measure moves from counting to distance and how the distance is to be summed is a decision; the baseline is the train mean (19.7665 m³; MAE 5.0806, MSE 42.6718). - Two members of the same family trade places on two measures: the model fitted to absolute error is first on the MAE with 3.6178, the model fitted to squared error is first on the MSE with 22.3164. - 13 rows of the test set carry 37.59% of the squared error and 17.71% of the absolute error; half of the squared error comes from 22 rows, half of the absolute error from 58 rows. - When those 13 rows are trimmed, the ranking reverses on both measures (3.1186 against 3.1674 and 14.3617 against 14.6846); the whole advantage of the squared measure is in those rows. - The coefficient of determination produces no third ranking; because its denominator is constant for all models, it is the form of the mean squared error read relative to the baseline and cannot be written without a baseline. #### Next Step All the numbers of this lesson were read from a single split: 756 rows of train, 252 rows of test. The gap between the two linear models came out as 0.0326 on the MAE and 0.0823 on the MSE. These are small numbers and so far there is no measure of fluctuation next to them — whether another split would give the same ranking is not known. The next lesson treats the split itself as a tuning choice: into how many folds the remaining data is split, how much the estimate fluctuates, how many trainings are done as the number of folds grows, and why the extreme that leaves a single row in each fold is not always better. ### Cross-Validation Source: https://academia.sh/en/courses/supervised-learning/cross-validation Measuring the number of folds itself: when k is swept from 2 to 10, the mean of the estimate stays between 0.8115 and 0.8204, the true number on the test set is 0.8056 and the closest estimate to the truth comes from k=4. Leave-one-out gives 0.8175, 0.0119 away from the truth, but for this it processes 1,008 trainings and 1,015,056 rows — 111.89 times k=10. Its per-fold spread is 0.3863 and this number is not uncertainty but the correct/incorrect scatter of a one-row fold; because none of the 1,008 folds contains both classes, precision and recall cannot be computed per fold. On the other hand, the dependence of leave-one-out on the shuffle is exactly zero. All the numbers of the previous lesson were read from a single split. The difference between the two linear models came out as 0.0326 on the mean absolute error and 0.0823 on the mean squared error, and there was no measure of fluctuation next to these numbers. The question of whether another split would give the same ranking was not answered, because there was a single number at hand. Cross-validation closes this gap: the remaining data is split into folds, each fold in turn becomes the measurement set, and instead of a single number a few numbers and their spread are obtained. The procedure itself and the discipline of preparing within the fold were established in the Data Preparation and Feature Engineering course; here they are assumed and not repeated. What this lesson measures is **k itself**: as the number of folds grows, how much the estimate fluctuates, how many trainings are done in return, and why the choice at the extreme — leaving a single row in each fold — is not always better. - **ET43.** The pool is the train and validation sets of the Introduction to Machine Learning course: 1,008 rows. The 252 rows of the test set do not enter any fold and are used only once, to read the true number. - **ET44.** The model is a linear classifier fitted to the 0/1 label by least squares; the output is cut at the 0.5 threshold. The reason it is small in scale is that it will be trained 1,008 times, and **this is the model**, not an imitation of a library. - **ET45.** The model has no learned preparation step; the only thing learned within a fold is the model itself. The measure of the preparation discipline belongs to the Data Preparation and Feature Engineering course. - **ET46.** The folds are assigned by the remainder of the row number divided by k in the shuffled pool; this way each row is a measurement row exactly once and the fold sizes part by at most one row. - **ET47.** The unit of cost is the **number of trainings** and the **number of rows processed** (the total size of the training sets of the folds). These units do not change from run to run. - **ET48.** The per-fold spread is the mean distance of the folds from their own mean and is the measure of variability of the Data Analytics curriculum. It is not the uncertainty of the estimate. - **ET49.** The number of candidates tried in this lesson is **one**: the number of folds is not a hyperparameter but part of the measurement procedure and does not change the model itself. #### What Moves When k Is Swept ```python # COMMON — MODEL: The FICTIONAL subscriber table of the Introduction to Machine Learning course # is rebuilt with the same seed; the model is a linear classifier fitted to the 0/1 label by # least squares. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TEST = K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_east", "area_west", "area_south"] D = len(FIELDS) + 1 def solve(A, b): n = len(A) M = [list(A[i]) + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda r: abs(M[r][i])) M[i], M[p] = M[p], M[i] for r in range(n): if r != i: f = M[r][i] / M[i][i] for c in range(i, n + 1): M[r][c] -= f * M[i][c] return [M[i][n] / M[i][i] for i in range(n)] def fit(s, penalty=1e-4): A = [[0.0] * D for _ in range(D)] b = [0.0] * D for x in s: v = [1.0] + [float(x[a]) for a in FIELDS] y = x["suspicious"] for i in range(D): b[i] += v[i] * y for j in range(i, D): A[i][j] += v[i] * v[j] for i in range(D): for j in range(i): A[i][j] = A[j][i] if i: A[i][i] += penalty return solve(A, b) def accuracy(w, s): d = 0 for x in s: p = w[0] + sum(w[i + 1] * x[a] for i, a in enumerate(FIELDS)) d += (1 if p >= 0.5 else 0) == x["suspicious"] return d / len(s) POOL = K[:1008] TRUE_NUMBER = accuracy(fit(POOL), TEST) c = int(sum(x["suspicious"] for x in POOL) * 2 > len(POOL)) BASELINE = sum(x["suspicious"] == c for x in TEST) / len(TEST) print(f"pool {len(POOL)} rows, test {len(TEST)} rows") print(f"true number {TRUE_NUMBER:.4f}, baseline {BASELINE:.4f}") ``` ``` pool 1008 rows, test 252 rows true number 0.8056, baseline 0.7579 ``` The true number is the number of the model learned from the whole pool on the test set: 0.8056, 0.0477 above the baseline. The job of cross-validation is to estimate this number **without looking at the test set**. Below, k is swept from 2 to 10 and the last row holds the choice at the extreme: 1,008 folds, that is, a single row in each fold. ```python def cross_validate(data, k): results = [] for i in range(k): inner = [x for j, x in enumerate(data) if j % k != i] held = [x for j, x in enumerate(data) if j % k == i] results.append((accuracy(fit(inner), held), len(inner), held)) return results def mean(v): return sum(v) / len(v) def spread(v): o = mean(v) return (sum((q - o) ** 2 for q in v) / len(v)) ** 0.5 print(f"{'k':>5} {'mean':>9} {'spread':>8} {'trainings':>9} {'train rows':>14} " f"{'rows processed':>14} {'difference':>10} {'two-class folds':>16}") for k in list(range(2, 11)) + [1008]: results = cross_validate(POOL, k) s = [q[0] for q in results] both = sum(1 for q in results if 0 < sum(x["suspicious"] for x in q[2]) < len(q[2])) print(f"{k:>5} {mean(s):>9.4f} {spread(s):>8.4f} {k:>9} {results[0][1]:>14} " f"{1008 * (k - 1):>14} {mean(s) - TRUE_NUMBER:>+10.4f} {both:>16}") ``` ``` k mean spread trainings train rows rows processed difference two-class folds 2 0.8135 0.0099 2 504 1008 +0.0079 2 3 0.8165 0.0014 3 672 2016 +0.0109 3 4 0.8115 0.0278 4 756 3024 +0.0060 4 5 0.8185 0.0239 5 806 4032 +0.0129 5 6 0.8145 0.0237 6 840 5040 +0.0089 6 7 0.8175 0.0243 7 864 6048 +0.0119 7 8 0.8204 0.0359 8 882 7056 +0.0149 8 9 0.8175 0.0286 9 896 8064 +0.0119 9 10 0.8194 0.0280 10 907 9072 +0.0139 10 1008 0.8175 0.3863 1008 1007 1015056 +0.0119 0 ``` The mean column wanders between 0.8115 and 0.8204: the total movement among nine different values of k is **0.0089**. The difference column shows that all of them are **positive**; cross-validation systematically gives a number above the true number, and this is in the same direction as the +0.0149 deviation measured with in-fold preparation in the Data Preparation and Feature Engineering course. The cause is not leakage but the pool being seen again in every round. The expected pattern is that the training set grows as k grows and the estimate approaches the true number. The train rows column shows this growth: from 504 to 907, then to 1,007. But the difference column does not follow this pattern. The closest estimate to the truth comes from **k=4** (+0.0060) and the farthest one from **k=8** (+0.0149). The ranking in between comes more from the chance of how the folds were divided than from the size of the training set. Raising k from 4 to 10 did not improve the estimate; it made the procedure do three times more training. The spread column cannot be read alone either. 0.0014 at k=3, 0.0359 at k=8 — a difference of about twenty-six times. The spread of three numbers and the spread of ten numbers are each noisy on their own; the spread is not an estimate of uncertainty but a rough summary of the distribution across the folds. The irregularity of the difference column comes from the superposition of two tendencies, and the two pull in opposite directions. The first is **pessimism**: when k is small, the model learns from fewer rows than the model that will be put to use. At k=2 the training is 504 rows, that is, half of the rows seen by the model that will finally be built; the measured number is the number of that weak model. The second is **optimism**: each row of the pool is in training in k−1 of the k rounds and in measurement in one, and the same 1,008 rows are reused from start to finish. As k grows, the first tendency weakens and the second stays as it is. Their sum does not cancel at any value of k, and which k will fall close to the truth cannot be known in advance — marking the closest row in the table is possible only after the test set has been read, and that removes the very reason for cross-validation to exist. #### The Price Paid by the Extreme Choice The last row is leave-one-out: each row in turn becomes the measurement set alone, and the model is learned from the remaining 1,007 rows. The estimate is 0.8175, 0.0119 above the truth. This number is **exactly the same** as the number given by k=7 and k=9, and it is farther from the truth than the one given by k=4. Its cost is read in three columns. 1,008 trainings are done and a total of **1,015,056 rows** are processed; 111.89 times the 9,072 rows processed by k=10. The difference gained is a 0.0020 shift in the estimate. The second price is in the spread column: 0.3863. This number is not uncertainty. In a one-row fold the accuracy can only be 0 or 1, so the spread necessarily settles at $\sqrt{p(1-p)}$; here $p = 0.8175$ and $\sqrt{0.8175 \times 0.1825} = 0.3863$. The number says nothing about the stability of the model; it only rewrites the mean itself. The third price is in the last column: **the number of two-class folds is zero**. A one-row fold carries a single label, so in that fold the counts of true positives, false positives and false negatives cannot all be found at once. Precision, recall, the F1 score and the area under the curve **cannot be computed per fold**. When k is between 2 and 10, every fold carries both classes and all of these measures can be computed per fold. If one is working with a measure other than accuracy, leave-one-out ceases to be an option. #### Dependence on the Shuffle While measuring the effect of the number of folds, one thing was held fixed: the order of the pool. Since the fold assignment depends on the row number, had the pool been shuffled differently, the estimate would also have changed. Below, the same values of k are measured again with three separate shuffles. ```python print(f"{'k':>5} " + " ".join(f"{'seed ' + str(i + 1):>9}" for i in range(3)) + f" {'spread':>9}") for k in (2, 4, 10, 1008): m = [] for i in range(3): h = shuffle(POOL, SEED + 300 + i * 17) m.append(mean([q[0] for q in cross_validate(h, k)])) print(f"{k:>5} " + " ".join(f"{q:>9.4f}" for q in m) + f" {spread(m):>9.4f}") ``` ``` k seed 1 seed 2 seed 3 spread 2 0.8145 0.8175 0.8125 0.0020 4 0.8194 0.8194 0.8185 0.0005 10 0.8194 0.8164 0.8164 0.0014 1008 0.8175 0.8175 0.8175 0.0000 ``` The zero in the last row is the real advantage of leave-one-out. Because the folds are single rows, shuffling does not change the folds; the same 1,008 folds, even if not in the same order, are the same set. The estimate is **completely deterministic** and the choice of seed cannot move the result. The same measurement at k=2 wanders between 0.8125 and 0.8175 across the three shuffles, a spread of 0.0020. This also says when leave-one-out is worth what it costs: when data is scarce, while the share of a single shuffle in moving the estimate is large, and when the measure is a number that can be collected from a single row per fold, like accuracy. With a pool of 1,008 rows none of these conditions holds, and a choice between k=4 and k=10 gives the same estimate tens of thousands of times cheaper. The size of the shuffle spreads is also a warning: the 0.0020 movement of k=2 with the shuffle is one fifteenth of the 0.0326 difference measured between the two linear models in the previous lesson. When comparing two models with cross-validation, the difference between them has to be larger than this movement. #### Summary - When the number of folds is swept from 2 to 10, the estimate stays between 0.8115 and 0.8204; the total movement is 0.0089 and all nine values are above the true number (0.8056). - The estimate does not improve regularly with k: the closest number to the truth comes from k=4 (+0.0060), the farthest from k=8 (+0.0149); despite the growth of the training set, the ranking comes from the chance of how the folds were divided. - Leave-one-out gives 0.8175 and processes 1,015,056 rows with 1,008 trainings; in return for 111.89 times the work of k=10, it moves the estimate by 0.0020. - The per-fold spread of leave-one-out is 0.3863 and this number is not uncertainty but the value $\sqrt{p(1-p)}$; moreover, because none of the 1,008 folds carries both classes, precision, recall and the area under the curve cannot be computed per fold. - The real advantage of leave-one-out is that it is deterministic: its spread over three separate shuffles is exactly zero, that of k=2 is 0.0020. #### Next Step The number of candidates tried in this lesson is one. The model was built in a single form, no hyperparameter was swept, and the number of folds was treated as part of the measurement procedure, not as a tuning choice. The rule of the course has not been paid yet: a model's number cannot be read without writing how many candidates were tried to find that number. The next lesson does that calculation — three search procedures are compared with the same budget, and as the number of candidates rises from 1 to 200, the gap that opens between the best number on the validation set and the number of the same candidate on the test set is printed. The difference added by changing the family and the difference inflated by the budget stand side by side there. ### Hyperparameter Search Source: https://academia.sh/en/courses/supervised-learning/tuning-variable-search The lesson where the rule of the course is paid: with the same budget of 50 candidates, the best candidate found by grid, random and model-based search and the number of that candidate on the test set are printed side by side. As the number of candidates rises from 1 to 200, the best number on the validation set climbs from 0.7937 to 0.8452, the number of the same candidate on the test set goes from 0.7778 to 0.7976, and the inflation between them opens from +0.0159 to +0.0476. The untuned default candidate already gives 0.8452 on the validation set, that is, 200 candidates found nothing. At budgets of 50 and 200 the single-split decision rule beats the linear model on the test set: the family difference is −0.0159 and −0.0040 while the inflation by the budget is +0.0516 and +0.0476. The previous lesson measured a single model, and the number of candidates tried was one. The rule of the course is this: the number of a model cannot be read without writing how many candidates were tried to find that number; a number whose tuning budget is not written counts as unmeasured. This lesson is where that rule is paid. The measure has two parts. The first is **the search itself**: when the same space and the same budget are given, which candidate do the three procedures — grid, random and model-based — find. The second is **what the budget inflates**: as the number of candidates rises from 1 to 200, what opens between the best number on validation and the test number of that candidate. This inflation is finally put side by side with the difference added by changing the family. - **ET50.** The tuning space has four axes: the penalty coefficient ($10^{-3}$–$10^{3}$, continuous on a log scale), the decision threshold (0.20–0.60, continuous), the polynomial degree (1 or 2) and the number of columns taken from a fixed ordering (3, 6 or 9). All three searches work in the **same** space. - **ET51.** The choice of candidate is read **only from the validation set** (252 rows); the test set enters no choice and is read only to report the number of the selected candidate. - **ET52.** The budget unit is **the number of candidates trained and measured on the validation set**; the proposals that model-based search tries on the surrogate model need no training and are not counted. Because the product of the discrete axes (2 degrees × 3 columns = 6) divides the budget, grid search **cannot fill** the given budget **exactly**. - **ET53.** Model-based search builds a **surrogate model** from the validation numbers of the candidates already tried: a distance-weighted mean plus an exploration bonus proportional to the distance to the nearest candidate tried. This is a model too. - **ET54.** The second family is the **single-split decision rule**: one column, one threshold, one direction. It is structurally weaker than the linear model and is searched with the same budget. - **ET55.** The baseline carries over from the Introduction to Machine Learning course: the procedure that says the most frequent class, 0.7579 on the test set. The number of candidates is printed at the end of the lesson and repeated candidates are counted once. #### Same Space, Same Budget, Three Searches ```python # COMMON — MODEL: The FICTIONAL subscriber table of the Introduction to Machine Learning course # is rebuilt with the same seed; the model is a linear classifier fitted to the 0/1 label by # least squares. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] ORDER = ["mean_consumption", "zero_readings", "volatility", "household", "satisfaction", "period", "area_east", "area_west", "area_south"] SQUARED = {"mean_consumption", "volatility", "household", "satisfaction", "period"} COLUMNS = (3, 6, 9) def select_fields(j, degree): a = ORDER[:j] return a + ([q + "^2" for q in a if q in SQUARED] if degree == 2 else []) def value(x, name): return float(x[name[:-2]]) ** 2 if name.endswith("^2") else float(x[name]) def solve(A, b): n = len(A) M = [list(A[i]) + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda r: abs(M[r][i])) M[i], M[p] = M[p], M[i] for r in range(n): if r != i: f = M[r][i] / M[i][i] for c in range(i, n + 1): M[r][c] -= f * M[i][c] return [M[i][n] / M[i][i] for i in range(n)] def fit(s, fields, penalty): d = len(fields) + 1 A = [[0.0] * d for _ in range(d)] b = [0.0] * d for x in s: v = [1.0] + [value(x, a) for a in fields] for i in range(d): b[i] += v[i] * x["suspicious"] for j in range(i, d): A[i][j] += v[i] * v[j] for i in range(d): for j in range(i): A[i][j] = A[j][i] if i: A[i][i] += penalty return solve(A, b) def accuracy(w, fields, threshold, s): d = 0 for x in s: p = w[0] + sum(w[i + 1] * value(x, a) for i, a in enumerate(fields)) d += (1 if p >= threshold else 0) == x["suspicious"] return d / len(s) CACHE, TRIED = {}, set() def evaluate(candidate): penalty, threshold, degree, j = candidate k = (round(penalty, 9), degree, j) if k not in CACHE: CACHE[k] = (fit(TRAIN, select_fields(j, degree), penalty), select_fields(j, degree)) TRIED.add(("linear", k, round(threshold, 9))) w, fields = CACHE[k] return accuracy(w, fields, threshold, VAL), accuracy(w, fields, threshold, TEST) def best_of(candidates, measure): best = None for a in candidates: d, s = measure(a) if best is None or d > best[0]: best = (d, s, a) return best c = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN)) BASELINE = sum(x["suspicious"] == c for x in TEST) / len(TEST) DEFAULT_VAL, DEFAULT_TEST = evaluate((1.0, 0.5, 1, 9)) print(f"baseline {BASELINE:.4f}") print(f"untuned default candidate: validation {DEFAULT_VAL:.4f}, test {DEFAULT_TEST:.4f}") ``` ``` baseline 0.7579 untuned default candidate: validation 0.8452, test 0.7976 ``` The untuned default candidate — penalty 1, threshold 0.5, degree 1, nine columns — gives 0.8452 on the validation set. The three searches below and the 200-candidate sweep will try to beat this number. ```python def random_candidate(r): return (10 ** (-3 + 6 * r()), 0.20 + 0.40 * r(), 1 + int(r() * 2), COLUMNS[int(r() * 3)]) def grid(budget): top = (1, 1) for n_pen in range(1, budget + 1): for n_thr in range(1, budget + 1): if n_pen * n_thr * 6 <= budget and \ (n_pen * n_thr, -abs(n_pen - n_thr), n_pen) > \ (top[0] * top[1], -abs(top[0] - top[1]), top[0]): top = (n_pen, n_thr) n_pen, n_thr = top return [(10 ** (-3 + 6 * (i + 0.5) / n_pen), 0.20 + 0.40 * (t + 0.5) / n_thr, degree, j) for i in range(n_pen) for t in range(n_thr) for degree in (1, 2) for j in COLUMNS] def random_search(budget, seed=SEED + 4100): r = rng(seed) return [random_candidate(r) for _ in range(budget)] def distance(a, b): return (((math.log10(a[0]) - math.log10(b[0])) / 6) ** 2 + ((a[1] - b[1]) / 0.4) ** 2 + (a[2] - b[2]) ** 2 + ((a[3] - b[3]) / 6) ** 2) def model_based(budget, seed=SEED + 4200, initial=8, proposals=40): r, history = rng(seed), [] for _ in range(min(initial, budget)): a = random_candidate(r) history.append((a, evaluate(a)[0])) while len(history) < budget: top_score, top_candidate = None, None for _ in range(proposals): a = random_candidate(r) w = [1.0 / (distance(a, g) + 0.01) for g, _ in history] score = (sum(q * s for q, (_, s) in zip(w, history)) / sum(w) + 0.02 * min(distance(a, g) for g, _ in history) ** 0.5) if top_score is None or score > top_score: top_score, top_candidate = score, a history.append((top_candidate, evaluate(top_candidate)[0])) return [g for g, _ in history] print(f"{'search':<16} {'candidates':>10} {'validation':>10} {'test':>8} {'inflation':>9} setting") for name, fn in (("grid", grid), ("random", random_search), ("model-based", model_based)): A = fn(50) d, s, a = best_of(A, evaluate) print(f"{name:<16} {len(A):>10} {d:>10.4f} {s:>8.4f} {d - s:>+9.4f} " f"penalty {a[0]:.3g}, threshold {a[1]:.3f}, degree {a[2]}, columns {a[3]}") print(f"\n{'budget':>6} {'validation':>10} {'test':>8} {'inflation':>9}") R = random_search(200) for b in (1, 2, 5, 10, 20, 50, 100, 200): d, s, a = best_of(R[:b], evaluate) print(f"{b:>6} {d:>10.4f} {s:>8.4f} {d - s:>+9.4f}") ``` ``` search candidates validation test inflation setting grid 48 0.8452 0.7937 +0.0516 penalty 5.62, threshold 0.500, degree 2, columns 9 random 50 0.8373 0.7857 +0.0516 penalty 0.00481, threshold 0.495, degree 1, columns 9 model-based 50 0.8452 0.8016 +0.0437 penalty 0.00103, threshold 0.540, degree 1, columns 9 budget validation test inflation 1 0.7937 0.7778 +0.0159 2 0.7937 0.7778 +0.0159 5 0.8254 0.8056 +0.0198 10 0.8294 0.7976 +0.0317 20 0.8294 0.7976 +0.0317 50 0.8373 0.7857 +0.0516 100 0.8373 0.7857 +0.0516 200 0.8452 0.7976 +0.0476 ``` In the first table grid search can use **48** candidates against a budget of 50. The product of the discrete axes is six, only 8 points are left for the continuous axes, and these split 4×2: four values on the penalty axis, that is, six orders of magnitude are represented by four points. Random search with the same budget tries 50 distinct values on that axis. The test numbers of the three searches are 0.7937, 0.7857 and 0.8016 — not the same as the validation ranking. Model-based search is ahead on both sets and has the smallest inflation (+0.0437), but the test difference among the three is 0.0159, that is, four rows; there is not enough evidence to attribute a difference of this size to the nature of the searches. The main finding is in the second table. The validation column **rises by definition**: as the budget grows, the best number is the maximum of a growing set and cannot shrink. From 0.7937 to 0.8452, a total of +0.0515. The test column does not follow this climb: from 0.7778 to 0.7976, a total of +0.0198. **62%** of the climb never passes to the test set, and in the inflation column it opens from +0.0159 to +0.0476. The cause is that the validation set is 252 rows. Each row is worth 0.0040; when the real difference between two candidates is below this resolution, which rows happened to fall into that set decides the winner. When 200 candidates are tried, the candidate with the highest number is not the best one but **the one that best fit the noise of the validation set** — the form, in the tuning space, of the selection bias measured in the Statistical Analysis and Exploratory Data Analysis course. The point of comparison is in the opening block: the untuned default candidate already gave 0.8452 on the validation set. 200 candidates **could not beat** this number, and where they reached it they stopped at the same point (0.7976) on the test set as well. The search produced zero gain in return for 200 trainings; we can say this only once the test set is read. #### What the Family Adds and What the Budget Inflates The second claim of the course turns into a number here: the difference added by changing the family is often smaller than the difference inflated by the tuning budget. The second family is structurally weaker — a single-split decision rule: one column, one threshold, one direction. ```python THRESHOLD_TABLE = {} for a in ORDER: d = sorted({x[a] for x in TRAIN}) THRESHOLD_TABLE[a] = (d[1:] if len(d) < 12 else [d[int(i * len(d) / 12)] for i in range(1, 12)]) def stump_evaluate(candidate): a, v, direction = candidate TRIED.add(("stump",) + candidate) def predict(x): return int((x[a] >= v) == (direction == 1)) return (sum(predict(x) == x["suspicious"] for x in VAL) / len(VAL), sum(predict(x) == x["suspicious"] for x in TEST) / len(TEST)) def stump_random(budget, seed=SEED + 4300): r, out = rng(seed), [] for _ in range(budget): a = ORDER[int(r() * len(ORDER))] out.append((a, THRESHOLD_TABLE[a][int(r() * len(THRESHOLD_TABLE[a]))], 1 + int(r() * 2))) return out print(f"{'budget':>6} {'stump test':>10} {'linear test':>11} {'family difference':>17} " f"{'budget inflation':>16}") for b in (1, 10, 50, 200): gd, gs, ga = best_of(stump_random(b), stump_evaluate) dd, ds, da = best_of(R[:b], evaluate) print(f"{b:>6} {gs:>10.4f} {ds:>11.4f} {ds - gs:>+17.4f} {dd - ds:>+16.4f}") gd, gs, ga = best_of(stump_random(200), stump_evaluate) print(f"\nstump budget 200: single split '{ga[0]}', validation {gd:.4f}, test {gs:.4f}") print(f"number of candidates tried in this lesson {len(TRIED)}") ``` ``` budget stump test linear test family difference budget inflation 1 0.7659 0.7778 +0.0119 +0.0159 10 0.7381 0.7976 +0.0595 +0.0317 50 0.8016 0.7857 -0.0159 +0.0516 200 0.8016 0.7976 -0.0040 +0.0476 stump budget 200: single split 'mean_consumption', validation 0.8254, test 0.8016 number of candidates tried in this lesson 352 ``` The family difference column changes sign across the budget: +0.0119 at budget 1, +0.0595 at budget 10, it turns **negative** at budget 50 (−0.0159) and stays negative at budget 200 (−0.0040). The structurally stronger family **could not beat** the structurally weaker family on the test set at two budgets. The 0.0595 at budget 10 comes from the stump happening to get stuck on a bad candidate; there the stump is at 0.7381, below the baseline. The last column carries the difference inflated by the budget on the same rows: +0.0159, +0.0317, +0.0516, +0.0476. At budget 200, while the family difference is −0.0040, the budget inflation is +0.0476 — the inflation is **nearly twelve times** the family difference and in the opposite direction. Changing the family of the model did not move the number on this set; what moved it was how many candidates were tried, and that movement was not real. The rule the stump found rests on a single column: a single split on mean consumption, 0.8016 on the test set. The nine-column, penalized linear models of this lesson could not beat a single threshold. The number of candidates tried is **352** and it is smaller than the sum of the given budgets. There are 299 distinct candidates in the linear family: one default candidate, 148 candidates of the three searches and 150 candidates added by the budget sweep. In the stump family, on the other hand, there are 53 distinct candidates despite a budget of 200, because the whole tuning space of that family has only 74 points (37 thresholds, two directions). This is why budgets 50 and 200 give the same number: the space is saturated and enlarging the budget does not produce new candidates. The tuning budget of a family cannot be read without writing the size of its space. The reported number is the test-set number of the candidate selected at budget 200: **0.7976**, 0.0397 above the baseline. #### Summary - With the same budget of 50 candidates, grid search can use only 48 (the product of the discrete axes divides the budget); the test numbers of the three searches are 0.7937, 0.7857 and 0.8016. - As the budget rises from 1 to 200, the best number on the validation set climbs by definition from 0.7937 to 0.8452, the test number of the same candidate goes from 0.7778 to 0.7976, and the inflation opens from +0.0159 to +0.0476. - The untuned default candidate already gives 0.8452 on the validation set; 200 candidates could not beat this number and the search produced zero gain in return for the budget spent. - At budgets of 50 and 200 the single-split decision rule beats the linear model on the test set: the family difference is −0.0159 and −0.0040 while the budget inflation is +0.0516 and +0.0476, that is, the inflation is nearly twelve times the family difference. - 352 distinct candidates were tried in this lesson (299 in the linear family, 53 in the stump family); the budget of the stump family was 200 but the search saturated because its space has only 74 points, and enlarging the budget did not produce new candidates. The reported number is the test number of the candidate selected at budget 200 (0.7976). #### Next Step All the numbers of this lesson are a single ratio: in how many of 252 rows the right decision was made. Which regions, which tariff steps, which household sizes and how many readings the rows in the denominator of that ratio came from was never asked. The next lesson asks the question by splitting the error into these slices, and measures that the number of bad slices found also grows by chance as the number of slices increases. ### Error Analysis Source: https://academia.sh/en/courses/supervised-learning/error-analysis Breaking the 51 errors on the test set down by region, tariff step, household size and number of readings: in two of the fourteen measurable slices the model falls below the baseline, and the central region holds 23% of the rows while carrying 35.3% of the errors. As the number of slices goes from 14 to 41 and from 41 to 50, the number of bad slices found goes from 2 to 6 and from 6 to 9 — but when the same errors are distributed randomly over the rows, an average of 4.41, 14.26 and 18.46 bad slices appears, that is, the observed number is below chance in all three sets. The course wrap-up gathers the family, the number of candidates and the margin above the baseline of twenty lessons in a single table. The previous lesson ended with a number: of the 252 rows of the test set, the right decision in 201 and the wrong one in 51. Those 51 rows were squeezed into a single ratio throughout the course; which regions, which tariff steps, which household sizes and how many readings they came from was not asked. Error analysis asks this question: is the error distributed equally over the set, or is it concentrated in one **slice**. The question has a trap and this lesson measures that too: if enough slices are defined, some of them will look bad **by chance**. - **ET56.** The model is in the form chosen by the previous lesson: degree 1, nine columns. The penalty and the threshold are chosen with the same seed, with a 200-candidate search and **only from the validation set**. The analysis is done on the test set and no choice is made from it. - **ET57.** The slice axes are four: region (five values), tariff step (four intervals with the limits of 10, 25 and 40 m³), household size (1–2, 3–4, 5–6) and number of readings (1, 2, 3). A slice that does not carry **at least 10 rows** counts as not measurable; the threshold is fixed before the measurement. - **ET58.** The baseline inside a slice is the same procedure as the baseline of the course: saying the most frequent class of the training set. Because the class composition changes, the baseline varies from slice to slice. - **ET59.** The measure of chance is the **reshuffling** of the right/wrong decisions across the rows; the procedure was established in the Statistical Analysis and Exploratory Data Analysis course and is not repeated. - **ET60.** This lesson measures the **performance difference** across slices. What the difference does to whom and for whom the model is built is left to the Model Evaluation, Interpretability and Ethics course. #### Where the Error Concentrates ```python # COMMON — MODEL: The set and the model come from the FICTIONAL subscriber table of the # Introduction to Machine Learning course; the tuning is chosen with the 200-candidate search of # the previous lesson and from the validation set only. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(items, seed): r, s = rng(seed), list(range(len(items))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [items[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "zero_readings", "volatility", "household", "satisfaction", "period", "area_east", "area_west", "area_south"] def solve(A, b): n = len(A) M = [list(A[i]) + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda r: abs(M[r][i])) M[i], M[p] = M[p], M[i] for r in range(n): if r != i: f = M[r][i] / M[i][i] for c in range(i, n + 1): M[r][c] -= f * M[i][c] return [M[i][n] / M[i][i] for i in range(n)] def fit(s, penalty): d = len(FIELDS) + 1 A = [[0.0] * d for _ in range(d)] b = [0.0] * d for x in s: v = [1.0] + [float(x[a]) for a in FIELDS] for i in range(d): b[i] += v[i] * x["suspicious"] for j in range(i, d): A[i][j] += v[i] * v[j] for i in range(d): for j in range(i): A[i][j] = A[j][i] if i: A[i][i] += penalty return solve(A, b) CACHE = {} def decisions(candidate, subset): penalty, threshold = candidate if penalty not in CACHE: CACHE[penalty] = fit(TRAIN, penalty) w = CACHE[penalty] return [1 if w[0] + sum(w[i + 1] * float(x[a]) for i, a in enumerate(FIELDS)) >= threshold else 0 for x in subset] r = rng(SEED + 4100) CANDIDATES = [(10 ** (-3 + 6 * r()), 0.20 + 0.40 * r(), r(), r()) for _ in range(200)] CHOSEN, BEST = None, -1.0 for a in CANDIDATES: d = sum(k == x["suspicious"] for k, x in zip(decisions(a[:2], VAL), VAL)) / len(VAL) if d > BEST: BEST, CHOSEN = d, a[:2] MAJORITY = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN)) CORRECT = [k == x["suspicious"] for k, x in zip(decisions(CHOSEN, TEST), TEST)] BASELINE = [MAJORITY == x["suspicious"] for x in TEST] OVERALL = sum(CORRECT) / len(TEST) print(f"chosen candidate: penalty {CHOSEN[0]:.4g}, threshold {CHOSEN[1]:.4f}, " f"validation {BEST:.4f}") print(f"test set {len(TEST)} rows, model {OVERALL:.4f}, " f"baseline {sum(BASELINE) / len(TEST):.4f}, errors {len(TEST) - sum(CORRECT)}") ``` ``` chosen candidate: penalty 0.00372, threshold 0.5540, validation 0.8452 test set 252 rows, model 0.7976, baseline 0.7579, errors 51 ``` Below, the 51 errors are broken down along four axes. The number of the model, the baseline in the slice, their difference and two shares are printed: the **error share** (how many of the errors are there) and the **row share** (how many of the rows are there). When the two shares part, the error is concentrated. ```python AXIS = {"region": lambda x: x["region_name"], "household": lambda x: ("1-2", "3-4", "5-6")[min(2, (x["household"] - 1) // 2)], "readings": lambda x: str(x["period"]), "tariff": lambda x: ("<10", "10-25", "25-40", ">=40")[ 0 if x["mean_consumption"] < 10 else 1 if x["mean_consumption"] < 25 else 2 if x["mean_consumption"] < 40 else 3]} MIN_ROWS = 10 def slice_up(sets): d = {} for axes in sets: for i, x in enumerate(TEST): d.setdefault((axes, tuple(AXIS[e](x) for e in axes)), []).append(i) return d SINGLE = [("region",), ("tariff",), ("household",), ("readings",)] ERRORS = len(TEST) - sum(CORRECT) print(f"{'axis':<9} {'slice':<7} {'n':>4} {'model':>7} {'baseline':>8} {'difference':>10} " f"{'error share':>11} {'row share':>9}") for k, idx in sorted(slice_up(SINGLE).items()): if len(idx) < MIN_ROWS: print(f"{k[0][0]:<9} {k[1][0]:<7} {len(idx):>4} not measurable (below {MIN_ROWS} rows)") continue m = sum(CORRECT[i] for i in idx) / len(idx) t = sum(BASELINE[i] for i in idx) / len(idx) h = sum(1 for i in idx if not CORRECT[i]) print(f"{k[0][0]:<9} {k[1][0]:<7} {len(idx):>4} {m:>7.4f} {t:>8.4f} {m - t:>+10.4f} " f"{h / ERRORS:>11.3f} {len(idx) / len(TEST):>9.3f}") ``` ``` axis slice n model baseline difference error share row share household 1-2 73 0.8219 0.7534 +0.0685 0.255 0.290 household 3-4 143 0.7972 0.7692 +0.0280 0.569 0.567 household 5-6 36 0.7500 0.7222 +0.0278 0.176 0.143 readings 1 22 0.8636 0.8636 +0.0000 0.059 0.087 readings 2 63 0.7460 0.7619 -0.0159 0.314 0.250 readings 3 167 0.8084 0.7425 +0.0659 0.627 0.663 region central 58 0.6897 0.6552 +0.0345 0.353 0.230 region east 36 0.6944 0.4722 +0.2222 0.216 0.143 region north 68 0.8088 0.8235 -0.0147 0.255 0.270 region south 58 0.8448 0.8276 +0.0172 0.176 0.230 region west 32 1.0000 1.0000 +0.0000 0.000 0.127 tariff 10-25 194 0.8402 0.8402 +0.0000 0.608 0.770 tariff 25-40 46 0.6522 0.4565 +0.1957 0.314 0.183 tariff <10 11 0.6364 0.6364 +0.0000 0.078 0.044 tariff >=40 1 not measurable (below 10 rows) ``` Of fifteen slices fourteen are measurable; in the highest tariff step there is **a single subscriber** and the table states that nothing can be said about that step. There are **two negatives** in the difference column: in the north region the model is 0.8088 against a baseline of 0.8235; among two-reading subscribers it is 0.7460 against 0.7619. The overall number of 0.7976 says nothing in these two slices. Concentration is read in the comparison of the two shares. The central region holds 0.230 of the rows while carrying **0.353** of the errors, that is, its error share is one and a half times its row share; the 25–40 m³ step is 0.314 against 0.183. The west region holds 0.127 of the rows and **produces no errors at all** — there is not a single subscriber with a suspicious label there, the baseline is 1.0000 and the best the model can do is to stay equal to the baseline. The east region and the 25–40 m³ step are both the places where the model gains most over the baseline (+0.2222 and +0.1957) and the places that are **worst** in absolute terms (0.6944 and 0.6522). #### As the Number of Slices Grows If the axes are crossed, the number of slices grows quickly and so does the number of "bad slices" to be found. The question is how much of this growth is real. The measure is this: the right and wrong decisions are reshuffled across the rows — the number of errors stays fixed at 51, the tie to the slice structure is broken — and the count is repeated. ```python PAIRS = SINGLE + [("region", "tariff"), ("region", "readings"), ("tariff", "household"), ("household", "readings")] TRIPLES = PAIRS + [("region", "tariff", "readings")] def count_bad(d, correct): measurable = bad = 0 for idx in d.values(): if len(idx) < MIN_ROWS: continue measurable += 1 if (sum(correct[i] for i in idx) / len(idx) < sum(BASELINE[i] for i in idx) / len(idx)): bad += 1 return measurable, bad print(f"{'slice set':<14} {'defined':>8} {'measurable':>10} {'bad':>5} " f"{'chance mean':>12} {'chance max':>11} {'baseline>model':>14}") for name, subset in (("single axis", SINGLE), ("two axes", PAIRS), ("three axes", TRIPLES)): d = slice_up(subset) measurable, bad = count_bad(d, CORRECT) chance = [count_bad(d, shuffle(CORRECT, SEED + 8000 + i))[1] for i in range(200)] above = sum(1 for idx in d.values() if len(idx) >= MIN_ROWS and sum(BASELINE[i] for i in idx) / len(idx) > OVERALL) print(f"{name:<14} {len(d):>8} {measurable:>10} {bad:>5} " f"{sum(chance) / len(chance):>12.2f} {max(chance):>11} {above:>14}") worst = min(((sum(CORRECT[i] for i in idx) / len(idx) - sum(BASELINE[i] for i in idx) / len(idx), k, len(idx)) for k, idx in slice_up(PAIRS).items() if len(idx) >= MIN_ROWS)) print(f"\nworst two-axis slice: {worst[1][0]} = {worst[1][1]}, " f"{worst[2]} rows, difference {worst[0]:+.4f}") ``` ``` slice set defined measurable bad chance mean chance max baseline>model single axis 15 14 2 4.41 6 5 two axes 62 41 6 14.26 18 16 three axes 92 50 9 18.46 23 21 worst two-axis slice: ('region', 'tariff') = ('east', '10-25'), 17 rows, difference -0.1176 ``` The number of bad slices goes from 2 to 6 and from 6 to 9. Read alone, this looks like proof that the model falls flat in more places as the slices are thinned. The chance columns say the opposite: when the same 51 errors are distributed **randomly** over the rows, an average of 4.41, 14.26 and 18.46 bad slices appears, and the observed number is **below** this average in all three sets. The last column says where this comes from. The overall number of the model is 0.7976; in a slice whose baseline is above this number, a model whose errors are randomly distributed falls below the baseline. Such slices number 5 on a single axis, 16 on two axes and 21 on three axes, and the chance averages sit right next to these numbers. The "bad slice" criterion is really counting **the slices whose baseline is higher than the overall performance**. The observed staying below chance is in the model's favor. This is the form here of the rule measured in the Statistical Analysis and Exploratory Data Analysis course — a number is not a finding until it is measured how often the same number would be found by chance. What the differences found mean for whom is left to the Model Evaluation, Interpretability and Ethics course. #### Summary - Of fifteen slices fourteen are measurable; in the highest tariff step there is a single subscriber and nothing can be said about that step. - In two slices the model is below the baseline: the north region (0.8088 against 0.8235) and two-reading subscribers (0.7460 against 0.7619). - The central region holds 0.230 of the rows and carries 0.353 of the errors; the east region is both the slice that gains most over the baseline (+0.2222) and the one that is worst in absolute terms (0.6944). - As the number of slices goes from 14 to 50, the bad slices go from 2 to 9, but with randomly distributed errors they go from 4.41 to 18.46; the observed number is below chance in every set. #### Course Wrap-Up The account of twenty lessons is in a single table, and every number was read from its own lesson. The second column is, in every lesson, the test-set number of the candidate **selected from the validation set**. The classification rows are accuracy, with a baseline of 0.7579; the regression rows are root mean squared error (m³), with a baseline of 6.5324. | Lesson | Number of the family | Candidates tried | Above baseline | |---|---|---|---| | Linear 01 | 5.3096 m³ | 1 | +1.2228 | | Linear 02 | 5.3775 m³ | 8 | +1.1549 | | Linear 03 | 5.2954 m³ | 18 | +1.2370 | | Linear 04 | 0.7937 | 9 | +0.0357 | | Linear 05 | 0.8056 | 8 | +0.0477 | | Nonlinear 01 | 0.7937 | 24 | +0.0357 | | Nonlinear 02 | 0.7817 | 20 | +0.0238 | | Nonlinear 03 | 0.7857 | 17 | +0.0278 | | Nonlinear 04 | 0.7857 | 12 | +0.0278 | | Nonlinear 05 | 0.7897 | 31 | +0.0317 | | Nonlinear 06 | 0.7738 | 1,279 | +0.0159 | | Nonlinear 07 | 0.7778 | 3 | +0.0199 | | Evaluation 01 | 0.8016 | 49 | +0.0437 | | Evaluation 02 | 0.8016 | 5 | +0.0437 | | Evaluation 03 | 181 cost | 99 | −185 cost | | Evaluation 04 | 0.0242 calibration | 5 | −0.0044 | | Evaluation 05 | 0.4770 explained variation | 3 | +0.4770 | | Evaluation 06 | 0.8056 | 1 | +0.0477 | | Evaluation 07 | 0.7976 | 200 | +0.0397 | | Evaluation 08 | 0.7976 | 1 | +0.0397 | Three rows are read in their own units. The unit of Evaluation 03 is cost and smaller is better: 181 units, 185 below the baseline's 366. Evaluation 04 is the calibration error and is the only negative row in the table: the baseline that says a constant number is better at 0.0198 than the model's 0.0242, that is, the learned model could not beat the unlearned procedure there. The rule of the course is the third column: **the number of a model cannot be read without writing how many candidates were tried to find that number; a number whose tuning budget is not written counts as unmeasured.** When the column is summed, the number of candidates tried in the course comes out: **1,793**. The second claim of the course is read from the table. The best of the classification rows is 0.8056, the worst 0.7738: all seven families are inside a band of **0.0318 points**. The difference inflated by the tuning budget is the size of this band — in Evaluation 07, **0.0476** between validation and test at budget 200, and in the lesson with the table's highest budget, 1,279 candidates, **0.0318** between two choices. **The whole difference among the seven families is equal to the difference produced by the tuning budget of a single lesson.** In twenty lessons seven model families were tried and the budget was written next to every number. But all of these measurements had a common support: **the label**. The baseline said the most frequent label, the measure compared the predicted label with its truth, the search looked at the labels on the validation set. In a set where there is no label, what is learned — even whether anything is learned — is read **against which baseline**, was never asked. The next course, Unsupervised Learning, opens with this question.