--- title: 'Error Analysis' source: 'https://academia.sh/en/courses/supervised-learning/error-analysis' course: 'Supervised Learning' language: en updated: '2026-10-21T05:03:30+00:00' license: 'CC BY-SA 4.0' objectives: - 'Breaks the error down by region, tariff step, household size and number of readings and counts in which slice the model falls below the baseline' - 'Measures where the error concentrates by comparing the error share of a slice with its row share' - 'Shows by reshuffling that the number of bad slices found grows by chance as the number of slices grows' --- # Error Analysis Breaking the 51 errors on the test set down by region, tariff step, household size and number of readings: in two of the fourteen measurable slices the model falls below the baseline, and the central region holds 23% of the rows while carrying 35.3% of the errors. As the number of slices goes from 14 to 41 and from 41 to 50, the number of bad slices found goes from 2 to 6 and from 6 to 9 — but when the same errors are distributed randomly over the rows, an average of 4.41, 14.26 and 18.46 bad slices appears, that is, the observed number is below chance in all three sets. The course wrap-up gathers the family, the number of candidates and the margin above the baseline of twenty lessons in a single table. The previous lesson ended with a number: of the 252 rows of the test set, the right decision in 201 and the wrong one in 51. Those 51 rows were squeezed into a single ratio throughout the course; which regions, which tariff steps, which household sizes and how many readings they came from was not asked. Error analysis asks this question: is the error distributed equally over the set, or is it concentrated in one **slice**. The question has a trap and this lesson measures that too: if enough slices are defined, some of them will look bad **by chance**. - **ET56.** The model is in the form chosen by the previous lesson: degree 1, nine columns. The penalty and the threshold are chosen with the same seed, with a 200-candidate search and **only from the validation set**. The analysis is done on the test set and no choice is made from it. - **ET57.** The slice axes are four: region (five values), tariff step (four intervals with the limits of 10, 25 and 40 m³), household size (1–2, 3–4, 5–6) and number of readings (1, 2, 3). A slice that does not carry **at least 10 rows** counts as not measurable; the threshold is fixed before the measurement. - **ET58.** The baseline inside a slice is the same procedure as the baseline of the course: saying the most frequent class of the training set. Because the class composition changes, the baseline varies from slice to slice. - **ET59.** The measure of chance is the **reshuffling** of the right/wrong decisions across the rows; the procedure was established in the Statistical Analysis and Exploratory Data Analysis course and is not repeated. - **ET60.** This lesson measures the **performance difference** across slices. What the difference does to whom and for whom the model is built is left to the Model Evaluation, Interpretability and Ethics course. ## Where the Error Concentrates ```python # COMMON — MODEL: The set and the model come from the FICTIONAL subscriber table of the # Introduction to Machine Learning course; the tuning is chosen with the 200-candidate search of # the previous lesson and from the validation set only. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(items, seed): r, s = rng(seed), list(range(len(items))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [items[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "zero_readings", "volatility", "household", "satisfaction", "period", "area_east", "area_west", "area_south"] def solve(A, b): n = len(A) M = [list(A[i]) + [b[i]] for i in range(n)] for i in range(n): p = max(range(i, n), key=lambda r: abs(M[r][i])) M[i], M[p] = M[p], M[i] for r in range(n): if r != i: f = M[r][i] / M[i][i] for c in range(i, n + 1): M[r][c] -= f * M[i][c] return [M[i][n] / M[i][i] for i in range(n)] def fit(s, penalty): d = len(FIELDS) + 1 A = [[0.0] * d for _ in range(d)] b = [0.0] * d for x in s: v = [1.0] + [float(x[a]) for a in FIELDS] for i in range(d): b[i] += v[i] * x["suspicious"] for j in range(i, d): A[i][j] += v[i] * v[j] for i in range(d): for j in range(i): A[i][j] = A[j][i] if i: A[i][i] += penalty return solve(A, b) CACHE = {} def decisions(candidate, subset): penalty, threshold = candidate if penalty not in CACHE: CACHE[penalty] = fit(TRAIN, penalty) w = CACHE[penalty] return [1 if w[0] + sum(w[i + 1] * float(x[a]) for i, a in enumerate(FIELDS)) >= threshold else 0 for x in subset] r = rng(SEED + 4100) CANDIDATES = [(10 ** (-3 + 6 * r()), 0.20 + 0.40 * r(), r(), r()) for _ in range(200)] CHOSEN, BEST = None, -1.0 for a in CANDIDATES: d = sum(k == x["suspicious"] for k, x in zip(decisions(a[:2], VAL), VAL)) / len(VAL) if d > BEST: BEST, CHOSEN = d, a[:2] MAJORITY = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN)) CORRECT = [k == x["suspicious"] for k, x in zip(decisions(CHOSEN, TEST), TEST)] BASELINE = [MAJORITY == x["suspicious"] for x in TEST] OVERALL = sum(CORRECT) / len(TEST) print(f"chosen candidate: penalty {CHOSEN[0]:.4g}, threshold {CHOSEN[1]:.4f}, " f"validation {BEST:.4f}") print(f"test set {len(TEST)} rows, model {OVERALL:.4f}, " f"baseline {sum(BASELINE) / len(TEST):.4f}, errors {len(TEST) - sum(CORRECT)}") ``` ``` chosen candidate: penalty 0.00372, threshold 0.5540, validation 0.8452 test set 252 rows, model 0.7976, baseline 0.7579, errors 51 ``` Below, the 51 errors are broken down along four axes. The number of the model, the baseline in the slice, their difference and two shares are printed: the **error share** (how many of the errors are there) and the **row share** (how many of the rows are there). When the two shares part, the error is concentrated. ```python AXIS = {"region": lambda x: x["region_name"], "household": lambda x: ("1-2", "3-4", "5-6")[min(2, (x["household"] - 1) // 2)], "readings": lambda x: str(x["period"]), "tariff": lambda x: ("<10", "10-25", "25-40", ">=40")[ 0 if x["mean_consumption"] < 10 else 1 if x["mean_consumption"] < 25 else 2 if x["mean_consumption"] < 40 else 3]} MIN_ROWS = 10 def slice_up(sets): d = {} for axes in sets: for i, x in enumerate(TEST): d.setdefault((axes, tuple(AXIS[e](x) for e in axes)), []).append(i) return d SINGLE = [("region",), ("tariff",), ("household",), ("readings",)] ERRORS = len(TEST) - sum(CORRECT) print(f"{'axis':<9} {'slice':<7} {'n':>4} {'model':>7} {'baseline':>8} {'difference':>10} " f"{'error share':>11} {'row share':>9}") for k, idx in sorted(slice_up(SINGLE).items()): if len(idx) < MIN_ROWS: print(f"{k[0][0]:<9} {k[1][0]:<7} {len(idx):>4} not measurable (below {MIN_ROWS} rows)") continue m = sum(CORRECT[i] for i in idx) / len(idx) t = sum(BASELINE[i] for i in idx) / len(idx) h = sum(1 for i in idx if not CORRECT[i]) print(f"{k[0][0]:<9} {k[1][0]:<7} {len(idx):>4} {m:>7.4f} {t:>8.4f} {m - t:>+10.4f} " f"{h / ERRORS:>11.3f} {len(idx) / len(TEST):>9.3f}") ``` ``` axis slice n model baseline difference error share row share household 1-2 73 0.8219 0.7534 +0.0685 0.255 0.290 household 3-4 143 0.7972 0.7692 +0.0280 0.569 0.567 household 5-6 36 0.7500 0.7222 +0.0278 0.176 0.143 readings 1 22 0.8636 0.8636 +0.0000 0.059 0.087 readings 2 63 0.7460 0.7619 -0.0159 0.314 0.250 readings 3 167 0.8084 0.7425 +0.0659 0.627 0.663 region central 58 0.6897 0.6552 +0.0345 0.353 0.230 region east 36 0.6944 0.4722 +0.2222 0.216 0.143 region north 68 0.8088 0.8235 -0.0147 0.255 0.270 region south 58 0.8448 0.8276 +0.0172 0.176 0.230 region west 32 1.0000 1.0000 +0.0000 0.000 0.127 tariff 10-25 194 0.8402 0.8402 +0.0000 0.608 0.770 tariff 25-40 46 0.6522 0.4565 +0.1957 0.314 0.183 tariff <10 11 0.6364 0.6364 +0.0000 0.078 0.044 tariff >=40 1 not measurable (below 10 rows) ``` Of fifteen slices fourteen are measurable; in the highest tariff step there is **a single subscriber** and the table states that nothing can be said about that step. There are **two negatives** in the difference column: in the north region the model is 0.8088 against a baseline of 0.8235; among two-reading subscribers it is 0.7460 against 0.7619. The overall number of 0.7976 says nothing in these two slices. Concentration is read in the comparison of the two shares. The central region holds 0.230 of the rows while carrying **0.353** of the errors, that is, its error share is one and a half times its row share; the 25–40 m³ step is 0.314 against 0.183. The west region holds 0.127 of the rows and **produces no errors at all** — there is not a single subscriber with a suspicious label there, the baseline is 1.0000 and the best the model can do is to stay equal to the baseline. The east region and the 25–40 m³ step are both the places where the model gains most over the baseline (+0.2222 and +0.1957) and the places that are **worst** in absolute terms (0.6944 and 0.6522). ## As the Number of Slices Grows If the axes are crossed, the number of slices grows quickly and so does the number of "bad slices" to be found. The question is how much of this growth is real. The measure is this: the right and wrong decisions are reshuffled across the rows — the number of errors stays fixed at 51, the tie to the slice structure is broken — and the count is repeated. ```python PAIRS = SINGLE + [("region", "tariff"), ("region", "readings"), ("tariff", "household"), ("household", "readings")] TRIPLES = PAIRS + [("region", "tariff", "readings")] def count_bad(d, correct): measurable = bad = 0 for idx in d.values(): if len(idx) < MIN_ROWS: continue measurable += 1 if (sum(correct[i] for i in idx) / len(idx) < sum(BASELINE[i] for i in idx) / len(idx)): bad += 1 return measurable, bad print(f"{'slice set':<14} {'defined':>8} {'measurable':>10} {'bad':>5} " f"{'chance mean':>12} {'chance max':>11} {'baseline>model':>14}") for name, subset in (("single axis", SINGLE), ("two axes", PAIRS), ("three axes", TRIPLES)): d = slice_up(subset) measurable, bad = count_bad(d, CORRECT) chance = [count_bad(d, shuffle(CORRECT, SEED + 8000 + i))[1] for i in range(200)] above = sum(1 for idx in d.values() if len(idx) >= MIN_ROWS and sum(BASELINE[i] for i in idx) / len(idx) > OVERALL) print(f"{name:<14} {len(d):>8} {measurable:>10} {bad:>5} " f"{sum(chance) / len(chance):>12.2f} {max(chance):>11} {above:>14}") worst = min(((sum(CORRECT[i] for i in idx) / len(idx) - sum(BASELINE[i] for i in idx) / len(idx), k, len(idx)) for k, idx in slice_up(PAIRS).items() if len(idx) >= MIN_ROWS)) print(f"\nworst two-axis slice: {worst[1][0]} = {worst[1][1]}, " f"{worst[2]} rows, difference {worst[0]:+.4f}") ``` ``` slice set defined measurable bad chance mean chance max baseline>model single axis 15 14 2 4.41 6 5 two axes 62 41 6 14.26 18 16 three axes 92 50 9 18.46 23 21 worst two-axis slice: ('region', 'tariff') = ('east', '10-25'), 17 rows, difference -0.1176 ``` The number of bad slices goes from 2 to 6 and from 6 to 9. Read alone, this looks like proof that the model falls flat in more places as the slices are thinned. The chance columns say the opposite: when the same 51 errors are distributed **randomly** over the rows, an average of 4.41, 14.26 and 18.46 bad slices appears, and the observed number is **below** this average in all three sets. The last column says where this comes from. The overall number of the model is 0.7976; in a slice whose baseline is above this number, a model whose errors are randomly distributed falls below the baseline. Such slices number 5 on a single axis, 16 on two axes and 21 on three axes, and the chance averages sit right next to these numbers. The "bad slice" criterion is really counting **the slices whose baseline is higher than the overall performance**. The observed staying below chance is in the model's favor. This is the form here of the rule measured in the Statistical Analysis and Exploratory Data Analysis course — a number is not a finding until it is measured how often the same number would be found by chance. What the differences found mean for whom is left to the Model Evaluation, Interpretability and Ethics course. ## Summary - Of fifteen slices fourteen are measurable; in the highest tariff step there is a single subscriber and nothing can be said about that step. - In two slices the model is below the baseline: the north region (0.8088 against 0.8235) and two-reading subscribers (0.7460 against 0.7619). - The central region holds 0.230 of the rows and carries 0.353 of the errors; the east region is both the slice that gains most over the baseline (+0.2222) and the one that is worst in absolute terms (0.6944). - As the number of slices goes from 14 to 50, the bad slices go from 2 to 9, but with randomly distributed errors they go from 4.41 to 18.46; the observed number is below chance in every set. ## Course Wrap-Up The account of twenty lessons is in a single table, and every number was read from its own lesson. The second column is, in every lesson, the test-set number of the candidate **selected from the validation set**. The classification rows are accuracy, with a baseline of 0.7579; the regression rows are root mean squared error (m³), with a baseline of 6.5324. | Lesson | Number of the family | Candidates tried | Above baseline | |---|---|---|---| | Linear 01 | 5.3096 m³ | 1 | +1.2228 | | Linear 02 | 5.3775 m³ | 8 | +1.1549 | | Linear 03 | 5.2954 m³ | 18 | +1.2370 | | Linear 04 | 0.7937 | 9 | +0.0357 | | Linear 05 | 0.8056 | 8 | +0.0477 | | Nonlinear 01 | 0.7937 | 24 | +0.0357 | | Nonlinear 02 | 0.7817 | 20 | +0.0238 | | Nonlinear 03 | 0.7857 | 17 | +0.0278 | | Nonlinear 04 | 0.7857 | 12 | +0.0278 | | Nonlinear 05 | 0.7897 | 31 | +0.0317 | | Nonlinear 06 | 0.7738 | 1,279 | +0.0159 | | Nonlinear 07 | 0.7778 | 3 | +0.0199 | | Evaluation 01 | 0.8016 | 49 | +0.0437 | | Evaluation 02 | 0.8016 | 5 | +0.0437 | | Evaluation 03 | 181 cost | 99 | −185 cost | | Evaluation 04 | 0.0242 calibration | 5 | −0.0044 | | Evaluation 05 | 0.4770 explained variation | 3 | +0.4770 | | Evaluation 06 | 0.8056 | 1 | +0.0477 | | Evaluation 07 | 0.7976 | 200 | +0.0397 | | Evaluation 08 | 0.7976 | 1 | +0.0397 | Three rows are read in their own units. The unit of Evaluation 03 is cost and smaller is better: 181 units, 185 below the baseline's 366. Evaluation 04 is the calibration error and is the only negative row in the table: the baseline that says a constant number is better at 0.0198 than the model's 0.0242, that is, the learned model could not beat the unlearned procedure there. The rule of the course is the third column: **the number of a model cannot be read without writing how many candidates were tried to find that number; a number whose tuning budget is not written counts as unmeasured.** When the column is summed, the number of candidates tried in the course comes out: **1,793**. The second claim of the course is read from the table. The best of the classification rows is 0.8056, the worst 0.7738: all seven families are inside a band of **0.0318 points**. The difference inflated by the tuning budget is the size of this band — in Evaluation 07, **0.0476** between validation and test at budget 200, and in the lesson with the table's highest budget, 1,279 candidates, **0.0318** between two choices. **The whole difference among the seven families is equal to the difference produced by the tuning budget of a single lesson.** In twenty lessons seven model families were tried and the budget was written next to every number. But all of these measurements had a common support: **the label**. The baseline said the most frequent label, the measure compared the predicted label with its truth, the search looked at the labels on the validation set. In a set where there is no label, what is learned — even whether anything is learned — is read **against which baseline**, was never asked. The next course, Unsupervised Learning, opens with this question.