--- title: 'Validation Strategies' source: 'https://academia.sh/en/courses/model-ethics-and-interpretation/validation-strategies' course: 'Model Evaluation, Interpretability and Ethics' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures a performance number separately on the whole and on thirteen subgroups defined on four axes, and computes the gap between them' - 'Compares the group-based split with the row split: counts, as capacity grows, the difference inflated by the same subscriber landing in two sets' - 'Shows that leakage hides the subgroup gap by shrinking it, and writes next to every subgroup number how many subscribers it was read from' --- # Validation Strategies Measuring the same number separately on the whole and on thirteen subgroups, and testing whether the held-out set was truly held out: a depth-6 decision tree beats the baseline on the test set by 0.0198 yet falls below the baseline in five of the subgroups, and the gap among the twelve subgroups with an n above 25 comes out at 0.3576. The same subscriber landing in both sets in the 3,199-row reading table splits 416 subscribers across them and raises the difference above the baseline at depth 12 from 0.0108 to 0.1000; the group-based split undoes this 0.0892 inflation. The inflation corrupts not only the overall number but the gap as well: the subgroup gap that reads 0.1718 in the row split is 0.3846 in the group-based split. Across seven courses the baseline was written next to every number: a model was never read on its own, it was put side by side with the same-metric number of a procedure dumber than itself, and in the last course the fact that the system making the decision changes its own data was counted too. The one thing never asked is **for whom** the number was measured. All of those numbers were averages over subscribers, and an average may belong to none of the groups beneath it. The rule of this course is this: **a number cannot be read until it says for whom it was measured; performance whose subgroup is not written down has not been measured.** The rule is paid for in every lesson with three numbers — the number measured overall, the same number measured on subgroups, the **subgroup gap** between them — and, when a gap is seen, with the sentence that changes in the report. This lesson sets up the triple and the subgroups, but first it answers a question that takes priority: whether the held-out set was truly held out. - **ED1.** The dataset carries over unchanged from the seven courses and it is **fictional**: a 3,199-row reading table, 1,329 subscribers, three periods, five regions, seed 20260218. The label is fictional too — a subscriber carrying a suspicion of illicit use or a fault — and is not a real audit record. - **ED2.** At subscriber level 1,260 rows remain and are split **756/252/252**. The split, the metric (accuracy) and the baseline (the most frequent class of the training set) are inherited from the Introduction to Machine Learning course; the baseline on the test set is **0.7579**. The model is a **depth-6 decision tree**, written with the standard library. - **ED3.** The separation of the three sets and the time-ordered split were measured in the generalization topic of the Introduction to Machine Learning course and are not repeated here; the only thing added is the **group-based split**. - **ED4.** The subgroups are defined on **four axes** and stay fixed throughout the course: **region** (five values), **tariff tier** (the tier the mean consumption falls in: 10, 25, 40 m³), **household size** (1–2, 3–4, 5 and above), **reading count** (few: one or two periods; full: three periods). **13 subgroups** in total. - **ED5.** The subgroup definition is a decision, not a finding. **How many subgroups were looked at is written in every lesson**, because looking at many produces findings out of noise — this was measured in the Statistical Analysis and Exploratory Data Analysis course and is not repeated. - **ED6.** Next to every subgroup number is written **how many subscribers it was read from**; **a group whose n is below 25 does not enter the gap calculation**, because in 25 subscribers a single wrong count moves the number by 0.04. - **ED7.** In the row-level table every row is one **(subscriber, period)** reading, the label belongs to the subscriber, and the subscriber-level summaries are written onto the rows too. Depth is swept over **six candidates**. ## Thirteen Readings of the Same Number The first block generates the setup, defines the subgroups and prints the course's triple in a single table. The tree is a model: at each node it picks a threshold, and at the leaf it states the majority class. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def subgroups(x): # four axes, thirteen subgroups; fixed throughout the course return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M = tree(TRAIN, 6, FIELDS) def report_row(name, rows): base, acc = baseline(TRAIN, rows), accuracy(M, rows) print(f"{name:<14} {len(rows):>4} {base:>8.4f} {acc:>7.4f} {acc - base:>+8.4f}" f"{' n<25, not in gap' if len(rows) < 25 else ''}") return len(rows), base, acc print(f"subscribers {len(DATA)} -- 756/252/252 -- subgroups {len(SUBGROUPS)} (four axes)\n" f"{'subgroup':<14} {'n':>4} {'baseline':>8} {'tree':>7} {'diff':>8}") report_row("TOTAL", TEST) measured = {g: report_row(" ".join(g), [x for x in TEST if g in subgroups(x)]) for g in SUBGROUPS} scored = [v[2] for v in measured.values() if v[0] >= 25] below = [" ".join(g) for g, v in measured.items() if v[2] < v[1]] print(f"\nsubgroup gap ({len(scored)} subgroups): best {max(scored):.4f}, worst {min(scored):.4f}, " f"gap {max(scored) - min(scored):.4f}\nbelow baseline: {len(below)}/" f"{len(SUBGROUPS)} -- " + ", ".join(below)) ``` ``` subscribers 1260 -- 756/252/252 -- subgroups 13 (four axes) subgroup n baseline tree diff TOTAL 252 0.7579 0.7778 +0.0198 household 1-2 73 0.7534 0.7945 +0.0411 household 3-4 143 0.7692 0.7762 +0.0070 household 5+ 36 0.7222 0.7500 +0.0278 reading few 85 0.7882 0.7412 -0.0471 reading full 167 0.7425 0.7964 +0.0539 region central 58 0.6552 0.6724 +0.0172 region east 36 0.4722 0.6111 +0.1389 region north 68 0.8235 0.7794 -0.0441 region south 58 0.8276 0.8793 +0.0517 region west 32 1.0000 0.9688 -0.0312 tariff 10 12 0.6667 0.5833 -0.0833 n<25, not in gap tariff 25 193 0.8394 0.8290 -0.0104 tariff 40 47 0.4468 0.6170 +0.1702 subgroup gap (12 subgroups): best 0.9688, worst 0.6111, gap 0.3576 below baseline: 5/13 -- reading few, region north, region west, tariff 10, tariff 25 ``` The first row is the row that seven courses read: the model beats the baseline by **0.0198**. The thirteen rows beneath it are other numbers of the same metric for the same model, and the **gap** between the best and the worst is **0.3576** — eighteen times the difference read overall. The 0.7778 reported as performance means 0.6111 for a subscriber in the east region and 0.9688 for one in the west. The harsher number is on the last line: the model beats the baseline overall but falls **below the baseline in five of the thirteen subgroups**. In the 85 subscribers with few readings the baseline is 0.7882 and the model 0.7412 — the model is 4.7 points worse than the procedure that learns nothing. The difference measured overall contains all of these groups and shows none of them. The baseline column says why it is this way. None of the west's 32 test subscribers is suspicious; there, saying "none is suspicious" gives 1.0000, and no model can beat that. In the east and in the tariff 40 tier the suspicious share exceeds half, so the baseline drops below 0.50 and the model beats it most there. The subgroup gap also carries the group's own difficulty, which is why the baseline stands next to every number. The tariff 10 tier does not enter the gap because it holds 12 subscribers. ## The Same Subscriber in Two Sets This table assumes the held-out sets were truly separated. The raw form of the setup is not at **subscriber level but at reading level**: 3,199 rows for 1,260 subscribers, and a subscriber who produced three periods of readings has three rows, all with the same label. The moment the rows are shuffled and split, one row of the same subscriber falls into the training set and another into the test set. ```python ROWS = [] for x in DATA: for i, o in enumerate(READINGS[x["id"]]): ROWS.append(dict(x, reading=o, position=i)) ROW_FIELDS = ["reading", "position"] + FIELDS ROW_TRAIN, ROW_VAL, ROW_TEST = three_way_split(shuffle(ROWS, SEED + 93000)) # row split ids = shuffle([x["id"] for x in DATA], SEED + 93000) # group-based split a, b = int(len(ids) * 0.6), int(len(ids) * 0.8) ASSIGN = {n: (0 if i < a else (1 if i < b else 2)) for i, n in enumerate(ids)} GROUP_TRAIN = [y for y in ROWS if ASSIGN[y["id"]] == 0] GROUP_TEST = [y for y in ROWS if ASSIGN[y["id"]] == 2] print(f"rows {len(ROWS)} -- row split: train {len(ROW_TRAIN)} test {len(ROW_TEST)}, subscribers " f"seen in both sets {len({y['id'] for y in ROW_TEST} & {y['id'] for y in ROW_TRAIN})}" f"\ngroup-based split: train {len(GROUP_TRAIN)} test {len(GROUP_TEST)}, shared subscribers " f"{len({y['id'] for y in GROUP_TEST} & {y['id'] for y in GROUP_TRAIN})}\nbaseline: row split " f"{baseline(ROW_TRAIN, ROW_TEST):.4f}, group-based split {baseline(GROUP_TRAIN, GROUP_TEST):.4f}\n\n" f"{'depth':>8} | {'row':>7} {'diff':>8} | {'group':>7} {'diff':>8} " f"| {'inflation':>9}") for d in (2, 4, 6, 8, 10, 12): m_row, m_group = tree(ROW_TRAIN, d, ROW_FIELDS), tree(GROUP_TRAIN, d, ROW_FIELDS) d_row = accuracy(m_row, ROW_TEST) - baseline(ROW_TRAIN, ROW_TEST) d_group = accuracy(m_group, GROUP_TEST) - baseline(GROUP_TRAIN, GROUP_TEST) print(f"{d:>8} | {accuracy(m_row, ROW_TEST):>7.4f} {d_row:>+8.4f} | {accuracy(m_group, GROUP_TEST):>7.4f} " f"{d_group:>+8.4f} | {d_row - d_group:>+9.4f}") ``` ``` rows 3199 -- row split: train 1919 test 640, subscribers seen in both sets 416 group-based split: train 1921 test 649, shared subscribers 0 baseline: row split 0.7594, group-based split 0.7319 depth | row diff | group diff | inflation 2 | 0.7812 +0.0219 | 0.7411 +0.0092 | +0.0126 4 | 0.7781 +0.0187 | 0.7689 +0.0370 | -0.0182 6 | 0.8078 +0.0484 | 0.7535 +0.0216 | +0.0269 8 | 0.8266 +0.0672 | 0.7458 +0.0139 | +0.0533 10 | 0.8375 +0.0781 | 0.7396 +0.0077 | +0.0704 12 | 0.8594 +0.1000 | 0.7427 +0.0108 | +0.0892 ``` In the row split, **416** of the subscribers who own the 640 test rows are also in the training set; in the group-based split this number is zero. The training sets of the two splits are nearly the same size (1,921 rows against 1,919), so the difference does not come from the amount of data. The inflation grows with capacity. At depth 2 the tree has no room to memorize; at depth 4 the inflation is temporarily negative, because in a shallow tree the split noise is larger than the leakage. At depth 12 the table opens up: the row split shows **0.1000** above the baseline, the group-based split **0.0108**. The **0.0892** between them is not something the model learned — the tree recognizes a subscriber from its feature values and recalls the label it saw from the other row. The time-ordered split was measured in the Introduction to Machine Learning course, and there the sets were separated by **when** they were measured. The group-based split works on another axis: the sets are separated by **whose** data it is. If a record produces more than one row, the unit of the split is that record, not the row. ## Inflation Shrinks the Gap That leakage inflates the overall number is a known result. The question of this course is different: what happens to the gap. The block below takes the depth-12 models of the two splits down to the subgroups. ```python row_acc, group_acc, drops = [], [], [] for g in SUBGROUPS: row_sub = [y for y in ROW_TEST if g in subgroups(y)] group_sub = [y for y in GROUP_TEST if g in subgroups(y)] if min(len(row_sub), len(group_sub)) >= 60: # 60 rows ~ 25 subscribers; the row equivalent of ED6 row_acc.append(accuracy(m_row, row_sub)) group_acc.append(accuracy(m_group, group_sub)) drops.append((row_acc[-1] - group_acc[-1], " ".join(g), row_acc[-1], group_acc[-1])) for f, name, a1, a2 in sorted(drops, reverse=True)[:2] + sorted(drops)[:1]: print(f"{name:<14} row {a1:.4f} | group {a2:.4f} | drop {f:+.4f}") print(f"\nsubgroup gap -- row split {max(row_acc) - min(row_acc):.4f} ({min(row_acc):.4f} .. " f"{max(row_acc):.4f}), group-based split {max(group_acc) - min(group_acc):.4f} " f"({min(group_acc):.4f} .. {max(group_acc):.4f}), {len(row_acc)} subgroups") ``` ``` tariff 40 row 0.7623 | group 0.5413 | drop +0.2210 region central row 0.8364 | group 0.6279 | drop +0.2085 region west row 0.9341 | group 0.9259 | drop +0.0081 subgroup gap -- row split 0.1718 (0.7623 .. 0.9341), group-based split 0.3846 (0.5413 .. 0.9259), 12 subgroups ``` Leakage does not inflate the subgroups equally. The best subgroup is in the same place in both splits and has almost the same number: west, 0.9341 and 0.9259. What drops is **the bad end** — the tariff 40 tier from 0.7623 to 0.5413, the central region from 0.8364 to 0.6279. In the row split the subgroup gap reads **0.1718**, in the group-based split **0.3846**: leakage shrinks the difference between groups by more than half. A badly designed split thus makes two errors at once: it raises the overall number and suppresses the difference between subgroups. The second is quieter, because whoever looks at the table reads "the groups are close to each other" and finds the subgroup measurement unnecessary. ## Summary - The triple of the course was set up: the number measured overall, the same number measured on **13 subgroups**, and the **subgroup gap** between them. The axes are region, tariff tier, household size and reading count; the definition is a decision, and how many subgroups were looked at is written in every lesson. - The depth-6 decision tree beats the baseline on the test set by 0.0198 but varies between 0.6111 and 0.9688 across the subgroups: a gap of **0.3576**, eighteen times the difference overall. In **five of the thirteen subgroups the model falls below** the baseline. - A subgroup number cannot be read without writing how many subscribers it was read from; the tariff 10 tier does not enter the gap because it holds 12 subscribers. - When the 3,199 reading-level rows are shuffled, 416 subscribers fall into both sets, and at depth 12 the difference above the baseline reads 0.1000 instead of 0.0108; the group-based split undoes this **0.0892** inflation, and the inflation grows with capacity. - Leakage corrupts the subgroup gap too: the gap that reads 0.1718 in the row split is **0.3846** in the group-based split. A bad split does not stop at raising the number, it also suppresses the difference between groups. ## Next Step Throughout, the metric was taken as accuracy and never questioned. Yet accuracy counts both error types with the same weight: the cost of sending a subscriber who is not suspicious to inspection is not the same as the cost of missing a suspicious one, and when the ratio between the two changes, the **threshold** of the same model changes. Metric definitions, threshold and curves were set up in the Supervised Learning course and will not be repeated. The next lesson measures only this: the same model gives two different operating points under two different cost matrices — and when the cost is not the same across subgroups, what a single threshold does to which group is counted.