--- title: 'Model Evaluation, Interpretability and Ethics' source: 'https://academia.sh/en/courses/model-ethics-and-interpretation' language: en updated: '2026-10-21T05:03:25+00:00' license: 'CC BY-SA 4.0' lessons: 15 outcomes: - 'Protects the evaluation design against data leakage and distribution shift and designs the baseline itself' - 'Examines model decisions with global and local explanation methods and measures the faithfulness of the explanation' - 'Counts bias and privacy risks in subgroups and shows that fairness definitions conflict' --- # Model Evaluation, Interpretability and Ethics The baseline, a tool for seven courses, becoming an object of design here, and asking for whom each number is measured: the inflation that group-based splitting takes back, the operating point that the cost matrix moves, the choice of baseline making the same model read in two different ways, the narrow margin that a paired comparison gives, input drift and label drift requiring different solutions, the number of questions an explanation closes, the importance ranking read from the model diverging from the ranking measured by perturbation, two methods giving two different explanations for the same example, an opposite-direction interaction that the average hides, the trade-off between rule list length and accuracy, data, label and deployment bias hitting the same number by different routes, three fairness metrics not being satisfiable together, a single record moving the output and the cost of noise, the question a disclosure leaves unanswered, and the number moved by a single unpinned item. ## Evaluation Design Designing the measurement itself and writing down which subgroup every number holds for: the inflated number produced when the same subscriber lands in two sets and the group-based split that undoes it, the same model making two different operating points and two different decisions under two cost matrices, how many baselines a problem has and how the choice of baseline makes the same model read two different ways, how much narrower a margin the paired comparison gives for the difference between two models than the independent comparison, the difference between the training distribution and the production distribution reduced to a single number, and covariate shift and label shift calling for different fixes. ### Validation Strategies Source: https://academia.sh/en/courses/model-ethics-and-interpretation/validation-strategies Measuring the same number separately on the whole and on thirteen subgroups, and testing whether the held-out set was truly held out: a depth-6 decision tree beats the baseline on the test set by 0.0198 yet falls below the baseline in five of the subgroups, and the gap among the twelve subgroups with an n above 25 comes out at 0.3576. The same subscriber landing in both sets in the 3,199-row reading table splits 416 subscribers across them and raises the difference above the baseline at depth 12 from 0.0108 to 0.1000; the group-based split undoes this 0.0892 inflation. The inflation corrupts not only the overall number but the gap as well: the subgroup gap that reads 0.1718 in the row split is 0.3846 in the group-based split. Across seven courses the baseline was written next to every number: a model was never read on its own, it was put side by side with the same-metric number of a procedure dumber than itself, and in the last course the fact that the system making the decision changes its own data was counted too. The one thing never asked is **for whom** the number was measured. All of those numbers were averages over subscribers, and an average may belong to none of the groups beneath it. The rule of this course is this: **a number cannot be read until it says for whom it was measured; performance whose subgroup is not written down has not been measured.** The rule is paid for in every lesson with three numbers — the number measured overall, the same number measured on subgroups, the **subgroup gap** between them — and, when a gap is seen, with the sentence that changes in the report. This lesson sets up the triple and the subgroups, but first it answers a question that takes priority: whether the held-out set was truly held out. - **ED1.** The dataset carries over unchanged from the seven courses and it is **fictional**: a 3,199-row reading table, 1,329 subscribers, three periods, five regions, seed 20260218. The label is fictional too — a subscriber carrying a suspicion of illicit use or a fault — and is not a real audit record. - **ED2.** At subscriber level 1,260 rows remain and are split **756/252/252**. The split, the metric (accuracy) and the baseline (the most frequent class of the training set) are inherited from the Introduction to Machine Learning course; the baseline on the test set is **0.7579**. The model is a **depth-6 decision tree**, written with the standard library. - **ED3.** The separation of the three sets and the time-ordered split were measured in the generalization topic of the Introduction to Machine Learning course and are not repeated here; the only thing added is the **group-based split**. - **ED4.** The subgroups are defined on **four axes** and stay fixed throughout the course: **region** (five values), **tariff tier** (the tier the mean consumption falls in: 10, 25, 40 m³), **household size** (1–2, 3–4, 5 and above), **reading count** (few: one or two periods; full: three periods). **13 subgroups** in total. - **ED5.** The subgroup definition is a decision, not a finding. **How many subgroups were looked at is written in every lesson**, because looking at many produces findings out of noise — this was measured in the Statistical Analysis and Exploratory Data Analysis course and is not repeated. - **ED6.** Next to every subgroup number is written **how many subscribers it was read from**; **a group whose n is below 25 does not enter the gap calculation**, because in 25 subscribers a single wrong count moves the number by 0.04. - **ED7.** In the row-level table every row is one **(subscriber, period)** reading, the label belongs to the subscriber, and the subscriber-level summaries are written onto the rows too. Depth is swept over **six candidates**. #### Thirteen Readings of the Same Number The first block generates the setup, defines the subgroups and prints the course's triple in a single table. The tree is a model: at each node it picks a threshold, and at the leaf it states the majority class. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def subgroups(x): # four axes, thirteen subgroups; fixed throughout the course return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M = tree(TRAIN, 6, FIELDS) def report_row(name, rows): base, acc = baseline(TRAIN, rows), accuracy(M, rows) print(f"{name:<14} {len(rows):>4} {base:>8.4f} {acc:>7.4f} {acc - base:>+8.4f}" f"{' n<25, not in gap' if len(rows) < 25 else ''}") return len(rows), base, acc print(f"subscribers {len(DATA)} -- 756/252/252 -- subgroups {len(SUBGROUPS)} (four axes)\n" f"{'subgroup':<14} {'n':>4} {'baseline':>8} {'tree':>7} {'diff':>8}") report_row("TOTAL", TEST) measured = {g: report_row(" ".join(g), [x for x in TEST if g in subgroups(x)]) for g in SUBGROUPS} scored = [v[2] for v in measured.values() if v[0] >= 25] below = [" ".join(g) for g, v in measured.items() if v[2] < v[1]] print(f"\nsubgroup gap ({len(scored)} subgroups): best {max(scored):.4f}, worst {min(scored):.4f}, " f"gap {max(scored) - min(scored):.4f}\nbelow baseline: {len(below)}/" f"{len(SUBGROUPS)} -- " + ", ".join(below)) ``` ``` subscribers 1260 -- 756/252/252 -- subgroups 13 (four axes) subgroup n baseline tree diff TOTAL 252 0.7579 0.7778 +0.0198 household 1-2 73 0.7534 0.7945 +0.0411 household 3-4 143 0.7692 0.7762 +0.0070 household 5+ 36 0.7222 0.7500 +0.0278 reading few 85 0.7882 0.7412 -0.0471 reading full 167 0.7425 0.7964 +0.0539 region central 58 0.6552 0.6724 +0.0172 region east 36 0.4722 0.6111 +0.1389 region north 68 0.8235 0.7794 -0.0441 region south 58 0.8276 0.8793 +0.0517 region west 32 1.0000 0.9688 -0.0312 tariff 10 12 0.6667 0.5833 -0.0833 n<25, not in gap tariff 25 193 0.8394 0.8290 -0.0104 tariff 40 47 0.4468 0.6170 +0.1702 subgroup gap (12 subgroups): best 0.9688, worst 0.6111, gap 0.3576 below baseline: 5/13 -- reading few, region north, region west, tariff 10, tariff 25 ``` The first row is the row that seven courses read: the model beats the baseline by **0.0198**. The thirteen rows beneath it are other numbers of the same metric for the same model, and the **gap** between the best and the worst is **0.3576** — eighteen times the difference read overall. The 0.7778 reported as performance means 0.6111 for a subscriber in the east region and 0.9688 for one in the west. The harsher number is on the last line: the model beats the baseline overall but falls **below the baseline in five of the thirteen subgroups**. In the 85 subscribers with few readings the baseline is 0.7882 and the model 0.7412 — the model is 4.7 points worse than the procedure that learns nothing. The difference measured overall contains all of these groups and shows none of them. The baseline column says why it is this way. None of the west's 32 test subscribers is suspicious; there, saying "none is suspicious" gives 1.0000, and no model can beat that. In the east and in the tariff 40 tier the suspicious share exceeds half, so the baseline drops below 0.50 and the model beats it most there. The subgroup gap also carries the group's own difficulty, which is why the baseline stands next to every number. The tariff 10 tier does not enter the gap because it holds 12 subscribers. #### The Same Subscriber in Two Sets This table assumes the held-out sets were truly separated. The raw form of the setup is not at **subscriber level but at reading level**: 3,199 rows for 1,260 subscribers, and a subscriber who produced three periods of readings has three rows, all with the same label. The moment the rows are shuffled and split, one row of the same subscriber falls into the training set and another into the test set. ```python ROWS = [] for x in DATA: for i, o in enumerate(READINGS[x["id"]]): ROWS.append(dict(x, reading=o, position=i)) ROW_FIELDS = ["reading", "position"] + FIELDS ROW_TRAIN, ROW_VAL, ROW_TEST = three_way_split(shuffle(ROWS, SEED + 93000)) # row split ids = shuffle([x["id"] for x in DATA], SEED + 93000) # group-based split a, b = int(len(ids) * 0.6), int(len(ids) * 0.8) ASSIGN = {n: (0 if i < a else (1 if i < b else 2)) for i, n in enumerate(ids)} GROUP_TRAIN = [y for y in ROWS if ASSIGN[y["id"]] == 0] GROUP_TEST = [y for y in ROWS if ASSIGN[y["id"]] == 2] print(f"rows {len(ROWS)} -- row split: train {len(ROW_TRAIN)} test {len(ROW_TEST)}, subscribers " f"seen in both sets {len({y['id'] for y in ROW_TEST} & {y['id'] for y in ROW_TRAIN})}" f"\ngroup-based split: train {len(GROUP_TRAIN)} test {len(GROUP_TEST)}, shared subscribers " f"{len({y['id'] for y in GROUP_TEST} & {y['id'] for y in GROUP_TRAIN})}\nbaseline: row split " f"{baseline(ROW_TRAIN, ROW_TEST):.4f}, group-based split {baseline(GROUP_TRAIN, GROUP_TEST):.4f}\n\n" f"{'depth':>8} | {'row':>7} {'diff':>8} | {'group':>7} {'diff':>8} " f"| {'inflation':>9}") for d in (2, 4, 6, 8, 10, 12): m_row, m_group = tree(ROW_TRAIN, d, ROW_FIELDS), tree(GROUP_TRAIN, d, ROW_FIELDS) d_row = accuracy(m_row, ROW_TEST) - baseline(ROW_TRAIN, ROW_TEST) d_group = accuracy(m_group, GROUP_TEST) - baseline(GROUP_TRAIN, GROUP_TEST) print(f"{d:>8} | {accuracy(m_row, ROW_TEST):>7.4f} {d_row:>+8.4f} | {accuracy(m_group, GROUP_TEST):>7.4f} " f"{d_group:>+8.4f} | {d_row - d_group:>+9.4f}") ``` ``` rows 3199 -- row split: train 1919 test 640, subscribers seen in both sets 416 group-based split: train 1921 test 649, shared subscribers 0 baseline: row split 0.7594, group-based split 0.7319 depth | row diff | group diff | inflation 2 | 0.7812 +0.0219 | 0.7411 +0.0092 | +0.0126 4 | 0.7781 +0.0187 | 0.7689 +0.0370 | -0.0182 6 | 0.8078 +0.0484 | 0.7535 +0.0216 | +0.0269 8 | 0.8266 +0.0672 | 0.7458 +0.0139 | +0.0533 10 | 0.8375 +0.0781 | 0.7396 +0.0077 | +0.0704 12 | 0.8594 +0.1000 | 0.7427 +0.0108 | +0.0892 ``` In the row split, **416** of the subscribers who own the 640 test rows are also in the training set; in the group-based split this number is zero. The training sets of the two splits are nearly the same size (1,921 rows against 1,919), so the difference does not come from the amount of data. The inflation grows with capacity. At depth 2 the tree has no room to memorize; at depth 4 the inflation is temporarily negative, because in a shallow tree the split noise is larger than the leakage. At depth 12 the table opens up: the row split shows **0.1000** above the baseline, the group-based split **0.0108**. The **0.0892** between them is not something the model learned — the tree recognizes a subscriber from its feature values and recalls the label it saw from the other row. The time-ordered split was measured in the Introduction to Machine Learning course, and there the sets were separated by **when** they were measured. The group-based split works on another axis: the sets are separated by **whose** data it is. If a record produces more than one row, the unit of the split is that record, not the row. #### Inflation Shrinks the Gap That leakage inflates the overall number is a known result. The question of this course is different: what happens to the gap. The block below takes the depth-12 models of the two splits down to the subgroups. ```python row_acc, group_acc, drops = [], [], [] for g in SUBGROUPS: row_sub = [y for y in ROW_TEST if g in subgroups(y)] group_sub = [y for y in GROUP_TEST if g in subgroups(y)] if min(len(row_sub), len(group_sub)) >= 60: # 60 rows ~ 25 subscribers; the row equivalent of ED6 row_acc.append(accuracy(m_row, row_sub)) group_acc.append(accuracy(m_group, group_sub)) drops.append((row_acc[-1] - group_acc[-1], " ".join(g), row_acc[-1], group_acc[-1])) for f, name, a1, a2 in sorted(drops, reverse=True)[:2] + sorted(drops)[:1]: print(f"{name:<14} row {a1:.4f} | group {a2:.4f} | drop {f:+.4f}") print(f"\nsubgroup gap -- row split {max(row_acc) - min(row_acc):.4f} ({min(row_acc):.4f} .. " f"{max(row_acc):.4f}), group-based split {max(group_acc) - min(group_acc):.4f} " f"({min(group_acc):.4f} .. {max(group_acc):.4f}), {len(row_acc)} subgroups") ``` ``` tariff 40 row 0.7623 | group 0.5413 | drop +0.2210 region central row 0.8364 | group 0.6279 | drop +0.2085 region west row 0.9341 | group 0.9259 | drop +0.0081 subgroup gap -- row split 0.1718 (0.7623 .. 0.9341), group-based split 0.3846 (0.5413 .. 0.9259), 12 subgroups ``` Leakage does not inflate the subgroups equally. The best subgroup is in the same place in both splits and has almost the same number: west, 0.9341 and 0.9259. What drops is **the bad end** — the tariff 40 tier from 0.7623 to 0.5413, the central region from 0.8364 to 0.6279. In the row split the subgroup gap reads **0.1718**, in the group-based split **0.3846**: leakage shrinks the difference between groups by more than half. A badly designed split thus makes two errors at once: it raises the overall number and suppresses the difference between subgroups. The second is quieter, because whoever looks at the table reads "the groups are close to each other" and finds the subgroup measurement unnecessary. #### Summary - The triple of the course was set up: the number measured overall, the same number measured on **13 subgroups**, and the **subgroup gap** between them. The axes are region, tariff tier, household size and reading count; the definition is a decision, and how many subgroups were looked at is written in every lesson. - The depth-6 decision tree beats the baseline on the test set by 0.0198 but varies between 0.6111 and 0.9688 across the subgroups: a gap of **0.3576**, eighteen times the difference overall. In **five of the thirteen subgroups the model falls below** the baseline. - A subgroup number cannot be read without writing how many subscribers it was read from; the tariff 10 tier does not enter the gap because it holds 12 subscribers. - When the 3,199 reading-level rows are shuffled, 416 subscribers fall into both sets, and at depth 12 the difference above the baseline reads 0.1000 instead of 0.0108; the group-based split undoes this **0.0892** inflation, and the inflation grows with capacity. - Leakage corrupts the subgroup gap too: the gap that reads 0.1718 in the row split is **0.3846** in the group-based split. A bad split does not stop at raising the number, it also suppresses the difference between groups. #### Next Step Throughout, the metric was taken as accuracy and never questioned. Yet accuracy counts both error types with the same weight: the cost of sending a subscriber who is not suspicious to inspection is not the same as the cost of missing a suspicious one, and when the ratio between the two changes, the **threshold** of the same model changes. Metric definitions, threshold and curves were set up in the Supervised Learning course and will not be repeated. The next lesson measures only this: the same model gives two different operating points under two different cost matrices — and when the cost is not the same across subgroups, what a single threshold does to which group is counted. ### Metric Selection Source: https://academia.sh/en/courses/model-ethics-and-interpretation/metric-selection The same model taking two separate operating points and making two separate decisions under two cost matrices: in the structure where inspection is costly the selected threshold is 0.95 and 17 of 252 subscribers are flagged, in the structure where illicit use is costly the threshold is 0.25 and 77 are flagged; the decision of 60 subscribers flips between the two structures. The baseline moves with the cost as well — flagging nobody is 0.2421 in one, flagging everybody is 0.7579 in the other. A single threshold spreads the gain across subgroups from +0.4138 to -0.4468, the gap is 0.8606 and in four of twelve subgroups the model stays below its own baseline; choosing a subgroup-specific threshold lowers the test cost in only four of them. The previous lesson took the metric to be accuracy and never questioned it. Accuracy counts the two error types with the same weight: sending a subscriber who is not suspicious to inspection and missing a suspicious subscriber are equally wrong. In operations they are not equal. An inspection crew's travel, its time and the mark a wasted visit leaves on the subscriber are one cost; an illicit use that slipped through or a fault left unrepaired is another cost, and the ratio of the two is a business decision. This lesson writes that ratio as a **cost matrix** and measures one thing: the same model, the same scores, two separate cost matrices — how many separate operating points, how many separate decisions. Metric definitions, the threshold and the curves were set up in the Evaluation and Tuning topic of the Supervised Learning course and are not repeated here. The only thing added is reading the threshold from the **business objective**. The subgroups and the triple carry over unchanged from the previous lesson. - **ED8.** The setup, the split (**756/252/252**), the seed and the subgroup definitions carry over unchanged from the previous lesson: four axes, **13 subgroups**, an n written next to every number, and **a group whose n is below 25 does not enter the gap**. - **ED9.** The model is the same depth-6 decision tree, with one difference: the leaf holds not the class but the **suspicious share**. This share is the model's score; a subscriber above the threshold is sent to inspection. - **ED10.** The metric is the **cost per subscriber** and **small is good**. A false alarm (a flagged subscriber who is not suspicious) costs $a$ units, a miss (an unflagged subscriber who is suspicious) costs $m$ units; correct decisions cost zero. The unit is abstract and is not money. - **ED11.** Two cost matrices are tested. **Inspection costly**: $a = 2$, $m = 1$. **Illicit use costly**: $a = 1$, $m = 4$. Both are fictional and stand for a business decision. - **ED12.** The threshold is swept over **10 candidates** (from 0.05 to 0.95 in steps of one tenth) and is selected on the **validation set**; the test set is read only for the report. - **ED13.** The baseline is computed separately for each cost matrix and is **the cheaper one** of two procedures that learn nothing: flagging nobody, or flagging everybody. - **ED14.** A subgroup-specific threshold is selected on that subgroup's **validation** rows and reported on that subgroup's test rows. Subgroup validation sets are small, and the selection share comes from there. - **ED15.** Accuracy is never printed in this lesson. The 0.7778 of the previous lesson and the costs of this lesson are not on the same axis and are not compared. #### Two Cost Structures, Two Thresholds The first block sets up the scenario, trains the tree so that it leaves a share at the leaf, and sweeps ten thresholds under the two cost matrices. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): # the leaf holds the SHARE, not the class p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def score(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from lesson 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] def cost(rows, t, fa, miss): # cost per subscriber; small is good total = 0 for x in rows: total += fa * (1 - x["suspicious"]) if score(M, x) >= t else miss * x["suspicious"] return total / len(rows) def flagged(rows, t): return sum(score(M, x) >= t for x in rows) / len(rows) SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M = tree(TRAIN, 6, FIELDS) THRESHOLDS = [0.05 + 0.10 * i for i in range(10)] STRUCTURES = (("inspection costly", 2, 1), ("illicit use costly", 1, 4)) print(f"{'threshold':>9} | {'A val':>7} {'A test':>7} | {'B val':>7} {'B test':>7} " f"| {'flagged':>7}") for t in THRESHOLDS: print(f"{t:>9.2f} | {cost(VAL, t, 2, 1):>7.4f} {cost(TEST, t, 2, 1):>7.4f} " f"| {cost(VAL, t, 1, 4):>7.4f} {cost(TEST, t, 1, 4):>7.4f} " f"| {flagged(TEST, t):>7.4f}") CHOSEN = {} for name, fa, miss in STRUCTURES: t = CHOSEN[name] = min(THRESHOLDS, key=lambda c: (cost(VAL, c, fa, miss), c)) flag_none, flag_all = cost(TEST, 1.1, fa, miss), cost(TEST, -0.1, fa, miss) print(f"\n{name:<18} threshold {t:.2f} test cost {cost(TEST, t, fa, miss):.4f} baseline " f"{min(flag_none, flag_all):.4f} ({'none' if flag_none < flag_all else 'all'}) gain " f"{min(flag_none, flag_all) - cost(TEST, t, fa, miss):+.4f} flagged " f"{sum(score(M, x) >= t for x in TEST)}/{len(TEST)}") ``` ``` threshold | A val A test | B val B test | flagged 0.05 | 1.0952 0.9762 | 0.6032 0.5437 | 0.7063 0.15 | 0.7659 0.7659 | 0.4802 0.5913 | 0.5357 0.25 | 0.4802 0.4246 | 0.4762 0.5595 | 0.3056 0.35 | 0.3889 0.3214 | 0.5278 0.5913 | 0.2183 0.45 | 0.3492 0.2976 | 0.5079 0.6349 | 0.1825 0.55 | 0.2778 0.2579 | 0.5833 0.7262 | 0.1151 0.65 | 0.2540 0.2460 | 0.6548 0.7341 | 0.1032 0.75 | 0.2222 0.2302 | 0.6667 0.7540 | 0.0833 0.85 | 0.2222 0.2302 | 0.6667 0.7540 | 0.0833 0.95 | 0.2183 0.2222 | 0.6786 0.7778 | 0.0675 inspection costly threshold 0.95 test cost 0.2222 baseline 0.2421 (none) gain +0.0198 flagged 17/252 illicit use costly threshold 0.25 test cost 0.5595 baseline 0.7579 (all) gain +0.1984 flagged 77/252 ``` The two cost columns fall in opposite directions. In the structure where inspection is costly, the cost drops as the threshold rises and the cheapest point is **0.95**; in the structure where illicit use is costly the cheapest point is **0.25**, after which the cost starts to climb. The model is a single model, the scores are a single set of scores, and the two structures pick two separate rows from the same table. In one case **17** of 252 subscribers go to inspection, in the other **77**. The baseline moves with the cost as well. When inspection is costly, the cheapest procedure that learns nothing is **flagging nobody**, and it costs 0.2421; when illicit use is costly it is **flagging everybody**, and it costs 0.7579. Both are computed on the same set with the same labels. The model's gain changes accordingly: in the first structure it can drop only **0.0198** below the baseline, in the second **0.1984**. The same model, the same test set — and one is reported as "helps little", the other as "helps clearly". #### Who the Decision Changes For The difference between the two operating points is two rows in a table, but on the subscriber side it is something concrete: some subscribers go to inspection under one structure and not under the other. ```python tA, tB = CHOSEN["inspection costly"], CHOSEN["illicit use costly"] changed = [x for x in TEST if (score(M, x) >= tA) != (score(M, x) >= tB)] shares = sorted(((sum(g in subgroups(x) for x in changed) / sum(g in subgroups(x) for x in TEST), g) for g in SUBGROUPS), reverse=True) print(f"decision changed for {len(changed)}/{len(TEST)} subscribers, {sum(x['suspicious'] for x in changed)}" f" of them suspicious; share by subgroup:") for o, g in shares[:2] + shares[-1:]: print(f" {' '.join(g):<14} {o:.4f}") ``` ``` decision changed for 60/252 subscribers, 23 of them suspicious; share by subgroup: tariff 10 0.7500 region east 0.5556 region west 0.0938 ``` The decision of sixty subscribers flips, and 23 of them are truly suspicious. Choosing the cost structure is not a metric preference; it is whether sixty households get a knock on the door or not. The distribution is not even either: **0.5556** of the subscribers in the east region move between the two structures, while in the west the share is **0.0938**. Once a cost matrix is chosen, the subgroups do not carry its load equally. #### The Cost Is Not the Same Across Subgroups The single threshold selected in the structure where illicit use is costly is 0.25, and it is applied to all thirteen subgroups in the same way. The block below prints, side by side, the cost of that single threshold in each subgroup, that subgroup's own baseline, and what would happen if the subgroup selected its own threshold. ```python fa, miss = 1, 4 print(f"{'subgroup':<14} {'n':>4} {'flagged':>7} {'cost':>7} {'baseline':>8} {'gain':>8}" f" | {'own':>5} {'cost':>7} {'extra':>8}") gains, extras = [], [] for g in SUBGROUPS: val_g = [x for x in VAL if g in subgroups(x)] test_g = [x for x in TEST if g in subgroups(x)] c, base = cost(test_g, tB, fa, miss), min(cost(test_g, 1.1, fa, miss), cost(test_g, -0.1, fa, miss)) own_t = min(THRESHOLDS, key=lambda t: (cost(val_g, t, fa, miss), t)) own_cost = cost(test_g, own_t, fa, miss) if len(test_g) >= 25: gains.append(base - c) extras.append(c - own_cost) print(f"{' '.join(g):<14} {len(test_g):>4} {flagged(test_g, tB):>7.4f} {c:>7.4f} {base:>8.4f} " f"{base - c:>+8.4f} | {own_t:>5.2f} {own_cost:>7.4f} {c - own_cost:>+8.4f}" f"{' n<25' if len(test_g) < 25 else ''}") print(f"\ngain gap ({len(gains)} subgroups): best {max(gains):+.4f}, worst " f"{min(gains):+.4f}, gap {max(gains) - min(gains):.4f}; below baseline " f"{sum(1 for v in gains if v < 0)}/{len(gains)}") print(f"subgroups whose own threshold lowers the test cost: {sum(1 for v in extras if v > 0)}" f"/{len(extras)}, unchanged {sum(1 for v in extras if v == 0)}, raised " f"{sum(1 for v in extras if v < 0)}") ``` ``` subgroup n flagged cost baseline gain | own cost extra household 1-2 73 0.3562 0.5205 0.7534 +0.2329 | 0.15 0.6301 -0.1096 household 3-4 143 0.2657 0.5944 0.7692 +0.1748 | 0.25 0.5944 +0.0000 household 5+ 36 0.3611 0.5000 0.7222 +0.2222 | 0.55 0.7500 -0.2500 reading few 85 0.2706 0.7059 0.7882 +0.0824 | 0.45 0.8471 -0.1412 reading full 167 0.3234 0.4850 0.7425 +0.2575 | 0.15 0.5509 -0.0659 region central 58 0.3276 0.8448 0.6552 -0.1897 | 0.25 0.8448 +0.0000 region east 36 0.7500 0.7778 0.4722 -0.3056 | 0.45 0.7222 +0.0556 region north 68 0.2500 0.6618 0.7059 +0.0441 | 0.15 0.6765 -0.0147 region south 58 0.1897 0.2759 0.6897 +0.4138 | 0.15 0.5517 -0.2759 region west 32 0.0938 0.0938 0.0000 -0.0938 | 0.35 0.0625 +0.0312 tariff 10 12 0.7500 0.4167 0.6667 +0.2500 | 0.05 0.4167 +0.0000 n<25 tariff 25 193 0.1813 0.4870 0.6425 +0.1554 | 0.45 0.4560 +0.0311 tariff 40 47 0.7021 0.8936 0.4468 -0.4468 | 0.05 0.5957 +0.2979 gain gap (12 subgroups): best +0.4138, worst -0.4468, gap 0.8606; below baseline 4/12 subgroups whose own threshold lowers the test cost: 4/12, unchanged 2, raised 6 ``` The gain, 0.1984 overall, spreads across the subgroups between **+0.4138 and −0.4468**; the **gap is 0.8606**. In four subgroups the model is **below** its own baseline: west, east, central and the tariff 40 tier. The baseline of the tariff 40 tier is 0.4468 and the model's cost is 0.8936 — in that tier the model is twice as expensive as saying "flag everybody". The reason is in the flagged column: the 0.25 threshold flags 0.7021 of the subscribers in that tier, but because the suspicious share of the tier is 0.553, the suspicious subscribers left among the unflagged pay the four-unit cost of a miss. The last three columns suggest a correction and show the cost of the correction too. When each subgroup selects its own threshold from its own validation rows, the tariff 40 tier drops to 0.05 and its cost falls from 0.8936 to 0.5957. But the same procedure lowers the test cost in only **four** of the twelve subgroups; in two it changes nothing, and in **six it raises it** — by 0.2759 units in the south and 0.2500 in household size 5 and above. The reason is the smallness of the subgroup validation sets: a threshold selected from forty to seventy rows selects the noise of those rows. Tuning a threshold per subgroup requires a validation set per subgroup, and those sets come from the design, not from the split. #### Summary - A cost matrix turns the business objective into a number: the ratio between a false alarm and a miss is a business decision, and the threshold is read from that ratio. The metric is the **cost per subscriber** and small is good. - The same model, the same scores, two cost matrices: when inspection is costly the selected threshold is 0.95 and 17 of 252 subscribers are flagged; when illicit use is costly the threshold is 0.25 and 77 are flagged. **The decision of 60 subscribers flips**, and 23 of them are truly suspicious. - The baseline moves with the cost too: in the first structure flagging nobody (0.2421), in the second flagging everybody (0.7579) is the baseline. The model's gain is accordingly +0.0198 or +0.1984. - A single threshold does not give a single result across subgroups: the gain spreads between +0.4138 and −0.4468, the **gap is 0.8606**, and **in 4 of 12 subgroups the model stays below its own baseline**. - Selecting a subgroup-specific threshold lowers the test cost in only 4 of 12 subgroups and raises it in six; the smaller the subgroup validation set, the larger the selection share. #### Next Step In this lesson the baseline turned into a question of its own for the second time. When the cost structure changed, the baseline changed too: in one, flagging nobody, in the other, flagging everybody. Across the subgroups every group turned out to have its own baseline, which was 0.0000 in one group and 0.7882 in another. Over seven courses the baseline was a **tool** — a fixed mark that makes the model's number readable. Here it was seen not to be fixed. The next lesson takes the baseline up directly: how many baselines a problem has, which one is chosen, which two separate ways the choice lets the same model be read, and what shows when the baselines chosen in the seven courses of this curriculum are collected in a single table. ### Baseline Model Source: https://academia.sh/en/courses/model-ethics-and-interpretation/baseline-model Measuring that the baseline is not a tool but a design object: the same depth-6 decision tree reads between -0.0119 and +0.5357 against eight baseline candidates, and the choice alone moves the number by 0.5476. A lookup-table baseline has a capacity of its own — with five cells it ties the model at 0.7579, with fifteen cells it beats the model at 0.7897, with thirty-nine cells it drops to 0.7500. The choice of baseline changes direction in the subgroups too: the model that falls below the most-frequent-class baseline in four of twelve subgroups falls below the region-and-tier lookup table in eight, and the difference gap shrinks from 0.2173 to 0.1351. The baselines chosen in the seven courses of the Machine Learning and Data Science curriculum are collected in a single table. The previous lesson turned the baseline into a question of its own for the second time. When the cost matrix changed, the baseline changed too: in one, flagging nobody, in the other, flagging everybody. Across the subgroups every group turned out to have its own baseline, which was 0.0000 in one group and 0.7882 in another. Over seven courses the baseline was a **tool** — a fixed mark that makes the model's number readable and is not argued over again once it is chosen. This lesson measures that it is not fixed. How many baselines a problem has, how one is chosen among them, in which two separate ways the choice lets the same model be read. Here the baseline is not a tool but a **design object**: a procedure with a capacity of its own, an overfitting of its own and a subgroup behavior of its own. At the end of the lesson the baselines actually chosen in the seven courses of the Machine Learning and Data Science curriculum are collected in a single table. - **ED16.** The setup, the split (**756/252/252**), the seed, the metric (**accuracy**) and the subgroup definitions carry over unchanged from the previous two lessons: four axes, **13 subgroups**, **a group whose n is below 25 does not enter the gap**. - **ED17.** The model being compared is the **depth-6 decision tree**, and it gives **0.7778** on the test set. This number never changes during the lesson; the only thing that changes is the baseline placed opposite it. - **ED18.** A baseline candidate is a procedure that, even if learned from the training set, does not look at all of the features the model uses and does not itself run a tuning search. The boundary is a decision. - **ED19.** Eight candidates are tested: calling everyone suspicious, random assignment by class proportion, stating the most frequent class, a single-split rule, and four **lookup tables** — region; region and tier; those plus household; all four axes. A lookup table states the training majority in a cell, and the most frequent class in a cell it has not seen; **the number of cells is written**, because that is what is being measured. - **ED20.** Random assignment is **the mean of twenty runs**; if a single run were printed the number would swing from run to run. - **ED21.** No choice among the candidates is **made** in this lesson. What is measured is not the result of the choice but how far the choice can move the same model. - **ED22.** Every number in the course table of the last section was read from its source lesson; the units are not shared across courses and the rows are not added to one another. #### How Many Baselines a Problem Has The first block sets up the scenario, trains the model and measures the eight baseline candidates on the same test set. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from lesson 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M, STUMP = tree(TRAIN, 6, FIELDS), tree(TRAIN, 1, FIELDS) RATE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN) MAJORITY = int(RATE > 0.5) def groups_of(x): return dict(subgroups(x)) def lookup(key_fn): # lookup-table baseline: the training majority in the cell counts = {} for x in TRAIN: counts.setdefault(key_fn(x), [0, 0])[x["suspicious"]] += 1 votes = {c: int(v[1] > v[0]) for c, v in counts.items()} return lambda x: votes.get(key_fn(x), MAJORITY) def random_assign(rows, n=20): # assignment by class proportion, the mean of twenty runs t = 0.0 for j in range(n): r = rng(SEED + 61000 + j) t += sum((1 if r() < RATE else 0) == x["suspicious"] for x in rows) / len(rows) return t / n def hit_rate(f, rows): return sum(f(x) == x["suspicious"] for x in rows) / len(rows) KEYS = (("region", lambda x: x["region_name"]), ("region+tier", lambda x: (x["region_name"], groups_of(x)["tariff"])), ("+household", lambda x: (x["region_name"], groups_of(x)["tariff"], groups_of(x)["household"])), ("four axes", lambda x: (x["region_name"], groups_of(x)["tariff"], groups_of(x)["household"], groups_of(x)["reading"]))) CANDIDATES = [("all suspicious", 1, lambda x: 1), ("most frequent class", 1, lambda x: MAJORITY), ("single rule", 2, lambda x: predict(STUMP, x))] CANDIDATES += [(name, len({k(x) for x in TRAIN}), lookup(k)) for name, k in KEYS] model_acc = accuracy(M, TEST) print(f"model, depth-6 decision tree: train {accuracy(M, TRAIN):.4f} test {model_acc:.4f}\n\n" f"{'baseline candidate':<19} {'cells':>5} {'train':>7} {'test':>7} {'model diff':>12}") print(f"{'random assignment':<19} {'--':>5} {random_assign(TRAIN):>7.4f} {random_assign(TEST):>7.4f} " f"{model_acc - random_assign(TEST):>+12.4f}") diffs = [model_acc - random_assign(TEST)] for name, n, f in CANDIDATES: diffs.append(model_acc - hit_rate(f, TEST)) print(f"{name:<19} {n:>5} {hit_rate(f, TRAIN):>7.4f} {hit_rate(f, TEST):>7.4f} {diffs[-1]:>+12.4f}") print(f"\nthe baseline choice makes the same model read between {min(diffs):+.4f} and {max(diffs):+.4f}:" f" range {max(diffs) - min(diffs):.4f}") ``` ``` model, depth-6 decision tree: train 0.8466 test 0.7778 baseline candidate cells train test model diff random assignment -- 0.6522 0.6450 +0.1327 all suspicious 1 0.2222 0.2421 +0.5357 most frequent class 1 0.7778 0.7579 +0.0198 single rule 2 0.7778 0.7579 +0.0198 region 5 0.7778 0.7579 +0.0198 region+tier 15 0.7897 0.7897 -0.0119 +household 39 0.7976 0.7500 +0.0278 four axes 66 0.8095 0.7738 +0.0040 the baseline choice makes the same model read between -0.0119 and +0.5357: range 0.5476 ``` The last column is eight separate readings of a single model. The model did not change at all: 0.7778 on the test set. But against the "call everyone suspicious" baseline the model's contribution is **+0.5357**, against the "region and tier lookup table" baseline it is **−0.0119**. The **0.5476** between them is a number produced only by **the choice of baseline**, not by the data or the model. The same piece of work can be told in two separate reports, and both are computed correctly. The three middle rows give the second result. The most frequent class, a single-split rule and a lookup table that looks at the region produce **the same number**: 0.7579. Because the suspicious share does not exceed half in any region, all five cells of the region table say the same answer. Making a baseline more complicated does not make it stronger; it gets stronger only if the majority **changes** in the cells. The main finding is two rows further down. When region and tariff tier are taken together, the majority flips in some of the fifteen cells and the table reads 0.7897 — **above the depth-6 decision tree**. A tree trained on eleven features, which tried hundreds of candidate splits, loses to a lookup table with two columns. When household is added the cell count rises to 39: the train number climbs to 0.7976 while the test number falls to 0.7500. With all four axes there are 66 cells, train 0.8095, test 0.7738. **A baseline has a capacity too, and when that capacity is exceeded the baseline overfits as well.** This is what lifts the baseline out of being a tool and turns it into a design object: when it is chosen, the questions asked when choosing a model are asked. #### The Choice of Baseline Changes Direction in the Subgroups The overall difference between the two baselines is only 0.0318. Once we go down to the subgroups, the two do not say the same thing. ```python REGION_TIER = lookup(lambda x: (x["region_name"], groups_of(x)["tariff"])) print(f"{'subgroup':<14} {'n':>4} {'model':>7} {'A baseline':>10} {'A diff':>8} " f"{'B baseline':>10} {'B diff':>8}") diffs_a, diffs_b = [], [] for g in SUBGROUPS: rows = [x for x in TEST if g in subgroups(x)] base_a, base_b, acc_g = hit_rate(lambda x: MAJORITY, rows), hit_rate(REGION_TIER, rows), accuracy(M, rows) if len(rows) >= 25: diffs_a.append(acc_g - base_a) diffs_b.append(acc_g - base_b) print(f"{' '.join(g):<14} {len(rows):>4} {acc_g:>7.4f} {base_a:>10.4f} {acc_g - base_a:>+8.4f} " f"{base_b:>10.4f} {acc_g - base_b:>+8.4f}{' n<25' if len(rows) < 25 else ''}") print(f"\nA most frequent class: diff {min(diffs_a):+.4f} .. {max(diffs_a):+.4f}, gap " f"{max(diffs_a) - min(diffs_a):.4f}, below baseline {sum(1 for v in diffs_a if v < 0)}/{len(diffs_a)}") print(f"B region+tier : diff {min(diffs_b):+.4f} .. {max(diffs_b):+.4f}, gap " f"{max(diffs_b) - min(diffs_b):.4f}, below baseline {sum(1 for v in diffs_b if v < 0)}/{len(diffs_b)}") ``` ``` subgroup n model A baseline A diff B baseline B diff household 1-2 73 0.7945 0.7534 +0.0411 0.8219 -0.0274 household 3-4 143 0.7762 0.7692 +0.0070 0.7832 -0.0070 household 5+ 36 0.7500 0.7222 +0.0278 0.7500 +0.0000 reading few 85 0.7412 0.7882 -0.0471 0.7647 -0.0235 reading full 167 0.7964 0.7425 +0.0539 0.8024 -0.0060 region central 58 0.6724 0.6552 +0.0172 0.6552 +0.0172 region east 36 0.6111 0.4722 +0.1389 0.6944 -0.0833 region north 68 0.7794 0.8235 -0.0441 0.8235 -0.0441 region south 58 0.8793 0.8276 +0.0517 0.8276 +0.0517 region west 32 0.9688 1.0000 -0.0312 1.0000 -0.0312 tariff 10 12 0.5833 0.6667 -0.0833 0.6667 -0.0833 n<25 tariff 25 193 0.8290 0.8394 -0.0104 0.8394 -0.0104 tariff 40 47 0.6170 0.4468 +0.1702 0.6170 +0.0000 A most frequent class: diff -0.0471 .. +0.1702, gap 0.2173, below baseline 4/12 B region+tier : diff -0.0833 .. +0.0517, gap 0.1351, below baseline 8/12 ``` The east region is where the two baselines part. Against the most frequent class the model beats the baseline there by **0.1389** and gives one of the best contributions of the course; against the region and tier table it stays **0.0833 below** the baseline. The same happens in the tariff 40 tier: 0.0000 instead of +0.1702. Both numbers are correct. The difference is that the first baseline is unaware of the east region's high suspicion rate, while the second already knows that rate. A model can say what it adds in a subgroup only when it is compared with a baseline that **already knows** that subgroup. The two summary lines give the result. Against the most-frequent-class baseline the model is below the baseline in **four** of the twelve subgroups; against the lookup-table baseline, in **eight**. The difference gap also falls from 0.2173 to 0.1351. A strong baseline both shrinks the model's contribution and flattens it across subgroups; this is not a flaw but the consequence of the baseline already taking the subgroup into account. Which baseline is reported is therefore the decision that determines what the study claims. #### Baselines Chosen in the Seven Courses of Machine Learning and Data Science Over seven courses every lesson chose a baseline and wrote down its reason. The table below collects those choices from the source lessons. The units are not shared and the rows are not added to one another. | Course | Chosen procedure that learns nothing | Unit | Example number | |---|---|---|---| | Introduction to Machine Learning | the most frequent class of the training set | accuracy | 0.7579 on the test set | | Data Preparation and Feature Engineering | the same setup with the preparation decision not applied | accuracy | between 0.7302 and 0.8294 | | Supervised Learning | the most frequent class; stating the mean in regression | accuracy, m³ error | 0.7579 and 6.5324 m³ on the test set | | Unsupervised Learning | random assignment, column mean, a single cluster | within-cluster distance, reconstruction error | random assignment 5.9790 | | Deep Learning | a separate most frequent class for each dataset | accuracy | subscriber 0.7579; grid 0.5850; sequence 0.7100 | | Natural Language Processing and Computer Vision | a domain-specific baseline | accuracy, overlap, vocabulary length | twelve baselines in ratio units between 0.0400 and 0.8731 | | Reinforcement Learning and Recommender Systems | random action; the most-chosen item | return; hit within the first five | 10.432 and 0.6527 | Two things can be read from the table. First, the baseline **changes type** from course to course: stating a class, stating a mean, behaving randomly, recommending the most chosen, or putting in the state where a decision was not applied. Second, the rows of the Deep Learning course and of the Natural Language Processing and Computer Vision course show that even within one course there is not a single baseline: four separate datasets, four separate baselines. Whenever the sentence "it beat the baseline" was written, what was beaten was a different procedure each time, and the table of this lesson gives the size of that difference. #### Summary - The baseline is not a tool but a design object. The same depth-6 decision tree reads between **−0.0119 and +0.5357** against eight baseline candidates; the choice alone moves the number by **0.5476**. - Making a baseline more complicated does not make it stronger: the most frequent class, the single-split rule and the region lookup table give the same 0.7579, because the majority flips in no region. - The baseline has a capacity too. The lookup table **beats the model** with 0.7897 in 15 cells, and in 39 cells it climbs to 0.7976 on train while falling to 0.7500 on test: the baseline overfits as well. - The choice of baseline changes direction in the subgroups. In the east region the model is +0.1389 against one baseline and −0.0833 against the other. The model that is below the baseline in 4 of 12 subgroups against the most frequent class is below it in **8** against the lookup table; the difference gap falls from 0.2173 to 0.1351. - In the seven courses of the Machine Learning and Data Science curriculum the baseline changed type, and even within a single course more than one baseline was used; in the sentence "it beat the baseline" what is beaten is a different procedure each time. #### Next Step Over three lessons fourteen separate number pairs were compared, and in all of them the difference was read directly: 0.0198, −0.0119, +0.1702. What was never asked in any of them is whether these differences can be told apart from **the noise of a 252-subscriber set**. A contribution of 0.1389 read from 36 subscribers in one subgroup could vanish if the class of a few subscribers flipped. Hypothesis testing, permutation testing and the bootstrap were measured in the Statistical Analysis and Exploratory Data Analysis course and will not be repeated. The next lesson measures only this: how large the **margin** of the difference between two models is, how much narrower a margin the **paired comparison** measured on the same samples gives than the independent comparison, and how the size of the held-out set determines that margin. ### Statistical Significance Source: https://academia.sh/en/courses/model-ethics-and-interpretation/statistical-significance Measuring the margin of the difference between two models and counting how much narrower a margin the paired comparison gives than the independent comparison: the margins of all the differences between the depth-6 decision tree and its four rivals contain zero, and only the 0.5357 difference stays outside. On the same 252 subscribers the paired margin runs from 0.0476 to 0.0794 and the independent margin from 0.1190 to 0.1310; the more the two models agree, the narrower the margin. As the set grows from 63 to 504, the paired margin falls from 0.1111 to 0.0556. Among the subgroups, the margin excludes zero for only four of thirteen differences, and +0.1389, one of the largest subgroup contributions of the course, is not one of them. Over three lessons fourteen separate number pairs were compared, and in all of them the difference was read directly: +0.0198 in one place, −0.0119 in another, +0.1702 in one subgroup. What was never asked in any of them is whether these differences can be told apart from **the noise of a set of 252 subscribers**. A contribution of 0.1389 read from thirty-six subscribers could vanish had the class of a few subscribers flipped. Hypothesis testing, permutation testing, the bootstrap and the margin of variation were measured in the Statistical Analysis and Exploratory Data Analysis course; the testing theory is not repeated here, and the same tool is applied only to model performance. This lesson measures three things: the margin of the difference between two models, how much measuring on the same samples (the **paired comparison**) narrows that margin, and how the size of the held-out set determines the margin. - **ED23.** The setup, the split (**756/252/252**), the seed, the metric (**accuracy**) and the subgroup definitions carry over unchanged from the earlier lessons: four axes, **13 subgroups**. - **ED24.** The model being compared is the **depth-6 decision tree** (0.7778 on the test set). The rivals: the most frequent class, the depth-2 tree, the depth-8 tree, the region and tier lookup table, and calling everyone suspicious. All of them were built in the earlier lessons. - **ED25.** The margin is computed by **bootstrap**: **400 repeats**, the 5% and 95% points. The method was measured in the Statistical Analysis and Exploratory Data Analysis course and is not justified here. - **ED26.** In the **paired comparison** the quantity that is resampled is the per-subscriber **difference** indicator: the same subscriber is right or wrong under both models, and the shared margin of error drops out. - **ED27.** In the **independent comparison** the numbers of the two models are resampled separately; this corresponds to the case where the two models were measured on two separate sets. - **ED28.** A margin that contains zero does not mean "the two models are the same"; it means **this difference could not be measured on this set**. A shortfall in measurement and equality are separate things. - **ED29.** In the set size sweep, the 504 row is built by combining the validation and test sets and exists **only to show how the margin narrows with size**; it is not a reportable performance number, because the validation set has been looked at before. - **ED30.** Subgroup margins are computed by the same procedure. In a small subgroup the margin is wide, and this is not a flaw but a direct consequence of sample size. #### The Difference Between Two Models and Its Share The first block builds the setup, compares five rivals with the tree on the same test set and prints two margins for each difference. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) MAJORITY = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN)) M, DEPTH2, DEPTH8 = tree(TRAIN, 6, FIELDS), tree(TRAIN, 2, FIELDS), tree(TRAIN, 8, FIELDS) REPEATS = 400 counts = {} for x in TRAIN: # region and tier lookup table baseline, carried over from 03 k = (x["region_name"], dict(subgroups(x))["tariff"]) counts.setdefault(k, [0, 0])[x["suspicious"]] += 1 votes = {k: int(v[1] > v[0]) for k, v in counts.items()} LOOKUP = (lambda x: votes.get((x["region_name"], dict(subgroups(x))["tariff"]), MAJORITY)) def paired_compare(rows, f, seed): # bootstrap was measured in the Statistical Analysis and Exploratory Data Analysis course; # here it is applied only to the DIFFERENCE hits_m = [int(predict(M, x) == x["suspicious"]) for x in rows] hits_f = [int(f(x) == x["suspicious"]) for x in rows] diff, n, paired, independent = [a - b for a, b in zip(hits_m, hits_f)], len(rows), [], [] for j in range(REPEATS): r, r1, r2 = rng(seed + j), rng(seed + 5000 + j), rng(seed + 9000 + j) paired.append(sum(diff[int(r() * n)] for _ in range(n)) / n) independent.append(sum(hits_m[int(r1() * n)] for _ in range(n)) / n - sum(hits_f[int(r2() * n)] for _ in range(n)) / n) paired.sort() independent.sort() low, high = int(0.05 * REPEATS), int(0.95 * REPEATS) - 1 return (sum(diff) / n, paired[low], paired[high], independent[low], independent[high], sum(1 for v in diff if v)) RIVALS = (("most frequent class", lambda x: MAJORITY), ("depth 2", lambda x: predict(DEPTH2, x)), ("depth 8", lambda x: predict(DEPTH8, x)), ("lookup table", LOOKUP), ("all suspicious", lambda x: 1)) print(f"{'rival':<19} {'disagree':>8} {'diff':>8} | {'paired':>21} {'width':>7}" f" | {'independent':>21} {'width':>7} | zero") for name, f in RIVALS: d, lo, hi, ilo, ihi, u = paired_compare(TEST, f, 71000) print(f"{name:<19} {u:>8} {d:>+8.4f} | [{lo:>+8.4f}, {hi:>+8.4f}] {hi - lo:>7.4f}" f" | [{ilo:>+8.4f}, {ihi:>+8.4f}] {ihi - ilo:>7.4f} | " f"{'outside' if lo > 0 or hi < 0 else 'inside'}") ``` ``` rival disagree diff | paired width | independent width | zero most frequent class 37 +0.0198 | [ -0.0198, +0.0556] 0.0754 | [ -0.0516, +0.0794] 0.1310 | inside depth 2 24 -0.0079 | [ -0.0437, +0.0238] 0.0675 | [ -0.0714, +0.0516] 0.1230 | inside depth 8 12 +0.0159 | [ -0.0079, +0.0397] 0.0476 | [ -0.0437, +0.0754] 0.1190 | inside lookup table 33 -0.0119 | [ -0.0556, +0.0238] 0.0794 | [ -0.0754, +0.0476] 0.1230 | inside all suspicious 215 +0.5357 | [ +0.4603, +0.6190] 0.1587 | [ +0.4683, +0.5992] 0.1310 | outside ``` The last column is the first result of the lesson. In **four** of the five comparisons the margin contains zero. The model's 0.0198 lead over the most frequent class, its 0.0119 loss to the lookup table, the 0.0079 between it and depth 2, the 0.0159 between it and depth 8 — all four cannot be measured on this set of 252 subscribers. Only the "call everyone suspicious" difference stays outside the margin, at 0.5357. The numbers in the tables of the previous three lessons were not computed wrongly; they are **below the measurement threshold**, and these are two separate things. The second result is the comparison of the two margin columns. The paired margin is narrower in every row: 0.0754 against 0.1310, 0.0675 against 0.1230, 0.0476 against 0.1190. The reason stands in the disagreement column. The depth-8 tree and the depth-6 tree part ways on only **12** of the 252 subscribers; on the remaining 240 both give the same answer, and the difficulty of those subscribers drops out of the computation of the difference entirely. In the independent measurement it does not drop out, because there the two models are assumed to have been measured on **separate sets** and each set's own variability of difficulty enters the margin. Measuring the two models on the same samples narrows the margin by **up to two and a half times**, without collecting any data. The last row reads in the opposite direction: because "call everyone suspicious" and the tree part ways on 215 subscribers, the paired margin there is **wider** than the independent one. Pairing does not always narrow; the more the two models agree, the more it narrows. #### The Set Size Determines the Share The width of the margin comes not from the model but from the set. The same comparison is repeated at different sizes taken from the head of the set. ```python POOL = VAL + TEST print(f"{'size':>6} | {'diff':>8} {'paired':>21} {'width':>7} | {'independent width':>17}") for n in (63, 126, 189, 252, 504): d, lo, hi, ilo, ihi, u = paired_compare(POOL[:n] if n > 252 else TEST[:n], LOOKUP, 74000) print(f"{n:>6} | {d:>+8.4f} [{lo:>+8.4f}, {hi:>+8.4f}] {hi - lo:>7.4f} | " f"{ihi - ilo:>17.4f}") ``` ``` size | diff paired width | independent width 63 | +0.0000 [ -0.0635, +0.0476] 0.1111 | 0.2540 126 | +0.0079 [ -0.0397, +0.0556] 0.0952 | 0.1667 189 | -0.0106 [ -0.0529, +0.0317] 0.0847 | 0.1376 252 | -0.0119 [ -0.0476, +0.0278] 0.0754 | 0.1270 504 | -0.0159 [ -0.0437, +0.0119] 0.0556 | 0.0794 ``` The difference column is unstable: 0.0000 from 63 subscribers, +0.0079 from 126, −0.0159 from 504. Even the sign changes. The margin column, on the other hand, narrows steadily — from 0.1111 to 0.0556, which is to say that as the set grows eightfold the margin is halved. This means the smallest measurable difference depends on the set size: on 252 subscribers no difference below 0.04 can be read, and most of the model differences reported throughout the Machine Learning and Data Science curriculum are below that threshold. The last row carries a warning. The set of 504 is the union of the validation and test sets, and the validation set was already looked at while the model was being chosen; the −0.0159 there is not a reportable performance number. The row stands only to show how the margin narrows with size. The size of the held-out set is therefore not a split ratio but a consequence of **the smallest difference one wants to measure**. #### In the Subgroups the Difference Disappears Inside the Share The subgroup gap was measured over three lessons. Now the margin of the difference in each subgroup is computed. ```python print(f"{'subgroup':<14} {'n':>4} {'disagree':>8} {'diff':>8} " f"{'paired margin':>22} {'zero':>8}") outside = 0 for g in SUBGROUPS: rows = [x for x in TEST if g in subgroups(x)] d, lo, hi, ilo, ihi, u = paired_compare(rows, lambda x: MAJORITY, 76000) outside += lo > 0 or hi < 0 print(f"{' '.join(g):<14} {len(rows):>4} {u:>8} {d:>+8.4f} [{lo:>+8.4f}, {hi:>+8.4f}]" f" {'outside' if lo > 0 or hi < 0 else 'inside':>8}") print(f"\nsubgroups whose margin excludes zero: {outside}/{len(SUBGROUPS)}") ``` ``` subgroup n disagree diff paired margin zero household 1-2 73 9 +0.0411 [ -0.0274, +0.1096] inside household 3-4 143 21 +0.0070 [ -0.0420, +0.0629] inside household 5+ 36 7 +0.0278 [ -0.0833, +0.1389] inside reading few 85 4 -0.0471 [ -0.0824, -0.0118] outside reading full 167 33 +0.0539 [ +0.0060, +0.1018] outside region central 58 7 +0.0172 [ -0.0690, +0.1034] inside region east 36 19 +0.1389 [ -0.0556, +0.3611] inside region north 68 7 -0.0441 [ -0.1176, +0.0147] inside region south 58 3 +0.0517 [ +0.0172, +0.1034] outside region west 32 1 -0.0312 [ -0.0938, +0.0000] inside tariff 10 12 1 -0.0833 [ -0.2500, +0.0000] inside tariff 25 193 22 -0.0104 [ -0.0466, +0.0311] inside tariff 40 47 14 +0.1702 [ +0.0426, +0.2766] outside subgroups whose margin excludes zero: 4/13 ``` The east region row is the harshest result of this lesson. The model's contribution there is **+0.1389**, and this was one of the largest subgroup contributions the course had measured so far. Its margin is **[−0.0556, +0.3611]** and contains zero. Nineteen disagreements among thirty-six subscribers spread the difference over an interval a third of a point wide. The same number was read in two lessons as "what the model adds in the east"; here it turns out that it cannot be read that way. Four subgroups leave zero outside, and one of them is in the opposite direction: among the 85 subscribers with few readings the difference is **−0.0471** and its margin is **[−0.0824, −0.0118]**. In that subgroup the model is measurably **worse** than stating the most frequent class. The remaining three are in the forward direction: south +0.0517, full readings +0.0539, the tariff 40 tier +0.1702. Two rules follow from this. The subgroup gap is the result of a design decision and **the gap itself can be measured**, but the individual differences that make it up cannot be measured in most subgroups. The second is that looking at thirteen subgroups and choosing the largest difference among them reproduces the multiple-comparison share counted in the Statistical Analysis and Exploratory Data Analysis course: stepping outside the margin in one of thirteen independent tests is not a finding unless it is written that thirteen looks were taken. #### Summary - In four of the five comparisons the margin contains zero: the model's 0.0198 lead over the most frequent class, its 0.0119 loss to the lookup table and the two depth differences **cannot be measured** on 252 subscribers. A margin that contains zero means a shortfall in measurement, not equality. - The paired comparison gives a narrower margin than the independent one: 0.0754 against 0.1310, 0.0476 against 0.1190. The narrowing comes from the agreement rate of the two models, and for a pair that parts ways on 215 subscribers pairing **widens** the margin. - The margin narrows with set size: 0.1111 at 63 subscribers, 0.0556 at 504. The size of the held-out set is not a split ratio but a consequence of the smallest difference one wants to measure. - Among the subgroups, the margin excludes zero for only **4** of the 13 differences. The +0.1389 in the east region, one of the largest subgroup contributions of the course, is not one of them: its margin is [−0.0556, +0.3611]. - Of the four differences that leave zero outside, one is in the opposite direction: among the subscribers with few readings the model is measurably below the baseline, at **−0.0471**. #### Next Step Over four lessons the measurement itself was designed: according to whom the set was split, from which cost the threshold was read, how the baseline was chosen, with which margin the difference was stated. The assumption they all share is that the test set comes from **the same distribution** as the data the model will meet. A measurement network changes over time: new regions join, tariff tiers shift, the rate of zero readings falls as meters are replaced. The next lesson measures that assumption: it reduces the difference between training and production data to a single number, places that number next to the drop in performance, and separates **covariate shift** from **label shift** to show why the two call for different solutions. ### Distribution Shift Source: https://academia.sh/en/courses/model-ethics-and-interpretation/distribution-shift Reducing the difference between training and production data to a single number and placing that number next to the drop in performance: a discriminator model reads a shift score of 0.1172 under covariate shift and the model's difference above the baseline falls from 0.0198 to 0.0066, but under label shift the same indicator stays at 0.0063 while the suspicious rate climbs from 0.2421 to 0.4865. The two shifts call for two separate fixes: prior correction raises accuracy from 0.6306 to 0.6622 under label shift while changing no decision under covariate shift, and retraining on the input distribution raises 0.7318 to 0.7384 under covariate shift and drops label shift to 0.6261. On the time-ordered production set the subgroup gap falls from 0.3576 to 0.2573, and in none of the twelve subgroups does the model stay below the baseline. In four lessons the measurement itself was designed: which way the sets were split, which cost the threshold was read from, how the baseline was chosen, which margin the difference was stated with. All four assume that the test set comes from the **same distribution** as the data the model will meet. A measurement network changes over time: new regions join, tariff tiers move, and the zero-reading rate falls as meters are replaced. This lesson measures that assumption. **Distribution shift** comes in two forms, and when the two are gathered under one name the wrong fix gets chosen. In **covariate shift** the distribution of the features changes, but the probability of being suspicious given a subscriber's features stays the same. In **label shift** the rate of suspicion changes. Data drift was mentioned in the Introduction to Machine Learning course and the time-ordered split was measured there; what is added here is **reducing the shift to a single number** and separating the two kinds. - **ED31.** The setup, the split (**756/252/252**), the seed, the metric (**accuracy**), the model (**depth-6 decision tree**, suspicious rate in the leaf) and the subgroup definitions carry over from the earlier lessons: four axes, **13 subgroups**. The suspicious rate of the training set is **0.2222**. - **ED32.** The **shift score** is measured with a **discriminator model**: a depth-4 tree that predicts whether a row came from the training set or the production set, trained with an equal number of rows from the two sets and measured on a held-out half. The number is **accuracy minus 0.5**; a value near zero means "cannot be told apart". The mean of five runs is taken, and the label column is **not given** to the discriminator. - **ED33.** The shift score sees only the **input** distribution. It cannot see a shift in the label distribution, and this is one of the things the lesson measures. - **ED34.** The three constructed production sets are derived from the union of the validation and test sets (504 subscribers). **Covariate shift**: the probability of selecting subscribers with a high mean consumption is raised, and the label rule is not touched. **Label shift**: the suspicious subscribers are kept and only 30% of the non-suspicious ones are retained. **Time-ordered production**: the last 20% of the set sorted by subscriber number; the subscriber number is the order of joining the network. - **ED35.** **Prior correction** multiplies the model's leaf rate by the ratio of the odds of suspicion in production and in training; it requires knowing labels in production, which is a cost. **Retraining** builds the same tree on a training set of **439** rows sampled with the covariate shift rule; 317 of the 756 rows are dropped, and the lost data is a cost. - **ED36.** The model is trained once and the same model is measured on four sets; the difference comes from the data, not from the model. The sizes of the production sets are not equal (302, 222, 252), and the margin calculations were measured in the previous lesson and are not repeated. #### The Size of the Shift Is a Number The first block sets up the scenario, produces the three production sets, measures the shift score for each, and prints on one row the baseline, the model's number and the results of the two fixes. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def score(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from lesson 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M = tree(TRAIN, 6, FIELDS) RATE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN) MAJORITY = int(RATE > 0.5) def hit_rate(f, rows): return sum(f(x) == x["suspicious"] for x in rows) / len(rows) def shift_score(a, b, seed, repeats=5): # DISCRIMINATOR MODEL: a depth-4 tree that predicts which set a row came from t = 0.0 for j in range(repeats): n = min(len(a), len(b)) ka = [dict(x, suspicious=0) for x in shuffle(a, seed + j)[:n]] kb = [dict(x, suspicious=1) for x in shuffle(b, seed + 500 + j)[:n]] d = tree(shuffle(ka[:n // 2] + kb[:n // 2], seed + 900 + j), 4, FIELDS) s = ka[n // 2:] + kb[n // 2:] t += sum((score(d, x) > 0.5) == x["suspicious"] for x in s) / len(s) return t / repeats - 0.5 def covariate_shift(rows, seed, power=0.12): # high consumption weighs more; the label rule is the SAME r = rng(seed) return [x for x in rows if r() < min(0.98, 0.03 + power * max(0.0, x["mean_consumption"] - 14))] def label_shift(rows, seed, keep=0.30): # suspicious rows are kept, 30% of the non-suspicious remain r = rng(seed) return [x for x in rows if x["suspicious"] or r() < keep] POOL = VAL + TEST TIME_SET = three_way_split(sorted(DATA, key=lambda x: x["id"]))[2] COVARIATE_SET, LABEL_SET = covariate_shift(POOL, 83000), label_shift(POOL, 84000) M_RETRAINED = tree(covariate_shift(TRAIN, 86000), 6, FIELDS) def prior_correction(rows): # prior correction: with the ratio of odds p1 = sum(x["suspicious"] for x in rows) / len(rows) o = (p1 / (1 - p1)) / (RATE / (1 - RATE)) return lambda x: int(score(M, x) * o / (score(M, x) * o + 1 - score(M, x)) >= 0.5) print(f"{'set':<18} {'n':>4} {'suspicious':>10} {'shift':>8} {'base':>7} {'model':>7} " f"{'diff':>8} {'prior':>7} {'retrain':>7}") for name, s in (("same distribution", TEST), ("covariate shift", COVARIATE_SET), ("label shift", LABEL_SET), ("time ordered", TIME_SET)): base = hit_rate(lambda x: MAJORITY, s) print(f"{name:<18} {len(s):>4} {sum(x['suspicious'] for x in s)/len(s):>10.4f} " f"{shift_score(TRAIN, s, 85000):>+8.4f} {base:>7.4f} " f"{hit_rate(lambda x: int(score(M, x) > 0.5), s):>7.4f} " f"{hit_rate(lambda x: int(score(M, x) > 0.5), s)-base:>+8.4f} " f"{hit_rate(prior_correction(s), s):>7.4f} {hit_rate(lambda x: int(score(M_RETRAINED, x) > 0.5), s):>7.4f}") ``` ``` set n suspicious shift base model diff prior retrain same distribution 252 0.2421 -0.0127 0.7579 0.7778 +0.0198 0.7817 0.7698 covariate shift 302 0.2748 +0.1172 0.7252 0.7318 +0.0066 0.7318 0.7384 label shift 222 0.4865 +0.0063 0.5135 0.6306 +0.1171 0.6622 0.6261 time ordered 252 0.3532 -0.0056 0.6468 0.7500 +0.1032 0.7579 0.7381 ``` The first row validates the measurement. Training and test sets came from the same split, so the discriminator model cannot separate them and the shift score is **−0.0127**; the minus sign is not a direction, it is run noise on a number indistinguishable from zero. In the second row the shift score is **+0.1172**: the discriminator model predicts which set a row came from 11.7 points better than random guessing, so there is a real feature-level difference. On the performance side the counterpart is this: the model drops from 0.7778 to 0.7318, and its difference above the baseline falls **from 0.0198 to 0.0066**. The size of the shift and the erosion of the contribution are read on one row. The third and fourth rows give the distinction of the lesson. Under label shift the suspicious rate climbs from 0.2421 to **0.4865**, and under time-ordered production to **0.3532**; the sets are unrecognizably different. Yet the shift score is **+0.0063** and **−0.0056**, which is zero. The indicator that measures covariate shift sees nothing even when half the labels have changed, because it looks only at the distribution of the features. **The silence of the shift indicator does not mean there is no shift.** #### Two Shifts Call for Two Separate Fixes The last two columns carry the two fixes, and each works only on its own shift. Prior correction rescales the leaf rate of the model to the new suspicion rate. Under label shift it raises accuracy **from 0.6306 to 0.6622**, and under time-ordered production **from 0.7500 to 0.7579**. Under covariate shift it does not move the number at all: 0.7318 and 0.7318. The reason is arithmetic: there the suspicious rate has gone only to 0.2748, the ratio of the odds is close to one, and no decision flips. It has a cost too: knowing the suspicion rate in production requires collecting labels there. Retraining works in the opposite direction. The tree built on a 439-row training set sampled with the covariate shift rule rises **from 0.7318 to 0.7384** under covariate shift; on the other three sets it **lowers** the number (0.7698, **0.6261**, 0.7381). A third of the training set was thrown away, so the model learns from less data, and this loss is made up only on the targeted shift. The rule is this: a fix chosen **without determining the type** of the shift does harm. Collecting labels for covariate shift, throwing data away for label shift: both pull the number down here. #### The Shift Is Not Equal in the Subgroups The time-ordered production set is the most realistic of the four: as the network grows, later subscribers enter the set. The second block compares the share and the suspicious rate of the subgroups in that set with those in the training set. ```python print(f"{'subgroup':<14} | {'train share':>11} {'prod share':>11} | {'train susp':>10} " f"{'prod susp':>10} | {'n':>4} {'base':>7} {'model':>7} {'diff':>8}") accs, gaps = [], [] for g in SUBGROUPS: e = [x for x in TRAIN if g in subgroups(x)] u = [x for x in TIME_SET if g in subgroups(x)] base, mod = hit_rate(lambda x: MAJORITY, u), hit_rate(lambda x: int(score(M, x) > 0.5), u) if len(u) >= 25: accs.append(mod) gaps.append(mod - base) print(f"{' '.join(g):<14} | {len(e)/len(TRAIN):>11.4f} {len(u)/len(TIME_SET):>11.4f} | " f"{sum(x['suspicious'] for x in e)/len(e):>10.4f} " f"{sum(x['suspicious'] for x in u)/len(u):>10.4f} | {len(u):>4} {base:>7.4f} " f"{mod:>7.4f} {mod-base:>+8.4f}{' n<25' if len(u) < 25 else ''}") print(f"\nproduction set gap ({len(accs)} subgroups): best {max(accs):.4f}, worst " f"{min(accs):.4f}, gap {max(accs)-min(accs):.4f}; below baseline " f"{sum(1 for v in gaps if v < 0)}/{len(gaps)}") ``` ``` subgroup | train share prod share | train susp prod susp | n base model diff household 1-2 | 0.3148 0.3135 | 0.2227 0.4177 | 79 0.5823 0.6962 +0.1139 household 3-4 | 0.5265 0.5476 | 0.2060 0.3188 | 138 0.6812 0.7754 +0.0942 household 5+ | 0.1587 0.1389 | 0.2750 0.3429 | 35 0.6571 0.7714 +0.1143 reading few | 0.3373 0.3294 | 0.1725 0.2892 | 83 0.7108 0.7831 +0.0723 reading full | 0.6627 0.6706 | 0.2475 0.3846 | 169 0.6154 0.7337 +0.1183 region central | 0.1772 0.1905 | 0.2761 0.4375 | 48 0.5625 0.6458 +0.0833 region east | 0.1799 0.1548 | 0.4706 0.6667 | 39 0.3333 0.7692 +0.4359 region north | 0.2897 0.2937 | 0.1963 0.2973 | 74 0.7027 0.7027 +0.0000 region south | 0.1865 0.2183 | 0.1064 0.2727 | 55 0.7273 0.8000 +0.0727 region west | 0.1667 0.1429 | 0.0714 0.1389 | 36 0.8611 0.8889 +0.0278 tariff 10 | 0.0384 0.0476 | 0.2069 0.4167 | 12 0.5833 0.6667 +0.0833 n<25 tariff 25 | 0.7804 0.7262 | 0.1712 0.2732 | 183 0.7268 0.7923 +0.0656 tariff 40 | 0.1812 0.2262 | 0.4453 0.5965 | 57 0.4035 0.6316 +0.2281 production set gap (12 subgroups): best 0.8889, worst 0.6316, gap 0.2573; below baseline 0/12 ``` The first two columns are nearly stationary: the largest change is in tariff 25, from 0.7804 to 0.7262. The composition of the subgroups has not changed, so covariate shift is small here, and the shift score of −0.0056 agrees. In the next two columns the suspicious rate of **every subgroup** rises, but not equally: in the east from 0.4706 to 0.6667, in the household 1-2 group from 0.2227 to 0.4177, in the tariff 25 tier only from 0.1712 to 0.2732. Label shift looks like a single number overall, and like thirteen separate numbers in the subgroups. The last row completes the course's triple and goes in an unexpected direction. The subgroup gap in the production set is **0.2573**; the gap measured for the same model in the first lesson was 0.3576. The shifted set **brings the groups closer together**. Harsher still, in **none** of the twelve subgroups does the model stay below the baseline; on the unshifted set the same model with the same metric was below it in five. The reason is the baseline itself: as the suspicious rate rises, saying "none is suspicious" gets weaker in every subgroup, and the model passes it. A number measured on a shifted set shows that the baseline has worsened, not that the model has improved, and a report without the baseline beside it cannot tell the two apart. #### Summary - The shift comes down to one number: a discriminator model predicts whether a row came from training or from production, and the difference of the accuracy from 0.5 is the **shift score**. It is +0.1172 under covariate shift and −0.0127 on the same distribution. - The shift score sees only the input. Under label shift the suspicious rate climbs from 0.2421 to 0.4865 while the indicator stays at +0.0063, and under time-ordered production at −0.0056: **the silence of the indicator does not mean there is no shift**. - The size of the shift is read next to the drop in performance: under covariate shift the model's difference above the baseline falls from 0.0198 to **0.0066**. - The two shifts call for two separate fixes. Prior correction raises 0.6306 to 0.6622 under label shift and changes no decision under covariate shift. Retraining on the input distribution raises 0.7318 to 0.7384 under covariate shift and lowers the number on the other three sets. - Under time-ordered production the subgroup shares are stationary but the suspicious rates rise unequally (+0.1961 in the east, +0.1020 in tariff 25). The gap falls from 0.3576 to **0.2573**, and the model is no longer below the baseline in any subgroup, because what falls is not the model but the baseline. #### Next Step In five lessons, what a number **is** and **whom it is for** were measured: which way the unit of separation was chosen, which cost the threshold was read from, in how many ways the baseline could be built, which margin the difference was stated with, how far the production data had shifted. At every step the number produced by the same tree said something else in a subgroup, and the gap was re-measured between 0.2573 and 0.8606. Throughout all of this the model stayed a **closed box**. What the model **based** that number on was never asked: which feature carries the decision, which value the decision for a single subscriber rests on, and whether that rationale, once written down, describes what the model actually does. The next topic measures these three questions. ## Interpretability Measuring whether what the model bases its number on can be read: how many of the questions that can be asked about a decision are answered by the output and how many by an explanation, in how many ranks the importance ranking read from the model and the ranking measured by permuting disagree, and how two correlated features split the importance, how many examples the local surrogate describes the decision of the original model correctly and how two methods give two different explanations for the same example, how a partial dependence that looks flat overall slopes in opposite directions in two subgroups, where the accuracy arrives as the rule list grows, and in which subgroup the best family cannot be approached. ### The Need for Interpretability Source: https://academia.sh/en/courses/model-ethics-and-interpretation/need-for-interpretability Counting how many of the twelve questions that can be asked about a decision are closed by the model's output, how many by an explanation, and how many by neither: the output alone closes 4 questions, the explanation raises the average to 8.58, and three questions are closed by no explanation. The one-step change at which the decision flips exists for 159 subscribers of the test set and for 2 of the thirty-two subscribers in the west region; thirteen subgroups were examined, and among the twelve that pass the threshold the number of closed questions ranges from 8.89 at best to 7.88 at worst, a gap of 1.02 questions. The previous topic measured in five lessons **what** the number is and **for whom** it is: the share the split inflated, the operating point the cost matrix moved, the choice of baseline, the margin of the difference between two models, and distribution shift. In all five, what was measured was the model's **output**. What the model **based** that output on was never asked. This question is usually presented as a matter of preference and is generally backed by a list of requirements. No list of requirements is written here; what is measured is a **function**: that the rationale for a decision made about a subscriber can be **documented**. Documentability comes down to a concrete number. There is a set of questions that can be asked about a decision; how many of that set are answered by looking at the model's output, how many by looking at an explanation, and how many by looking at neither. That is the measure of this lesson, and it is read separately in the subgroups. - **IN1.** The set, the label, the seed and the split are the **fictional** setup used throughout the first seven courses of the Machine Learning and Data Science curriculum: the subscriber table derived from the regional measurement network, the label for suspected illicit use or a fault, seed 20260218, split 756/252/252, baseline 0.7579 on the test set. - **IN2.** The model to be explained is the linear model chosen in the Supervised Learning course: twelve scaled columns, learning rate 0.3, 300 epochs, 0.8016 on the test set. In this topic the model is **not chosen but taken as given**; what is measured is not the model's performance but the readability of its decision. - **IN3.** The subgroups are fixed throughout this topic and follow the four axes set up in the Evaluation Design topic: five regions, three tariff tiers (10, 25 and 40 cubic meters), three household size ranges, and two reading levels (a subscriber with readings in all three periods and one with fewer). **The number of subgroups examined is 13.** - **IN4.** A subgroup's number is read only if it has at least 25 records in the test set. A group that falls below the threshold enters the table, but its number does not count as a finding. - **IN5.** A counterfactual is a **one-step** change in a single column. The step is one standard deviation in the training set for numeric columns, a flip of the value for two-valued columns, and a move to another region for the region. These three steps are not the same size, and that is what makes the comparison hard. - **IN6.** Consumption per person is derived from mean consumption and household size; when its source is changed it is recomputed, and it is not changed on its own. - **IN7.** A question counts as closed if its answer can be read **mechanically** from the model's output or from an explanation. A question that requires expert judgment, field verification or a policy decision is open. #### Questions That Can Be Asked About a Decision When the subscriber in the setup is flagged as suspicious, the questions of the person who will listen to them on the phone, of the team that will go to the field, and of the person who will later audit the decision are not the same. The twelve questions are the union of the questions of these three people. What the model's output closes is four: did the subscriber receive the suspicious flag, what is the score and how close is it to the threshold, at what threshold is the decision made, and when the same row is submitted again does the same flag come out. What an explanation can close is five: which column pushes the score most, in which direction does that column push, which single column's one-step change flips the flag, do the other subscribers who receive the same flag rely on the same column, and in this subscriber's subgroup is the flag rate different from the overall one. What neither closes is three: is the flag correct, does the suspicion come from illicit use or from a fault, and what should be done after this flag. The first requires field verification, the second is already not separated in the setup's label, and the third is a policy decision. ```python # interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed; # the twelve-column feature set and the split stay fixed throughout the course. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def scale(train, fields): return ({a: statistics.fmean(x[a] for x in train) for a in fields}, {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields}) def train_model(train, fields, o, s, rate=0.3, epoch=300): # MODEL: logistic regression X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train] Y = [x["suspicious"] for x in train] w = [0.0] * len(X[0]) for _ in range(epoch): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= rate * g[j] / len(X) return w def score(w, fields, o, s, x): return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a] for j, a in enumerate(fields))) def decision(w, fields, o, s, x): return int(score(w, fields, o, s, x) > 0.5) O, S = scale(TRAIN, FIELDS) W = train_model(TRAIN, FIELDS, O, S) ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") print(f"baseline {BASELINE:.4f}, test accuracy of the linear model {ACCURACY:.4f}") ``` ``` train 756, validation 252, test 252 baseline 0.7579, test accuracy of the linear model 0.8016 ``` #### The One Step at Which the Decision Flips Three of the five explanation questions close for every subscriber, because the linear model's score is the sum of one term per column, and those terms can always be read. The fourth — **which single column's one-step change flips the flag** — may not close. For a subscriber who stands far enough from the decision boundary, no one-step change flips the flag, and that question goes unanswered. This is not a shortcoming of the explanation method but the **certainty** of that subscriber's decision; but the result leads to the same place: that row stays empty in the rationale document. ```python def tariff_tier(x): return ("tariff 10" if x["mean_consumption"] <= 10 else "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40") def household_group(x): return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4 else "household 5+") def reading_group(x): return "readings few" if x["period"] <= 2 else "readings all" def equals(f, value): return lambda x: f(x) == value SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION] + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")] + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")] + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")]) MIN_SIZE, SIZES = 25, {} for name, f in SUBGROUPS: SIZES[name] = sum(1 for x in TEST if f(x)) SHORT = [name for name in SIZES if SIZES[name] < MIN_SIZE] print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, " f"below the threshold {SHORT}") SINGLE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "region_name"] def change(x, a, v): d = dict(x) d[a] = v if a in ("mean_consumption", "household"): d["per_person"] = round(d["mean_consumption"] / d["household"], 3) if a == "region_name": for b in REGION: d["area_" + b[0]] = int(v == b[0]) return d def counterfactual(x): # one-step change in a single column k = decision(W, FIELDS, O, S, x) for a in SINGLE: if a == "zero_readings": candidates = [1 - x[a]] elif a == "region_name": candidates = [b[0] for b in REGION if b[0] != x["region_name"]] else: candidates = [x[a] + S[a], x[a] - S[a]] for v in candidates: if decision(W, FIELDS, O, S, change(x, a, v)) != k: return a return None COUNTERFACTUALS = [counterfactual(x) for x in TEST] EXAMPLE = TEST[4] print(f"\nexample subscriber: region {EXAMPLE['region_name']}, household {EXAMPLE['household']}, " f"readings {EXAMPLE['period']}, mean consumption {EXAMPLE['mean_consumption']}") print(f" score {score(W, FIELDS, O, S, EXAMPLE):.4f}, decision " f"{decision(W, FIELDS, O, S, EXAMPLE)}, counterfactual column {COUNTERFACTUALS[4]}") print(f"subscribers with a counterfactual in the whole test set " f"{sum(1 for k in COUNTERFACTUALS if k)}/{len(TEST)} " f"({sum(1 for k in COUNTERFACTUALS if k) / len(TEST):.4f})") ``` ``` subgroup count 13, threshold 25 records, below the threshold ['tariff 10'] example subscriber: region central, household 4, readings 2, mean consumption 34.16 score 0.4768, decision 0, counterfactual column mean_consumption subscribers with a counterfactual in the whole test set 159/252 (0.6310) ``` The example subscriber's score of 0.4768 is just below the threshold, and when mean consumption rises by one standard deviation the flag flips. For this subscriber, the row in the rationale document can be written. For 159 of the test set's 252 subscribers such a row can be written; for the remaining 93 subscribers, no one-step change flips the flag. #### Number of Closed Questions in the Subgroups The 0.6310 measured overall is not read on its own. The same number is measured in thirteen subgroups separately and converted into a number of closed questions: for every subscriber, four questions are closed by the output, three by the explanation that can always be produced, the fourth explanation question closes if a counterfactual exists, and the fifth closes if all of the subscriber's subgroups are above the threshold size. ```python FIXED = 4 + 3 # output closes four, three explanation questions always close def closed(i, x): readable = all(SIZES[name] >= MIN_SIZE for name, f in SUBGROUPS if f(x)) return FIXED + (1 if COUNTERFACTUALS[i] else 0) + (1 if readable else 0) ALL = sum(closed(i, x) for i, x in enumerate(TEST)) / len(TEST) print(f"{'subgroup':<16}{'records':>8}{'counterfactual rate':>21}{'closed questions':>18}") ROWS = [] for name, f in SUBGROUPS: g = [(i, x) for i, x in enumerate(TEST) if f(x)] cf_rate = sum(1 for i, x in g if COUNTERFACTUALS[i]) / len(g) closed_mean = sum(closed(i, x) for i, x in g) / len(g) ROWS.append((name, closed_mean)) print(f"{name:<16}{len(g):>8}{cf_rate:>21.4f}{closed_mean:>18.2f}") lowest = min(ROWS, key=lambda t: t[1]) highest = max(ROWS, key=lambda t: t[1]) print(f"\nclosed questions overall {ALL:.2f}/12; output alone 4/12; " f"closed by no explanation 3/12") print(f"best subgroup {highest[0]} {highest[1]:.2f}, worst subgroup {lowest[0]} " f"{lowest[1]:.2f}, gap {highest[1] - lowest[1]:.2f} questions") ``` ``` subgroup records counterfactual rate closed questions region north 68 0.8235 8.82 region south 58 0.3793 8.31 region east 36 0.8333 8.81 region west 32 0.0625 7.88 region central 58 0.8448 8.83 tariff 10 12 0.2500 7.25 tariff 25 193 0.5907 8.59 tariff 40 47 0.8936 8.89 household 1-2 73 0.5753 8.49 household 3-4 143 0.6503 8.62 household 5+ 36 0.6667 8.64 readings few 85 0.5647 8.49 readings all 167 0.6647 8.63 closed questions overall 8.58/12; output alone 4/12; closed by no explanation 3/12 best subgroup tariff 40 8.89, worst subgroup tariff 10 7.25, gap 1.64 questions ``` Overall, the explanation raises the number of closed questions from 4 to 8.58. In the subgroups, the same number is spread between 7.88 and 8.89 across the twelve groups that pass the threshold, and the gap is 1.02 questions; if tariff 10, which falls below the threshold, is also counted, the lower end drops to 7.25. The counterfactual rate, on the other hand, separates far more sharply: 0.8448 in the central region, 0.0625 in the west region. In the west region, the one step at which the decision flips can be shown for two of the thirty-two subscribers. The reason can be read in the setup: the west's base consumption of 14 cubic meters is the lowest of the five regions, so the subscribers there stand far from the decision boundary and no movement of one standard deviation takes them to the other side. Whether the decision changes is read here. Someone looking at the overall number writes the sentence "the rationale for the decisions can be documented." Someone looking at the subgroup cannot write the same sentence for the west region; in the rationale document of the subscribers there, the counterfactual row is empty. The number of the lowest tariff tier is not even read, because it has 12 records in the test set and the threshold is 25. The number of subgroups examined is 13, and the number read is 12. That three questions are closed by no explanation also sits in the same table, and it is the most important row for this lesson. An explanation tells **how** the decision was produced; it does not tell whether the decision is **correct**, what the **source** of the suspicion is, or **what to do next**. Closing these three questions takes not an explanation but field data, a finer label, and a written policy. #### Summary - Of the twelve questions that can be asked about a decision, 4 can be closed by the model's output, 5 by an explanation, and 3 by neither; the three that stay open are correctness, the source of the suspicion, and what to do next. - Overall, the average number of closed questions is 8.58/12; without an explanation this number stays at 4/12. - The one-step change at which the decision flips exists for 159 subscribers of the test set, a rate of 0.6310. - Thirteen subgroups were examined, and the 12 that pass the 25-record threshold were read; among the groups read, the number of closed questions ranges from 7.88 to 8.89, and the gap is 1.02 questions. - The counterfactual rate is 0.8448 in the central region and 0.0625 in the west region; because the west has the lowest base consumption, the subscribers there stand far from the decision boundary. #### Next Step In this lesson, three of the five explanation questions were passed over as "always closes," and the reason was that the linear model's score can be decomposed into one term per column. The ranking of those terms is read as the **global** answer to which column the model relies on, and the Data Preparation and Feature Engineering course and the Supervised Learning course had already measured that ranking. But a ranking is not produced in only one way. Measuring a column by **permuting** it instead of reading it from the model — shuffling its values among themselves and looking at how much performance drops — gives a second ranking, and there is no reason for the two rankings to come out the same. The next lesson puts the two rankings side by side, counts in how many ranks they disagree, and measures how mean consumption and consumption per person, two forms of the same quantity, split the importance. ### Feature Importance Source: https://academia.sh/en/courses/model-ethics-and-interpretation/feature-importance Measuring two global importance rankings side by side on the same model: the ranking read from the coefficients and the ranking measured by permuting the column disagree on ten of the twelve columns, and the largest shift is five ranks. Volatility is third in the coefficient ranking and eighth in the permutation ranking. Two correlated columns split the importance: the importance of consumption per person is 0.0018 in the full model and rises to 0.0387 when mean consumption is removed and the model is retrained. Across thirteen subgroups, the loss increase of the most important column ranges from 0.0162 to 0.1882, and in one subgroup the first place changes. The previous lesson said that three of the five explanation questions close for every subscriber, and the reason was that the linear model's score can be decomposed into one term per column. Ranking those terms by size is read as the **global explanation** of which column the model relies on: it is an answer that belongs not to a single subscriber but to the model as a whole. This ranking itself is not new. The Feature Selection topic of the Data Preparation and Feature Engineering course used importance as a selection criterion, and the Linear Models topic of the Supervised Learning course measured that the value of a coefficient depends on the set of columns beside it. Neither is repeated here. The question of this lesson is different and takes the answers of the earlier ones as input: **importance is not measured in a single way.** Reading a column from the model gives one ranking; **permuting** the column and looking at how much performance drops gives a second ranking. There is no reason for the two rankings to come out the same, and how far they disagree can be measured. - **IN8.** The setup is that of the previous lesson: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579, the twelve-column linear model at 0.8016 on the test set. The subgroups and the 25-record reading threshold continue unchanged; **the number of subgroups examined is 13**. - **IN9.** Importance read from the model is the ranking by the absolute value of the coefficients of the scaled columns. How this ranking is produced was measured in the earlier courses; here it is taken as input. - **IN10.** Importance measured by permuting is the shuffling of a single column's values among themselves within the test set and the measurement of the drop in performance. Shuffling changes the row order, not the set of values of the column; the model is not retrained. - **IN11.** The drop is measured not in accuracy but in **mean loss**. On a set of 252 records, accuracy changes in steps of 0.004 and a single subscriber can move the ranking; loss is a continuous number and can be read in small subgroups too. - **IN12.** Shuffling is done with 10 repeats and averaged; the seeds are derived from 20260218. - **IN13.** An importance ranking shows what the model **relies on**; it does not show a causal effect. The limit of causal interpretation had been measured separately in the Applied Statistics topic of the Data Analytics curriculum. #### Two Rankings The setup is the same as in the previous lesson, and the subgroups are defined in the same place. ```python # interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed; # the twelve-column feature set and the split stay fixed throughout the course. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def scale(train, fields): return ({a: statistics.fmean(x[a] for x in train) for a in fields}, {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields}) def train_model(train, fields, o, s, rate=0.3, epoch=300): # MODEL: logistic regression X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train] Y = [x["suspicious"] for x in train] w = [0.0] * len(X[0]) for _ in range(epoch): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= rate * g[j] / len(X) return w def score(w, fields, o, s, x): return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a] for j, a in enumerate(fields))) def decision(w, fields, o, s, x): return int(score(w, fields, o, s, x) > 0.5) O, S = scale(TRAIN, FIELDS) W = train_model(TRAIN, FIELDS, O, S) ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") print(f"baseline {BASELINE:.4f}, test accuracy of the linear model {ACCURACY:.4f}") def tariff_tier(x): return ("tariff 10" if x["mean_consumption"] <= 10 else "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40") def household_group(x): return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4 else "household 5+") def reading_group(x): return "readings few" if x["period"] <= 2 else "readings all" def equals(f, value): return lambda x: f(x) == value SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION] + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")] + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")] + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")]) MIN_SIZE = 25 SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS} print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, below the threshold " f"{[a for a in SIZES if SIZES[a] < MIN_SIZE]}") ``` ``` train 756, validation 252, test 252 baseline 0.7579, test accuracy of the linear model 0.8016 subgroup count 13, threshold 25 records, below the threshold ['tariff 10'] ``` Importance measured by permuting is a single operation: the values of a column are shuffled among themselves within the test set, so the column keeps its own distribution but loses its link to the row; the model is measured again on that corrupted set, and the increase in loss is looked at. ```python def loss(w, fields, o, s, rows): # accuracy changes in steps of 0.004 on 252 records t = 0.0 for x in rows: p = min(max(score(w, fields, o, s, x), 1e-9), 1 - 1e-9) t -= math.log(p) if x["suspicious"] else math.log(1 - p) return t / len(rows) def permute(w, fields, o, s, rows, permuted, repeats=10, seed=SEED + 400000): l0, t = loss(w, fields, o, s, rows), 0.0 for r in range(repeats): order = shuffle(list(range(len(rows))), seed + 977 * r) new = [] for i, x in enumerate(rows): d = dict(x) for a in permuted: d[a] = rows[order[i]][a] new.append(d) t += loss(w, fields, o, s, new) return t / repeats - l0 PERM = {a: permute(W, FIELDS, O, S, TEST, [a]) for a in FIELDS} FROM_MODEL = sorted(FIELDS, key=lambda a: -abs(W[FIELDS.index(a) + 1])) BY_PERMUTE = sorted(FIELDS, key=lambda a: -PERM[a]) print(f"{'column':<18}{'coefficient':>13}{'model rank':>11}{'loss increase':>15}" f"{'permutation rank':>18}{'shift':>7}") disagree = 0 for a in FROM_MODEL: ra, rb = FROM_MODEL.index(a) + 1, BY_PERMUTE.index(a) + 1 disagree += ra != rb print(f"{a:<18}{W[FIELDS.index(a) + 1]:>+13.3f}{ra:>11}{PERM[a]:>15.4f}" f"{rb:>18}{rb - ra:>+7}") print(f"\nranks that disagree {disagree}/{len(FIELDS)}, largest shift " f"{max(abs(FROM_MODEL.index(a) - BY_PERMUTE.index(a)) for a in FIELDS)}") print(f"top three of the model ranking {FROM_MODEL[:3]}") print(f"top three of the permutation ranking {BY_PERMUTE[:3]}") ``` ``` column coefficient model rank loss increase permutation rank shift mean_consumption +0.772 1 0.1332 1 +0 zero_readings +0.684 2 0.0564 2 +0 volatility +0.380 3 0.0012 8 +5 area_east +0.306 4 0.0146 3 -1 satisfaction -0.273 5 0.0016 7 +2 household +0.156 6 0.0053 5 -1 period +0.148 7 -0.0004 10 +3 area_west -0.147 8 0.0054 4 -4 area_south -0.109 9 -0.0011 11 +2 per_person +0.071 10 0.0018 6 -4 area_central -0.048 11 -0.0014 12 +1 area_north -0.005 12 0.0002 9 -3 ranks that disagree 10/12, largest shift 5 top three of the model ranking ['mean_consumption', 'zero_readings', 'volatility'] top three of the permutation ranking ['mean_consumption', 'zero_readings', 'area_east'] ``` Ten of the twelve columns stand in different places in the two rankings. The first two ranks are shared; the disagreement begins from the third rank. The largest shift is in volatility: third in the coefficient ranking, eighth in the permutation ranking. With 0.380 the coefficient carries a respectable size, but shuffling the column raises the loss by only 0.0012. The two numbers do not contradict each other; they say two different things. The coefficient says what an increase of one standard deviation in volatility adds to the score **when the other columns are held constant**. Permutation, on the other hand, measures **whether the information volatility carries also exists in other columns**: because volatility moves together with mean consumption, the model closes the gap when it is shuffled. Three columns have a negative sign on their loss increase. If shuffling a column has lowered the loss, that column is harming the model; since their magnitudes are 0.0014 and below, what is to be read here is not the sign itself but that the three columns stand **near zero**. #### How Two Correlated Columns Split the Importance Mean consumption and consumption per person are two forms of the same quantity; the second is produced by dividing the first by household size. In the permutation ranking one is first and the other is sixth. ```python PAIR = ["mean_consumption", "per_person"] together = permute(W, FIELDS, O, S, TEST, PAIR) print(f"one by one {PERM[PAIR[0]]:.4f} and {PERM[PAIR[1]]:.4f}, total " f"{PERM[PAIR[0]] + PERM[PAIR[1]]:.4f}, together {together:.4f}") for dropped in PAIR: remaining = [a for a in FIELDS if a != dropped] o2, s2 = scale(TRAIN, remaining) w2 = train_model(TRAIN, remaining, o2, s2) acc = sum(decision(w2, remaining, o2, s2, x) == x["suspicious"] for x in TEST) / len(TEST) other = PAIR[1] if dropped == PAIR[0] else PAIR[0] print(f"{dropped} removed: test {acc:.4f}, {other} importance " f"{permute(w2, remaining, o2, s2, TEST, [other]):.4f}") ``` ``` one by one 0.1332 and 0.0018, total 0.1350, together 0.1437 mean_consumption removed: test 0.7778, per_person importance 0.0387 per_person removed: test 0.8016, mean_consumption importance 0.1441 ``` The importance of consumption per person is 0.0018 in the full model, and someone looking at this number concludes that the column is standing there for nothing. When mean consumption is removed from the model and the model is retrained, the importance of the same column rises to 0.0387, that is, twenty-one times. The column did not change; **the model changed**. Importance is not a property of a column but of a column **in a particular model**. Shuffling the two together gives 0.1437; the sum of the one-by-one shufflings is 0.1350. The difference, 0.0087, is the share the two carry **redundantly**: because the other fills the gap when each is shuffled alone, that share does not show up in any single measure. An importance ranking always loses a share between correlated columns. #### The Most Important Column in the Subgroups The overall ranking is the ranking of a set, and that set is made up of subgroups. ```python print(f"{'subgroup':<16}{'records':>8}{'most important column':>23}{'loss increase':>15}" f"{'second column':>18}") READ = [] for name, f in SUBGROUPS: g = [x for x in TEST if f(x)] d = {a: permute(W, FIELDS, O, S, g, [a]) for a in FIELDS} order = sorted(FIELDS, key=lambda a: -d[a]) if SIZES[name] >= MIN_SIZE: READ.append((name, order[0], d[order[0]])) print(f"{name:<16}{len(g):>8}{order[0]:>23}{d[order[0]]:>15.4f}{order[1]:>18}") highest, lowest = max(READ, key=lambda t: t[2]), min(READ, key=lambda t: t[2]) print(f"\nsubgroups read {len(READ)}/{len(SUBGROUPS)}; subgroups with a column other than " f"{BY_PERMUTE[0]} in first place " f"{sum(1 for t in READ if t[1] != BY_PERMUTE[0])}") print(f"highest {highest[0]} {highest[2]:.4f}, lowest {lowest[0]} {lowest[2]:.4f}, " f"gap {highest[2] - lowest[2]:.4f}") ``` ``` subgroup records most important column loss increase second column region north 68 mean_consumption 0.0719 zero_readings region south 58 zero_readings 0.1882 mean_consumption region east 36 mean_consumption 0.1753 satisfaction region west 32 mean_consumption 0.0162 zero_readings region central 58 mean_consumption 0.1879 zero_readings tariff 10 12 zero_readings 0.1288 area_west tariff 25 193 mean_consumption 0.0408 zero_readings tariff 40 47 mean_consumption 0.1033 volatility household 1-2 73 mean_consumption 0.1339 zero_readings household 3-4 143 mean_consumption 0.1215 zero_readings household 5+ 36 mean_consumption 0.1513 zero_readings readings few 85 mean_consumption 0.1213 zero_readings readings all 167 mean_consumption 0.1181 zero_readings subgroups read 12/13; subgroups with a column other than mean_consumption in first place 1 highest region south 0.1882, lowest region west 0.0162, gap 0.1720 ``` Thirteen subgroups were examined, and the 12 that pass the threshold were read. In eleven of the twelve subgroups read, the first rank is mean consumption; in the south region the first rank moves to the zero-reading indicator, and its loss increase of 0.1882 is above the overall 0.1332. The sentence "the model relies most on mean consumption," written for the whole, is wrong when asked of the subscribers of the south region. The gap is sharper still: the loss increase of the most important column is 0.1882 in the south region and 0.0162 in the west region — an eleven-fold difference between them. In the west region, shuffling no column raises the loss to any notable degree, because the subscribers there stand far from the decision boundary and their scores are insensitive to column movements. The counterfactual count of the previous lesson had also collapsed in the same region; two separate measures point at the same fact from two separate places. #### Summary - For the same model, the ranking read from the coefficients and the ranking measured by permuting disagree on ten of the twelve columns; the largest shift is five ranks and it is in volatility. - The coefficient measures what a column adds when the other columns are held constant, and permutation measures whether that column's information exists in other columns; the coefficient of volatility is 0.380 while its loss increase is 0.0012. - Importance is a property not of a column but of the column's position in a particular model: the importance of consumption per person rises from 0.0018 to 0.0387 when mean consumption is removed and the model is retrained. - Shuffling the two correlated columns together gives 0.1437 and the sum of the one-by-one shufflings gives 0.1350; the 0.0087 between them is the redundant share that shows up in no single measure. - Thirteen subgroups were examined and 12 were read: in eleven the first rank is mean consumption, in the south region it is the zero-reading indicator; the loss increase of the most important column ranges from 0.0162 to 0.1882, and the gap is 0.1720. #### Next Step Both rankings belong to the whole model. When the flag of suspicion arrives, the subscriber in the setup asks not what the model relies on in general but what **their own** decision relies on, and the global ranking does not answer that question: the change of first place in the south region showed exactly this. The next lesson moves on to methods that explain a single prediction. A small surrogate model that imitates the behavior of the original model around an example is built, the output is distributed among the features to compute contribution shares, and the two methods are asked about the same subscriber. What will be measured is the **faithfulness** of the explanation: in how many examples the surrogate model describes the decision of the original model correctly, and how many times the two methods give two different explanations for the same subscriber. ### Local Explanation Methods Source: https://academia.sh/en/courses/model-ethics-and-interpretation/local-explanation-methods Asking two methods that explain a single prediction about the same subscribers and measuring the faithfulness of the explanation: the local surrogate gets an average of 0.9417 of the neighborhood decisions right, but for 41 of the 252 subscribers the neighborhood holds a single label, so the explanation comes out empty. The contribution share points to a different column than the surrogate in 145 of the 211 filled explanations. Faithfulness is highest in the west region at 0.9664, where the empty-explanation rate is 0.4688; the lowest faithfulness, 0.9027, is in the top tariff tier, and the gap is 0.0637. Both rankings in the previous lesson belonged to the model as a whole. When the fictional subscriber receives the suspicious flag, the question is not what the model relies on in general but what **its own** decision relies on. The change in the first place of the global ranking in the south region had shown that the answer for the whole may not be true for a single subscriber. A local explanation answers this question with two separate structures. The first builds a small model that imitates the behavior of the original model **around** an example; the second **distributes** the output among the features. Both produce a column name; this lesson counts how often those two names are the same and whether the explanation really describes the original model. - **IN14.** The setup is that of the previous two lessons: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579, linear model 0.8016 on the test set. **The number of subgroups examined is 13**, the reading threshold is 25 records. - **IN15.** The local surrogate is a small model with a **single split**, fitted to 40 neighbors generated around the subscriber being explained. The neighbors are generated by shifting the numeric columns within a range of one standard deviation and by changing the two-valued column and the region with probability 0.2; their labels are not the true labels but **the decision of the original model**. - **IN16.** The faithfulness of an explanation is the share of the neighborhood decisions the surrogate gets right. If the whole neighborhood carries a single label, the surrogate is constant: its faithfulness is 1 and its explanation is **empty**. - **IN17.** The contribution share distribution starts from the base record, puts the subscriber's columns in one by one in a random order, records the change in the score at each step, and averages over 24 orders. The base record is the mean of the training set, and its region is the training mean of the five indicators. - **IN18.** The single-feature perturbation is the single-order form of the contribution share: only one column is pulled to its base value. When the two methods are compared, the **first column** each one gives is compared. #### Two Methods, the Same Subscriber The setup is the same as in the previous lessons. ```python # interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed; # the twelve-column feature set and the split stay fixed throughout the course. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def scale(train, fields): return ({a: statistics.fmean(x[a] for x in train) for a in fields}, {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields}) def train_model(train, fields, o, s, rate=0.3, epoch=300): # MODEL: logistic regression X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train] Y = [x["suspicious"] for x in train] w = [0.0] * len(X[0]) for _ in range(epoch): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= rate * g[j] / len(X) return w def score(w, fields, o, s, x): return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a] for j, a in enumerate(fields))) def decision(w, fields, o, s, x): return int(score(w, fields, o, s, x) > 0.5) O, S = scale(TRAIN, FIELDS) W = train_model(TRAIN, FIELDS, O, S) ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, baseline " f"{BASELINE:.4f}, test accuracy of the linear model {ACCURACY:.4f}") def tariff_tier(x): return ("tariff 10" if x["mean_consumption"] <= 10 else "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40") def household_group(x): return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4 else "household 5+") def reading_group(x): return "readings few" if x["period"] <= 2 else "readings all" def equals(f, value): return lambda x: f(x) == value SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION] + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")] + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")] + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")]) MIN_SIZE = 25 SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS} print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, below the threshold " f"{[a for a in SIZES if SIZES[a] < MIN_SIZE]}") ``` ``` train 756, validation 252, test 252, baseline 0.7579, test accuracy of the linear model 0.8016 subgroup count 13, threshold 25 records, below the threshold ['tariff 10'] ``` The two methods are set up one after the other, and each produces one column name for every subscriber. ```python LOCAL = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "region_name"] BASE_FIELDS = {a: statistics.fmean(x[a] for x in TRAIN) for a in ("mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings")} BASE_FIELDS["region_name"] = None # in the base record the region is not a single region but the mean B_MEAN = {"area_" + b[0]: statistics.fmean(x["area_" + b[0]] for x in TRAIN) for b in REGION} def recompute(d): # recomputes the derived columns d["per_person"] = round(d["mean_consumption"] / max(1, d["household"]), 3) if d["region_name"] is None: d.update(B_MEAN) else: for b in REGION: d["area_" + b[0]] = int(d["region_name"] == b[0]) return d BASE_RECORD = recompute(dict(TEST[0], **BASE_FIELDS)) def neighborhood(x, count=40, seed=SEED + 600000): r, c = rng(seed + int(x["mean_consumption"] * 100) + x["household"]), [] for _ in range(count): d = dict(x) for a in ("mean_consumption", "volatility", "satisfaction", "period"): d[a] = x[a] + (2 * r() - 1) * S[a] d["household"] = max(1, round(x["household"] + (2 * r() - 1) * S["household"])) if r() < 0.2: d["zero_readings"] = 1 - x["zero_readings"] if r() < 0.2: d["region_name"] = REGION[int(r() * 5)][0] c.append(recompute(d)) return c SORT_KEY = {"west": 0, "east": 1, "south": 2, "north": 3, "central": 4} # keeps the source's order of the region names, because the first of two equally good splits wins def surrogate(x): # MODEL: single-split decision tree, local surrogate c = neighborhood(x) y = [decision(W, FIELDS, O, S, k) for k in c] if len(set(y)) == 1: return None, 1.0 # the whole neighborhood has a single label: the explanation is empty best = (0.0, None) for a in LOCAL: d = sorted({k[a] for k in c}, key=lambda v: SORT_KEY.get(v, v)) candidates = d[1:] if a in ("zero_readings", "region_name") else \ [d[int(i * len(d) / 8)] for i in range(1, 8)] for v in candidates: left = {i for i in range(len(c)) if (c[i][a] != v if a == "region_name" else c[i][a] < v)} if not left or len(left) == len(c): continue pred_left = sum(y[i] for i in left) * 2 > len(left) pred_right = sum(y[i] for i in range(len(c)) if i not in left) * 2 > len(c) - len(left) hit = sum((pred_left if i in left else pred_right) == y[i] for i in range(len(c))) / len(c) if hit > best[0]: best = (hit, a) return best[1], best[0] def contribution(x, epoch=24, seed=SEED + 700000): r = rng(seed + int(x["mean_consumption"] * 100) + x["household"]) share = {a: 0.0 for a in LOCAL} for _ in range(epoch): order = list(LOCAL) for i in range(len(order) - 1, 0, -1): j = int(r() * (i + 1)) order[i], order[j] = order[j], order[i] d, previous = dict(BASE_RECORD), score(W, FIELDS, O, S, BASE_RECORD) for a in order: d[a] = x[a] new = score(W, FIELDS, O, S, recompute(d)) share[a] += new - previous previous = new return {a: share[a] / epoch for a in LOCAL} def single_perturbation(x): # single perturbation: one column is pulled to the base value p0 = score(W, FIELDS, O, S, x) return {a: p0 - score(W, FIELDS, O, S, recompute(dict(x, **{a: BASE_FIELDS[a]}))) for a in LOCAL} FAITHFULNESS, SUR_COL, CONTRIB_COL, SINGLE_COL = [], [], [], [] for x in TEST: a, u = surrogate(x) p, t = contribution(x), single_perturbation(x) FAITHFULNESS.append(u) SUR_COL.append(a) CONTRIB_COL.append(max(LOCAL, key=lambda k: abs(p[k]))) SINGLE_COL.append(max(LOCAL, key=lambda k: abs(t[k]))) FILLED = [i for i in range(len(TEST)) if SUR_COL[i]] DIFFER = [i for i in FILLED if SUR_COL[i] != CONTRIB_COL[i]] print(f"mean faithfulness {sum(FAITHFULNESS) / len(FAITHFULNESS):.4f}; subscribers whose " f"neighborhood holds a single label, that is, who get an empty explanation {len(TEST) - len(FILLED)}") print(f"filled explanations {len(FILLED)}, mean faithfulness among them " f"{sum(FAITHFULNESS[i] for i in FILLED) / len(FILLED):.4f}") print(f"subscribers for whom the surrogate and the contribution share do not give the same column " f"{len(DIFFER)}/{len(FILLED)} ({len(DIFFER) / len(FILLED):.4f})") print(f"subscribers for whom the contribution share and the single perturbation disagree " f"{sum(1 for i in range(len(TEST)) if CONTRIB_COL[i] != SINGLE_COL[i])}/{len(TEST)}") i = min(DIFFER, key=lambda i: abs(score(W, FIELDS, O, S, TEST[i]) - 0.5)) x, p = TEST[i], contribution(TEST[i]) print(f"\nexample subscriber: region {x['region_name']}, mean consumption {x['mean_consumption']}, " f"zero readings {x['zero_readings']}, score {score(W, FIELDS, O, S, x):.4f}") print(f" surrogate split column {SUR_COL[i]}, faithfulness {FAITHFULNESS[i]:.4f}") print(" contribution shares " + ", ".join(f"{a} {p[a]:+.4f}" for a in sorted(LOCAL, key=lambda a: -abs(p[a]))[:3])) ``` ``` mean faithfulness 0.9417; subscribers whose neighborhood holds a single label, that is, who get an empty explanation 41 filled explanations 211, mean faithfulness among them 0.9303 subscribers for whom the surrogate and the contribution share do not give the same column 145/211 (0.6872) subscribers for whom the contribution share and the single perturbation disagree 27/252 example subscriber: region east, mean consumption 24.67, zero readings 0, score 0.4885 surrogate split column mean_consumption, faithfulness 0.7750 contribution shares region_name +0.1631, mean_consumption +0.1335, volatility +0.0428 ``` The example subscriber gets two explanations, and they say different things. The surrogate says that what separates the decision around that subscriber is mean consumption, and it gets 0.7750 of the neighborhood decisions right. The contribution share distribution says that the largest share that carries the subscriber from the base record to this score belongs to the region: being in the east region added 0.1631 to the score. Both are correct, and they answer two separate questions: the first is "which column leads out of here", the second is "which column brought it here". In 145 of the 211 filled explanations they do not give the same column. The single-feature perturbation, the single-order form of the contribution share, disagrees with it for 27 subscribers: what a column contributes depends on which columns were put in before it. #### Faithfulness Is Not Read Alone The mean faithfulness looks high at 0.9417. Inside this number are 41 subscribers whose neighborhood carries a single label: the surrogate is constant, its faithfulness is 1 by definition, and it says nothing at all. When the empty explanations are left out, faithfulness falls to 0.9303. ```python print(f"{'subgroup':<16}{'records':>8}{'faithfulness':>14}{'empty explanation':>19}{'disagreement':>14}") READ = [] for name, f in SUBGROUPS: g = [i for i, x in enumerate(TEST) if f(x)] faith = sum(FAITHFULNESS[i] for i in g) / len(g) empty = sum(1 for i in g if SUR_COL[i] is None) / len(g) d = [i for i in g if SUR_COL[i]] dis = sum(1 for i in d if SUR_COL[i] != CONTRIB_COL[i]) / len(d) if SIZES[name] >= MIN_SIZE: READ.append((name, faith, dis)) print(f"{name:<16}{len(g):>8}{faith:>14.4f}{empty:>19.4f}{dis:>14.4f}") highest, lowest = max(READ, key=lambda t: t[1]), min(READ, key=lambda t: t[1]) top_dis = max(READ, key=lambda t: t[2]) print(f"\nsubgroups read {len(READ)}/{len(SUBGROUPS)}") print(f"faithfulness highest {highest[0]} {highest[1]:.4f}, lowest {lowest[0]} {lowest[1]:.4f}, " f"gap {highest[1] - lowest[1]:.4f}") print(f"disagreement highest {top_dis[0]} {top_dis[2]:.4f}, lowest " f"{min(READ, key=lambda t: t[2])[0]} {min(t[2] for t in READ):.4f}") ``` ``` subgroup records faithfulness empty explanation disagreement region north 68 0.9401 0.0441 0.7231 region south 58 0.9586 0.2414 0.7273 region east 36 0.9028 0.1389 0.7097 region west 32 0.9664 0.4688 0.4118 region central 58 0.9371 0.0690 0.6852 tariff 10 12 0.9688 0.5833 1.0000 tariff 25 193 0.9495 0.1503 0.7012 tariff 40 47 0.9027 0.1064 0.5952 household 1-2 73 0.9538 0.2603 0.6852 household 3-4 143 0.9320 0.1329 0.7097 household 5+ 36 0.9556 0.0833 0.6061 readings few 85 0.9456 0.2353 0.8154 readings all 167 0.9397 0.1257 0.6301 subgroups read 12/13 faithfulness highest region west 0.9664, lowest tariff 40 0.9027, gap 0.0637 disagreement highest readings few 0.8154, lowest region west 0.4118 ``` Thirteen subgroups were examined, and 12 were read. The subgroup with the highest faithfulness is the west region, 0.9664; on the same row the empty-explanation rate is 0.4688, which means fifteen of the thirty-two subscribers get an explanation that says nothing. Someone who reports faithfulness alone writes the west down as the best-explained group; when the two columns are put side by side, that sentence turns around. **Faithfulness assumes that the explanation says something; when it says nothing, it still comes out high.** The lowest faithfulness, 0.9027, is in the top tariff tier, and the gap is 0.0637. Because subscribers with high consumption cluster near the decision boundary, their neighborhoods carry two labels and a single split is not enough; the same group is the one where the explanations are fullest. The disagreement of the two methods varies between 0.4118 and 0.8154 across the subgroups. #### Summary - The local surrogate gets an average of 0.9417 of the neighborhood decisions right; when the empty explanations are left out, it falls to 0.9303. - For 41 of the 252 subscribers the neighborhood carries a single label: the surrogate is constant, its faithfulness is 1, and its explanation is empty. This is why faithfulness is not read alone. - The surrogate and the contribution share distribution point to different columns in 145 of the 211 filled explanations; the two answer the questions "which column leads out of here" and "which column brought it here". - Thirteen subgroups were examined and 12 were read: faithfulness is between 0.9027 and 0.9664, the gap is 0.0637; in the group with the highest faithfulness, the empty-explanation rate is 0.4688. #### Next Step In this lesson the explanations belonged to individual subscribers. There is another way to see what a column does to the model **as a whole**: the value of the column is swept from one end to the other while the other columns are left at the set's own values, and one looks at how the mean score changes. This is partial dependence, and it is usually drawn as a curve. Here it is not drawn, the numbers are printed; when the numbers are printed, what the average hides becomes visible. The next lesson counts, in how many columns it happens, that a partial dependence that looks flat for the whole can slope in opposite directions in two subgroups. ### Partial Dependence and Interaction Source: https://academia.sh/en/courses/model-ethics-and-interpretation/partial-dependence-and-interaction Printing what a column does to the model as a whole as numbers and counting what the average hides: in the decision tree, three of the five columns look flat in the whole, and in two of them the subgroups slope in opposite directions. The partial dependence of household size is flat in the whole at -0.0069, +0.1301 in the third tariff tier, -0.1891 in the east region, and the gap is 0.3192. In satisfaction the gap is 0.2189. The number of readings has a slope of exactly 0.0000 both in the whole and in all twelve subgroups: not every number that looks flat hides an interaction. The previous two lessons tied the explanation either to the model as a whole or to a single subscriber. The question in between was not asked: while the value of a column is swept from one end to the other, how does the output of the model behave **on average**. This is partial dependence, and it summarizes the effect of a single column on the model as a curve. Here the curve is not drawn; the numbers at the grid points are printed, because the question of this lesson is precisely about what gets lost inside that average. Partial dependence is an average, and like every average it can cancel two movements in opposite directions. If a column pushes the score up in one subgroup and down in another, the curve for the whole comes out **flat**, and it looks as if the column plays no role in the model. This is the measure of this lesson: in how many of the columns that look flat in the whole do the subgroups slope in opposite directions. - **IN19.** The setup is that of the previous lessons: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579 on the test set. The subgroups and the reading threshold of 25 records continue; **the number of subgroups examined is 13**. - **IN20.** This lesson changes the model being explained. The linear model does not multiply the columns by each other, so the partial dependence of a column carries the same sign in every subgroup and the question of this lesson cannot be asked there. The model being explained is the decision tree family of the Supervised Learning course; here an unpruned tree of depth 6 is used, and it gives 0.7897 on the test set. The number the family reaches with pruning in the Supervised Learning course is 0.7857. - **IN21.** Partial dependence is computed as follows: one column is pulled to the value on the grid, the other columns are left at the set's own values, and the scores are averaged. The grid is at most five points of the distinct values in the training set; for a column with fewer than five distinct values, the number of points is that many. - **IN22.** The derived consumption per person is recomputed together with its source at each grid point. - **IN23.** The slope is the partial dependence at the last point of the grid minus the one at the first point. A column whose absolute slope in the whole is below 0.05 counts as **flat**; a column whose absolute slope in a subgroup exceeds 0.02 counts as **sloped**. Both thresholds are on the score scale, and the score varies between 0 and 1. - **IN24.** No chart is drawn; partial dependence is printed as numbers at the grid points. #### Partial Dependence as Numbers The setup is the same as in the previous lessons. ```python # interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed; # the twelve-column feature set and the split stay fixed throughout the course. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def tariff_tier(x): return ("tariff 10" if x["mean_consumption"] <= 10 else "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40") def household_group(x): return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4 else "household 5+") def reading_group(x): return "readings few" if x["period"] <= 2 else "readings all" def equals(f, value): return lambda x: f(x) == value SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION] + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")] + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")] + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")]) MIN_SIZE = 25 SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS} print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}; subgroup " f"count {len(SUBGROUPS)}, below the threshold {[a for a in SIZES if SIZES[a] < MIN_SIZE]}") ``` ``` train 756, validation 252, test 252; subgroup count 13, below the threshold ['tariff 10'] ``` The tree is built in its form from the Supervised Learning course, and the partial dependence is printed at the grid points for five columns. ```python def positive_rate(s): return sum(x["suspicious"] for x in s) / len(s) if s else 0.0 def impurity(s): p = positive_rate(s) return 2 * p * (1 - p) def best_split(s, min_leaf=2): best = None for a in FIELDS: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_leaf: continue k = impurity(s) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) return best def tree(s, depth, min_leaf=2): # MODEL: decision tree, the family of the Supervised Learning course p = positive_rate(s) best = (best_split(s, min_leaf) if depth and len(s) >= 2 * min_leaf and 0.0 < p < 1.0 else None) if best is None or best[0] <= 1e-9: return {"rate": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, min_leaf), "right": tree(best[4], depth - 1, min_leaf)} def tree_score(d, x): while "rate" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["rate"] TREE = tree(TRAIN, 6) TREE_ACCURACY = sum((tree_score(TREE, x) > 0.5) == x["suspicious"] for x in TEST) / len(TEST) print(f"baseline {BASELINE:.4f}, test accuracy of the decision tree {TREE_ACCURACY:.4f}") GRID_COLUMNS = ["mean_consumption", "volatility", "household", "satisfaction", "period"] def grid(a, points=5): d = sorted({x[a] for x in TRAIN}) k = min(points, len(d)) return [d[int(i * (len(d) - 1) / (k - 1))] for i in range(k)] def partial(rows, a, gr): # partial dependence: as numbers, not drawn p = [] for v in gr: t = 0.0 for x in rows: d = dict(x) d[a] = v d["per_person"] = round(d["mean_consumption"] / max(1, d["household"]), 3) t += tree_score(TREE, d) p.append(t / len(rows)) return p FLAT, SLOPE = 0.05, {} GRIDS = {a: grid(a) for a in GRID_COLUMNS} for a in GRID_COLUMNS: p = partial(TEST, a, GRIDS[a]) SLOPE[a] = p[-1] - p[0] print(f"{a:<17}" + " ".join(f"{v:g}:{q:.4f}" for v, q in zip(GRIDS[a], p)) + f" slope {SLOPE[a]:+.4f}") FLAT_COLUMNS = [a for a in GRID_COLUMNS if abs(SLOPE[a]) < FLAT] print(f"\ncolumns that look flat in the whole (with the {FLAT} threshold) {FLAT_COLUMNS}") ``` ``` baseline 0.7579, test accuracy of the decision tree 0.7897 mean_consumption 0:0.0848 15.23:0.1380 19.51:0.2234 23.81:0.2250 42.9:0.3838 slope +0.2990 volatility 0:0.1374 5.52:0.1374 9.89:0.1945 15.64:0.3158 40.57:0.7731 slope +0.6357 household 1:0.2546 2:0.2510 3:0.2274 4:0.2165 6:0.2477 slope -0.0069 satisfaction 1:0.2926 2:0.2588 3:0.2315 4:0.2354 5:0.2460 slope -0.0466 period 1:0.2491 2:0.2491 3:0.2491 slope +0.0000 columns that look flat in the whole (with the 0.05 threshold) ['household', 'satisfaction', 'period'] ``` Mean consumption and volatility are two rows that are easy to read: along the grid the score rises in a single direction, with slopes of +0.2990 and +0.6357. The remaining three columns are flat. The slope of household size is -0.0069, that is, seven thousandths of the score; that of satisfaction is -0.0466; that of the number of readings is exactly zero. Someone who looks at these three rows writes this sentence: the model does not base its decision on household size, satisfaction, or the number of readings. The sentence is written with the same justification for all three columns, but it does not describe the same thing in all three, and which one it describes cannot be read from the whole. The computation itself carries an assumption, and the assumption must be written down here explicitly. While a column is pulled to a grid point, the other columns are left where they are; that is, the columns are assumed to be movable independently of each other. The mean consumption of a subscriber whose household size is pulled to 6 does not change, whereas in the fiction the two move together. The permutation importance of the previous lesson rested on the same assumption, and there two correlated columns had split the importance. Partial dependence also computes scores on combinations never seen in the set; the numbers at the ends of the grid are therefore less reliable than those in the middle. #### The Interaction the Average Hides The same computation is done separately in the twelve readable subgroups. ```python THRESHOLD = 0.02 SLOPES = {a: {} for a in FLAT_COLUMNS} for name, f in SUBGROUPS: if SIZES[name] < MIN_SIZE: continue g = [x for x in TEST if f(x)] for a in FLAT_COLUMNS: p = partial(g, a, GRIDS[a]) SLOPES[a][name] = p[-1] - p[0] SHORT = [(name.replace("region ", "").replace("tariff ", "t").replace("household ", "h") .replace("readings ", "rd "), name) for name in SLOPES[FLAT_COLUMNS[0]]] print("subgroup " + "".join(f"{a:>13}" for a in FLAT_COLUMNS)) print("whole " + "".join(f"{SLOPE[a]:>+13.4f}" for a in FLAT_COLUMNS)) for short, name in SHORT: print(f"{short:<9}" + "".join(f"{SLOPES[a][name]:>+13.4f}" for a in FLAT_COLUMNS)) OPPOSITE = [a for a in FLAT_COLUMNS if max(SLOPES[a].values()) > THRESHOLD and min(SLOPES[a].values()) < -THRESHOLD] print(f"\nsubgroups read {len(SLOPES[FLAT_COLUMNS[0]])}/{len(SUBGROUPS)}, slope threshold {THRESHOLD}") print(f"columns that look flat in the whole {len(FLAT_COLUMNS)}, of which sloped in " f"opposite directions in the subgroups {len(OPPOSITE)}") for a in OPPOSITE: hi = max(SLOPES[a], key=lambda k: SLOPES[a][k]) lo = min(SLOPES[a], key=lambda k: SLOPES[a][k]) print(f" {a}: highest {hi} {SLOPES[a][hi]:+.4f}, lowest {lo} {SLOPES[a][lo]:+.4f}, " f"gap {SLOPES[a][hi] - SLOPES[a][lo]:.4f}") ``` ``` subgroup household satisfaction period whole -0.0069 -0.0466 +0.0000 north +0.0916 -0.1146 +0.0000 south -0.0477 -0.0108 +0.0000 east -0.1891 +0.0278 +0.0000 west -0.0442 -0.0313 +0.0000 central +0.0522 -0.0573 +0.0000 t25 -0.0264 -0.0143 +0.0000 t40 +0.1301 -0.1911 +0.0000 h1-2 -0.0354 -0.0665 +0.0000 h3-4 +0.0049 -0.0533 +0.0000 h5+ +0.0040 +0.0205 +0.0000 rd few +0.0155 -0.0359 +0.0000 rd all -0.0183 -0.0520 +0.0000 subgroups read 12/13, slope threshold 0.02 columns that look flat in the whole 3, of which sloped in opposite directions in the subgroups 2 household: highest tariff 40 +0.1301, lowest region east -0.1891, gap 0.3192 satisfaction: highest region east +0.0278, lowest tariff 40 -0.1911, gap 0.2189 ``` Household size is flat in the whole at -0.0069. In the top tariff tier it is +0.1301, in the east region -0.1891; the gap is 0.3192, that is, forty-six times the slope in the whole. Among subscribers with high consumption, suspicion rises as the household grows; in the east region it falls as the household grows. The two movements cancel each other in the whole, and a number of seven thousandths is left behind. **The flatness read in the whole is not an absence but a sum.** In satisfaction the same thing happens on a smaller scale: -0.0466 in the whole, -0.1911 in the top tariff tier, +0.0278 in the east region, a gap of 0.2189. The most extreme values of both columns stand in **the same two subgroups** and in opposite order; this means that the tree uses these two columns not independently but together with the consumption level and the region. This is exactly what interaction is. The number of readings is a third kind of flatness, and it is the control of this lesson. The slope is exactly 0.0000 both in the whole and in all twelve subgroups: the tree has never placed a split on this column. Under every number that looks flat there is not an interaction; sometimes the column has not been used. The way to tell the two cases apart is not to look at the whole but to look at the subgroups, and when one does not look, the two read as the same row. Whether the decision changes is read here too. Someone who looks at the whole proposes removing all three columns from the model. Someone who looks at the subgroups proposes this only for the number of readings; the other two are used, and it is only **for whom, in which direction** they are used that does not show in the whole. #### Summary - Partial dependence sweeps a column along the grid and averages the scores; here the curve was not drawn, and the numbers at the grid points were printed. - The computation assumes that the columns can be moved independently and produces combinations never seen in the set; the numbers at the ends of the grid are less reliable than those in the middle. - Two of the five columns clearly slope in the whole (+0.2990 and +0.6357), and three look flat. - In two of the three columns that look flat in the whole, the subgroups slope in opposite directions: the gap is 0.3192 in household size and 0.2189 in satisfaction. - The slope of household size is -0.0069 in the whole, +0.1301 in the third tariff tier, and -0.1891 in the east region; the flatness in the whole is not an absence but the sum of two movements in opposite directions. - The number of readings has a slope of exactly 0.0000 both in the whole and in all twelve subgroups: not every number that looks flat hides an interaction; sometimes the column has really not been used. #### Next Step In four lessons the explanation was always produced **from outside the model**: the decision of a model that had already been built was read, permuted, imitated, and averaged afterward. Each time the explanation itself was an approximation, and its faithfulness had to be measured separately. There is another way: building the model readable from the start. A model that gives the decision by applying a few rules in order does not produce the explanation separately; the model itself is the explanation, and its faithfulness is complete by definition. The next lesson measures the cost of this way: with how many rules the number of the best family of the Supervised Learning course is approached, and in which subgroup it is not approached at all. ### Inherently Interpretable Models Source: https://academia.sh/en/courses/model-ethics-and-interpretation/inherently-interpretable-models Measuring the cost of building the model readable from the start instead of producing the explanation afterward: a sequential list of two rules and four conditions gives 0.7897 on the test set, leaving a gap of 0.0159, that is, four subscribers, to the 0.8056 of the best family with thirty-three columns. Extending the list to three and four rules does not get closer; it drops back to 0.7857. Thirteen subgroups were examined, and in six of the twelve read, the best family cannot be approached within 0.02; the farthest is the central region, 0.6552 instead of 0.7414, a gap of 0.0862. In three subgroups the rule list beats the best family. In four lessons the explanation was always produced from outside the model: the decision of a model that had already been built was read, permuted, imitated, and averaged afterward. Each time the explanation was an approximation, and its faithfulness had to be measured separately. There is another way: building the model readable from the start. A model that gives the decision by applying a few rules in order does not produce the explanation separately — the model itself is the explanation, and its faithfulness is complete by definition. The question of this lesson is the cost of that way. The Linear Models topic of the Supervised Learning course had measured the cost once: 0.0040 between the model with twelve coefficients and the best family with thirty-three columns, that is, a single subscriber. That number is not repeated here, it is **taken as input** — because a rule list is far more readable than twelve coefficients, and how the cost grows with readability is the real question. - **IN25.** The setup is that of the previous lessons: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579 on the test set. The subgroups and the reading threshold of 25 records continue; **the number of subgroups examined is 13**. - **IN26.** The comparison lines come from the Supervised Learning course and are recomputed with the same seed: the linear model with twelve columns 0.8016, the best family (thirty-three columns, with a quadratic penalty) 0.8056. The number of the decision tree family from the same course is 0.7857. - **IN27.** The rule list is a sequential model: the first rule that matches gives the decision, and if none matches, the last label is given. Each rule carries at most two conditions, and the two conditions cannot be from the same column. - **IN28.** The list is built greedily: at each step, the rule that gains the most correct answers relative to the default label of the remaining records is added, and the records it covers are removed. The minimum coverage is 12 records, and the condition thresholds are taken from the deciles of the training set. - **IN29.** "Approaching the best family" means that the test accuracy gap does not exceed 0.02; on this test set, 0.02 is five subscribers. #### Three Comparison Lines The setup is that of the previous lessons; on top of it, the best family of the Supervised Learning course is rebuilt with the same seed. ```python # interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed; # the twelve-column feature set and the split stay fixed throughout the course. import itertools import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def scale(train, fields): return ({a: statistics.fmean(x[a] for x in train) for a in fields}, {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields}) def train_model(train, fields, o, s, rate=0.3, epoch=300, penalty=0.0): # MODEL: logistic regression X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train] Y = [x["suspicious"] for x in train] w = [0.0] * len(X[0]) for _ in range(epoch): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= rate * (g[j] / len(X) + (penalty * w[j] if j else 0.0)) return w def score(w, fields, o, s, x): return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a] for j, a in enumerate(fields))) def decision(w, fields, o, s, x): return int(score(w, fields, o, s, x) > 0.5) O, S = scale(TRAIN, FIELDS) W = train_model(TRAIN, FIELDS, O, S) ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") def tariff_tier(x): return ("tariff 10" if x["mean_consumption"] <= 10 else "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40") def household_group(x): return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4 else "household 5+") def reading_group(x): return "readings few" if x["period"] <= 2 else "readings all" def equals(f, value): return lambda x: f(x) == value SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION] + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")] + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")] + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")]) MIN_SIZE = 25 SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS} print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, below the threshold " f"{[a for a in SIZES if SIZES[a] < MIN_SIZE]}") PRODUCT = ["mean_consumption", "volatility", "household", "satisfaction", "period", "per_person"] def expand(x): d = dict(x) for a, b in itertools.combinations(PRODUCT, 2): d[a + "*" + b] = x[a] * x[b] for a in PRODUCT: d[a + "^2"] = x[a] * x[a] return d EXPANDED = (FIELDS + [a + "*" + b for a, b in itertools.combinations(PRODUCT, 2)] + [a + "^2" for a in PRODUCT]) EX_TRAIN, EX_TEST = [expand(x) for x in TRAIN], [expand(x) for x in TEST] OE, SE = scale(EX_TRAIN, EXPANDED) WE = train_model(EX_TRAIN, EXPANDED, OE, SE, penalty=0.05) # the best family of the Supervised Learning course BEST_FAMILY = sum(decision(WE, EXPANDED, OE, SE, x) == x["suspicious"] for x in EX_TEST) / len(EX_TEST) LINEAR_ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST) print(f"baseline {BASELINE:.4f}, twelve-column linear model {LINEAR_ACCURACY:.4f}, " f"best family ({len(EXPANDED)} columns) {BEST_FAMILY:.4f}") ``` ``` train 756, validation 252, test 252 subgroup count 13, threshold 25 records, below the threshold ['tariff 10'] baseline 0.7579, twelve-column linear model 0.8016, best family (33 columns) 0.8056 ``` #### As the Rule List Grows The list is built greedily, and the length is swept from one to six. ```python RULE_COLUMNS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "region_name"] SORT_KEY = {"west": 0, "east": 1, "south": 2, "north": 3, "central": 4} # keeps the source's order of the region names, because the first of two equally good splits wins def matches(k, x): a, op, e = k return x[a] == e if op == "=" else (x[a] < e if op == "<" else x[a] >= e) def conditions(subset): for a in RULE_COLUMNS: d = sorted({x[a] for x in subset}, key=lambda v: SORT_KEY.get(v, v)) if a == "region_name": for v in d: yield (a, "=", v) else: for v in (d if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): yield (a, "<", v) yield (a, ">=", v) def gain(g, v0, min_cover): # number of correct answers gained relative to the default if len(g) < min_cover: return None pos = sum(x["suspicious"] for x in g) label = int(pos * 2 > len(g)) return ((pos if label else len(g) - pos) - (pos if v0 else len(g) - pos), label) def rule_list(train, length, min_cover=12, beam=8): # MODEL: sequential rule list remaining, rules = list(train), [] for _ in range(length): v0 = int(sum(x["suspicious"] for x in remaining) * 2 > len(remaining)) candidates = [] for k1 in conditions(remaining): g1 = [x for x in remaining if matches(k1, x)] z = gain(g1, v0, min_cover) if z: p = sum(x["suspicious"] for x in g1) / len(g1) candidates.append((p if v0 == 0 else 1 - p, z[0], k1, g1, z[1])) if not candidates: break candidates.sort(key=lambda t: -t[0]) best = max(((t[1], [t[2]], t[4]) for t in candidates[:beam]), key=lambda t: t[0]) for _, _, k1, g1, _ in candidates[:beam]: for k2 in conditions(g1): if k2[0] == k1[0]: continue z = gain([x for x in g1 if matches(k2, x)], v0, min_cover) if z and z[0] > best[0]: best = (z[0], [k1, k2], z[1]) if best[0] <= 0: break rules.append((best[1], best[2])) remaining = [x for x in remaining if not all(matches(k, x) for k in best[1])] if not remaining: break p = sum(x["suspicious"] for x in remaining) return rules, (int(p * 2 > len(remaining)) if remaining else 0) def rule_predict(rules, fallback, x): for condition, label in rules: if all(matches(k, x) for k in condition): return label return fallback def describe(condition): return " and ".join(f"{a} {o} {e}" for a, o, e in condition) CLOSE = 0.02 # five subscribers on the test set print(f"{'rules':>6}{'conditions':>11}{'test':>9}{'gap to the best family':>24}") SWEEP = [] for n in range(1, 7): rules, fallback = rule_list(TRAIN, n) d = sum(rule_predict(rules, fallback, x) == x["suspicious"] for x in TEST) / len(TEST) k = sum(len(c) for c, e in rules) SWEEP.append((d, -len(rules), rules, fallback, k)) print(f"{len(rules):>6}{k:>11}{d:>9.4f}{BEST_FAMILY - d:>+24.4f}") CHOICE = max(SWEEP, key=lambda t: (t[0], t[1])) print(f"\nlist closest to the best family {-CHOICE[1]} rules, {CHOICE[4]} conditions, " f"test {CHOICE[0]:.4f}, gap {BEST_FAMILY - CHOICE[0]:+.4f} " f"({round((BEST_FAMILY - CHOICE[0]) * len(TEST))} subscribers)") for condition, label in CHOICE[2]: print(f" if {describe(condition)} then {label}") print(f" otherwise {CHOICE[3]}") ``` ``` rules conditions test gap to the best family 1 2 0.7857 +0.0198 2 4 0.7897 +0.0159 3 6 0.7857 +0.0198 4 8 0.7857 +0.0198 4 8 0.7857 +0.0198 4 8 0.7857 +0.0198 list closest to the best family 2 rules, 4 conditions, test 0.7897, gap +0.0159 (4 subscribers) if region_name = east and volatility >= 11.86 then 1 if zero_readings >= 1 and volatility >= 20.06 then 1 otherwise 0 ``` A single rule with two conditions gives 0.7857; this is the number of the decision tree family pruned in the Supervised Learning course. The second rule raises the number to 0.7897, and a gap of 0.0159, that is, four subscribers, remains to the best family. The third and fourth rules do not get closer, they drop back to 0.7857; at the fifth step the greedy procedure finds no gain and stops. **There is no one-directional relationship between length and accuracy.** The size of the cost is where the real reading lies. The Supervised Learning course had measured the gap for twelve coefficients as 0.0040, that is, a single subscriber; when it comes down to four conditions, the same gap rises to four subscribers. Readability is a continuous axis, and its cost grows along that axis; which point to choose is not a measurement but a decision, yet the decision can be made only once the numbers at both ends have been written down. The list itself is read in three lines, and each line is a sentence. The faithfulness of the explanation of this model is 1 by definition, because the explanation is the model itself; the 41 empty explanations of the third lesson do not occur here. #### In Which Subgroup the Best Family Cannot Be Approached The 0.0159 measured in the whole is not read alone. The same two models are put side by side in thirteen subgroups. ```python LIST, FALLBACK = CHOICE[2], CHOICE[3] print(f"{'subgroup':<16}{'records':>8}{'rule list':>15}{'best family':>13}" f"{'gap':>9}") FAR, READ = [], 0 for name, f in SUBGROUPS: g = [i for i, x in enumerate(TEST) if f(x)] rl = sum(rule_predict(LIST, FALLBACK, TEST[i]) == TEST[i]["suspicious"] for i in g) / len(g) bf = sum(decision(WE, EXPANDED, OE, SE, EX_TEST[i]) == TEST[i]["suspicious"] for i in g) / len(g) if SIZES[name] >= MIN_SIZE: READ += 1 if bf - rl > CLOSE: FAR.append((name, rl, bf)) print(f"{name:<16}{len(g):>8}{rl:>15.4f}{bf:>13.4f}{bf - rl:>+9.4f}") FAR.sort(key=lambda t: t[1] - t[2]) print(f"\nsubgroups read {READ}/{len(SUBGROUPS)}; subgroups that cannot be brought within {CLOSE} of " f"the best family {len(FAR)}, the farthest {FAR[0][0]} " f"({FAR[0][1]:.4f} instead of {FAR[0][2]:.4f}, gap {FAR[0][2] - FAR[0][1]:+.4f})") ``` ``` subgroup records rule list best family gap region north 68 0.8088 0.7794 -0.0294 region south 58 0.8621 0.8448 -0.0172 region east 36 0.6667 0.7222 +0.0556 region west 32 1.0000 1.0000 +0.0000 region central 58 0.6552 0.7414 +0.0862 tariff 10 12 0.6667 0.6667 +0.0000 tariff 25 193 0.8290 0.8342 +0.0052 tariff 40 47 0.6596 0.7234 +0.0638 household 1-2 73 0.7945 0.8356 +0.0411 household 3-4 143 0.8042 0.7972 -0.0070 household 5+ 36 0.7222 0.7778 +0.0556 readings few 85 0.7529 0.7765 +0.0235 readings all 167 0.8084 0.8204 +0.0120 subgroups read 12/13; subgroups that cannot be brought within 0.02 of the best family 6, the farthest region central (0.6552 instead of 0.7414, gap +0.0862) ``` The gap of four subscribers in the whole is spread across the subgroups. Thirteen subgroups were examined, the 12 that passed the threshold were read, and in six of them the best family cannot be approached within 0.02. The farthest is the central region: 0.6552 against 0.7414, a gap of 0.0862 — more than five times the gap in the whole. In the top tariff tier the gap is 0.0638, in the east region and among large households 0.0556. The reason can be read from the list itself. One of the two rules depends on the east region and the other on zero readings; the suspicious subscribers of the central region enter neither, and all of them receive the default label of the last line. **Brevity is paid for by the whole, and the bill falls on some subgroups.** The opposite direction is in the same table too. In three subgroups the rule list beats the best family: by 0.0294 in the north, 0.0172 in the south, and 0.0070 among middle-sized households. The single number for the whole does not say which model is better for whom. #### Summary - The sequential list of two rules and four conditions gives 0.7897 on the test set; a gap of 0.0159, that is, four subscribers, remains to the 0.8056 of the best family. - Extending the list does not get closer: three and four rules drop it to 0.7857, and the list saturates after the fourth rule. A single rule gives 0.7857, the number of the decision tree family. - The cost of readability grows along that axis: one subscriber for twelve coefficients in the Supervised Learning course, four subscribers for four conditions here. - The faithfulness of the rule list is 1 by definition; the 41 empty explanations of the third lesson do not occur here. - Thirteen subgroups were examined and 12 were read: in six the best family cannot be approached within 0.02, the farthest in the central region at 0.0862; in three subgroups, however, the rule list beats the best family. #### Next Step In ten lessons, what a number is, for whom it is, and what it rests on were measured: the share inflated by the split, the operating point moved by the cost matrix, the choice of the baseline, the margin of the gap, the distribution shift, the number of closed questions, the disagreement of two importance rankings, the faithfulness of the explanation, the interaction the average hides, and the cost of the rule list. In all of them, what was measured was **accuracy or readability**. None of them was a measure of **harm**. The gap of 0.0862 in the central region means that the subscribers there receive the suspicious flag wrongly more often; who receives how much of a wrong flag and what that flag does to them was never asked in ten lessons. The next topic opens with this question: where in the set the error comes from, how many separate definitions of being fair there are, whether those definitions can be satisfied at the same time, and what the model has learned leaks about individual records. ## Ethics and Responsibility Counting who is harmed by a system whose measure looks good: measuring data, label and deployment bias separately and seeing all three hit the same number by different paths, the impossibility of satisfying demographic parity, equal opportunity and predictive parity at the same time and how much fixing one breaks the others, how much adding or removing a single record moves the output, and reading side by side what is lost from accuracy and what is lost from recall as the noise level is swept, why anonymization alone is not enough, how many of the questions of a person who inherits a model are answered by the documentation, the number moved by the single item that is not pinned and the total number of items accumulated across the curriculum. ### Sources of Bias Source: https://academia.sh/en/courses/model-ethics-and-interpretation/sources-of-bias Counting separately that the same harm arises in three different places: the linear model measured on thirteen subgroups gives 0.8194 accuracy and 0.3704 recall overall, while recall stays at 0.2222 for subscribers with few readings. Because of the 69 subscribers who never enter the table, the share of few-reading subscribers reads 0.3349 instead of 0.3695; a training set built only from full-reading subscribers lowers precision in that subgroup from 0.5455 to 0.3774, the 19 suspicious cases that never reach the record drop recall to 0.0370, and a capacity of forty visits applied to the same subgroup at a rate of 0.0359 leaves recall at 0.1111. In all three, overall accuracy moves by at most 0.0337. The previous topic opened up the number the model produces over ten lessons: what the number is, who it is for and what the model based it on were measured. The explanation made a decision readable, but a readable decision is not a good decision. An explanation says which feature a decision rests on; it does not say **what the decision does to whom**, and a model that systematically harms a subgroup can be explained with the same clarity. This topic fills that gap, and the first question is the most concrete one: **where** the harm comes from. Who entered the table is one bias, whom the label missed is a second, and whom the model was applied to is a third. The three arise in different places, call for different fixes and, as this lesson will show, hit **the same number**. The bias that arises from how an observation is selected was measured in the Applied Statistics topic of the Statistical Analysis and Exploratory Data Analysis course; it is not repeated here, it is taken as input. - **ER1.** The set, the label, the seed and the split are inherited from the Introduction to Machine Learning through Reinforcement Learning and Recommender Systems courses and are **fictional**: 1,260 subscribers, a training set of 756 rows. The suspicion label is not a real inspection record. - **ER2.** Throughout this topic the validation and test sets are **merged** and read as a single held-out set of 504 subscribers; subgroup counts are already too thin at 252 subscribers. The baseline of the test set is 0.7579, and the baseline of the merged set is 0.7857. - **ER3.** The model is the twelve-column linear model of the Supervised Learning course; its settings are fixed and it gives 0.8016 on the test set; this is the line of comparison. - **ER4.** Subgroups are defined in this topic and stay fixed: region (five), tariff tier (three), household size (three), reading count (two). **Thirteen subgroups are examined** and this number is written in every lesson; in the held-out set, a subgroup smaller than fifty records does not enter the gap calculation. - **ER5.** Tariff tiers are built with thresholds of 10 and 25 m³; since two subscribers remain above 40 m³, the top tier is merged. The reading count axis is split in two: **few** (one or two periods) and **full** (three periods). - **ER6.** The measures are accuracy, recall and precision; their definitions were built in the Supervised Learning course and are not repeated here. In a subgroup where no flag is raised at all, precision is undefined. - **ER7.** On top of the inherited fiction, two layers valid only in this lesson are added for two measurements: a **recording process** in which part of the suspicious cases among subscribers with few readings never reach the record, and a **capacity** in which the field crew visits only the highest-scored. - **ER8.** Here bias is not an intention but a **measure**; all three mechanisms are by-products of ordinary decisions. #### Three Biases Arise in Three Different Places The setup generates the set, trains the linear model and measures the same model separately on thirteen subgroups. ```python # ethics.py — MODEL: The same FICTIONAL subscriber table, the same seed and the same split; # the target is the illicit-use or fault suspicion label, and that label is FICTIONAL too. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA, NO_READINGS = [], [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: NO_READINGS.append(k["region"][0]) continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "region_name": k["region"][0], "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, HELD_OUT = K[:756], K[756:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} def scaled(x): return [1.0] + [(x[a] - MEAN[a]) / STD[a] for a in FIELDS] def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def train_model(rows, label): # MODEL: linear model, gradient descent X = [scaled(x) for x in rows] Y = [x[label] for x in rows] w = [0.0] * len(X[0]) for _ in range(200): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= 0.3 * g[j] / len(X) return w def probability(w, x): return squash(sum(a * b for a, b in zip(w, scaled(x)))) def groups(x): t = 10 if x["mean_consumption"] <= 10 else 25 if x["mean_consumption"] <= 25 else 40 return ["region " + x["region_name"], "tariff %d" % t, "household " + ("1-2" if x["household"] <= 2 else "3-4" if x["household"] <= 4 else "5+"), "readings " + ("few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in HELD_OUT for g in groups(x)}) W = train_model(TRAIN, "suspicious") def measure(w, s): # accuracy, recall, precision susp = sum(x["suspicious"] for x in s) flagged = [x for x in s if probability(w, x) > 0.5] caught = sum(x["suspicious"] for x in flagged) return (sum((probability(w, x) > 0.5) == x["suspicious"] for x in s) / len(s), caught / susp if susp else float("nan"), caught / len(flagged) if flagged else float("nan")) print(f"subscribers {len(SUBSCRIBERS)}, none producing readings {len(NO_READINGS)}, " f"labeled {len(DATA)}, suspicious {sum(x['suspicious'] for x in DATA)}") print(f"train {len(TRAIN)}, held-out {len(HELD_OUT)}, subgroups examined {len(SUBGROUPS)}") print(f"baseline {sum(x['suspicious'] == 0 for x in HELD_OUT) / len(HELD_OUT):.4f}, overall" + "".join(f"{v:>11.4f}" for v in measure(W, HELD_OUT)) + "\n") print(f"{'subgroup':<14}{'train':>7}{'held-out':>9}{'suspicious':>11}" f"{'accuracy':>11}{'recall':>11}{'precision':>11}") for g in SUBGROUPS: s = [x for x in HELD_OUT if g in groups(x)] print(f"{g:<14}{sum(1 for x in TRAIN if g in groups(x)):>7}{len(s):>9}" f"{sum(x['suspicious'] for x in s):>11}" + "".join(f"{v:>11.4f}" for v in measure(W, s))) ``` ``` subscribers 1329, none producing readings 69, labeled 1260, suspicious 276 train 756, held-out 504, subgroups examined 13 baseline 0.7857, overall 0.8194 0.3704 0.6349 subgroup train held-out suspicious accuracy recall precision household 1-2 238 144 31 0.8264 0.3871 0.6667 household 3-4 398 289 61 0.8201 0.3607 0.6286 household 5+ 120 71 16 0.8028 0.3750 0.6000 readings few 255 167 27 0.8443 0.2222 0.5455 readings full 501 337 81 0.8071 0.4198 0.6538 region central 134 97 29 0.7423 0.2759 0.6667 region east 136 83 38 0.7108 0.6842 0.6842 region north 219 144 23 0.8264 0.1739 0.4000 region south 141 117 17 0.8632 0.1176 0.6667 region west 126 63 1 0.9841 0.0000 nan tariff 10 29 22 6 0.7273 0.0000 nan tariff 25 590 376 54 0.8511 0.2037 0.4583 tariff 40 137 106 48 0.7264 0.6042 0.7436 ``` The three numbers read overall look good: accuracy of 0.8194 is 0.0337 above the baseline of 0.7857, recall is 0.3704 and precision is 0.6349. On the subgroups the same model shows thirteen different behaviors: accuracy is 0.9841 in the west and 0.7108 in the east; recall is 0.6842 in the east and 0.1176 in the south. The two sharpest rows are the unmeasurable ones: in the west and in the lowest tariff tier no flag is raised at all, so precision is undefined, and that tier has only 29 rows in the training set. This lesson follows the reading count axis. Among few-reading subscribers accuracy is 0.8443, above the overall figure, but recall is 0.2222, below the 0.4198 of the full-reading subscribers; the gap is 0.1976. High accuracy is misleading: because the share of suspicious cases is low, flagging nobody at all already gives high accuracy. **The number to watch is 0.2222.** #### Data: Who Entered the Table Data bias is the training set not resembling the population the model will be applied to. Its source here is the measurement network: the fewer readings a subscriber produced, the more weakly it enters the table, and one that produced none does not enter at all. ```python FEW = [x for x in HELD_OUT if x["period"] < 3] FULL = [x for x in HELD_OUT if x["period"] == 3] few_in_table = sum(1 for x in DATA if x["period"] < 3) in_network = (few_in_table + len(NO_READINGS)) / (len(DATA) + len(NO_READINGS)) print(f"few-reading subscriber share in the table {few_in_table / len(DATA):.4f}, true share " f"in the network {in_network:.4f}, gap {in_network - few_in_table / len(DATA):.4f}") ONLY_FULL = [x for x in TRAIN if x["period"] == 3] W_FULL = train_model(ONLY_FULL, "suspicious") def show(name, w): a, b = measure(w, FEW), measure(w, HELD_OUT) print(f"{name:<26}{a[0]:>13.4f}{a[1]:>11.4f}{a[2]:>14.4f}{b[0]:>17.4f}{b[1]:>15.4f}") print(f"\n{'training set':<26}{'few accuracy':>13}{'few recall':>11}" f"{'few precision':>14}{'overall accuracy':>17}{'overall recall':>15}") show(f"{len(TRAIN)} rows, all", W) show(f"{len(ONLY_FULL)} rows, full-reading", W_FULL) ``` ``` few-reading subscriber share in the table 0.3349, true share in the network 0.3695, gap 0.0345 training set few accuracy few recall few precision overall accuracy overall recall 756 rows, all 0.8443 0.2222 0.5455 0.8194 0.3704 501 rows, full-reading 0.7605 0.7407 0.3774 0.7857 0.4815 ``` The first row gives the size of the shortfall: the share of few-reading subscribers is 0.3349 in the table and 0.3695 in the network, and the entire gap of 0.0345 comes from the 69 subscribers who never enter the table. The second table is one step further along the same mechanism: when training is built only from the 501 full-reading rows, recall in the few-reading subgroup **rises** to 0.7407, but precision falls from 0.5455 to 0.3774 and accuracy from 0.8443 to 0.7605. Because the model cannot tell that subgroup apart, it flags almost all of it. The direction of the harm is up in recall and down in precision; someone looking at a single number reads this as an improvement. Overall accuracy moves by only 0.0337. #### Label: Whom the Record Missed Label bias is concerned not with who is in the table but with **what is written next to them**. The suspicion label is the output of a recording process, and that process does not have the same information in every subgroup. ```python r = rng(SEED + 62000) missed = 0 for x in TRAIN: # FICTIONAL recording process: only in this measurement x["recorded"] = x["suspicious"] if x["period"] < 3 and x["suspicious"] and r() < 0.4: x["recorded"], missed = 0, missed + 1 positives = sum(1 for x in TRAIN if x["period"] < 3 and x["suspicious"]) print(f"few-reading training rows {sum(1 for x in TRAIN if x['period'] < 3)}, suspicious " f"{positives}, never reached the record {missed}; among full-reading ones, never recorded 0") W_REC = train_model(TRAIN, "recorded") print(f"\n{'trained label':<26}{'few accuracy':>13}{'few recall':>11}" f"{'few precision':>14}{'overall accuracy':>17}{'overall recall':>15}") show("ground truth", W) show("recorded label", W_REC) ``` ``` few-reading training rows 255, suspicious 44, never reached the record 19; among full-reading ones, never recorded 0 trained label few accuracy few recall few precision overall accuracy overall recall ground truth 0.8443 0.2222 0.5455 0.8194 0.3704 recorded label 0.8323 0.0370 0.3333 0.8056 0.2778 ``` Of the 255 few-reading rows in the training set, 44 are suspicious and 19 of them never reach the record; the missed records are 2.5% of the 756 rows in total. The effect is not small: recall in the few-reading subgroup drops to 0.0370, which means the model catches only one of the 27 suspicious cases there. Precision also falls to 0.3333; this mechanism damages both measures at once. The real difficulty is in the measure itself. If the model were trained with the recorded label and measured with the **recorded label**, nothing would be visible; the 19 missed suspicious cases would have been missed in the held-out set too, and the number would have come out high. A missing row can be counted, a wrong label cannot. #### Deployment: Whom the Model Was Applied To The third bias is not in the model but in its use. Every subscriber gets a score, but the field crew cannot visit them all; it goes only to the highest-scored. The model is then applied not to everyone but to **a subset**. ```python RANKED = sorted(HELD_OUT, key=lambda x: -probability(W, x)) susp_few, susp_full = sum(x["suspicious"] for x in FEW), sum(x["suspicious"] for x in FULL) def capacity(cap): top = RANKED[:cap] top_few = sum(1 for x in top if x["period"] < 3) return (top_few / len(FEW), (cap - top_few) / len(FULL), sum(1 for x in top if x["period"] < 3 and x["suspicious"]) / susp_few, sum(1 for x in top if x["period"] == 3 and x["suspicious"]) / susp_full, sum(x["suspicious"] for x in top) / (susp_few + susp_full)) print(f"{'capacity':>9}{'few applied':>17}{'full applied':>17}" f"{'few recall':>17}{'full recall':>17}{'overall recall':>17}") for cap in (40, 60, 80): print(f"{cap:>9}" + "".join(f"{v:>17.4f}" for v in capacity(cap))) c40 = capacity(40) print(f"\n{'bias':<26}{'few recall':>15}{'overall recall':>18}" f"{'overall accuracy':>18}") for name, w in (("none (base measurement)", W), ("data", W_FULL), ("label", W_REC)): a, b = measure(w, FEW), measure(w, HELD_OUT) print(f"{name:<26}{a[1]:>15.4f}{b[1]:>18.4f}{b[0]:>18.4f}") print(f"{'deployment (capacity 40)':<26}{c40[2]:>15.4f}{c40[4]:>18.4f}{'-':>18}") ``` ``` capacity few applied full applied few recall full recall overall recall 40 0.0359 0.1009 0.1111 0.3210 0.2685 60 0.0599 0.1484 0.2222 0.3951 0.3519 80 0.0958 0.1899 0.3333 0.4568 0.4259 bias few recall overall recall overall accuracy none (base measurement) 0.2222 0.3704 0.8194 data 0.7407 0.4815 0.7857 label 0.0370 0.2778 0.8056 deployment (capacity 40) 0.1111 0.2685 - ``` At a capacity of forty visits the model is applied to 0.0359 of the few-reading subscribers and 0.1009 of the full-reading ones; the application rate differs by nearly a factor of three between them. The result is read from recall: 0.1111 among few-reading subscribers and 0.3210 among full-reading ones. The model, the threshold and the training set did not change; only how many people were visited changed, and the gap rose to 0.2099. At a capacity of 80 the gap falls to 0.1235: deployment bias is a by-product of capacity. The last table is the total of the lesson. The same number of the same subgroup goes to 0.7407, 0.0370 and 0.1111 through three mechanisms — one up, two down — and overall accuracy moves by at most 0.0337. Counting the three separately is necessary, because each calls for a different fix: collecting the missing rows, auditing the recording process, distributing the capacity. Since thirteen subgroups are examined, part of these differences can also arise from the number of looks alone; the multiple comparisons lesson of the Statistical Analysis and Exploratory Data Analysis course measured its share, and this is why the subgroup count is written in every lesson. #### Summary - The linear model that gives 0.8194 accuracy and 0.3704 recall overall shows thirteen different behaviors on thirteen subgroups: accuracy is 0.9841 in the west and 0.7108 in the east, and in two subgroups precision stays undefined because no flag is raised at all. - Data bias is who entered the table: because of the 69 subscribers who produced no readings, the share of few-reading subscribers reads 0.3349 instead of 0.3695, and when training is built only from full-reading subscribers, precision falls from 0.5455 to 0.3774. - Label bias is what is written next to them: the 19 suspicious cases that never reach the record drop recall to 0.0370 and are not visible without a measure outside the label. - Deployment bias is whom the model was applied to: at a capacity of forty visits the model is applied to 0.0359 of the few-reading subscribers and 0.1009 of the full-reading ones, and the gap rises to 0.2099. - While the three mechanisms carry the same number in three different directions, overall accuracy moves by at most 0.0337. #### Next Step In this lesson, when the number of one subgroup came out lower than another's, it was assumed to be a harm and the gap was reported directly. Yet which measure a gap is measured in is a decision in its own right. Equalizing selection rates, equalizing the shares of suspicious cases caught and equalizing how many of the flagged are really suspicious are three different things. The next lesson builds these three criteria under their structural names, moves the threshold so as to fix each in turn and counts how much the others are broken. The result is not a tuning error. ### Fairness Metrics Source: https://academia.sh/en/courses/model-ethics-and-interpretation/fairness-metrics Counting that three fairness criteria cannot be satisfied at the same time: with a base rate of 0.4578 in the east region against 0.1663 in the other four regions, a single threshold gives a parity gap of 0.3984, an opportunity gap of 0.4842 and a predictive gap of 0.1242. While the same number of subscribers is flagged, tuning the threshold pair to demographic parity raises the predictive gap to 0.4800, tuning it to equal opportunity raises it to 0.4262, and tuning it to predictive parity raises the opportunity gap to 0.7038. The identity that the selection rate is the product of recall and base rate divided by precision is verified, and when two criteria are equalized the ratio of selection rates is shown to lock to the ratio of base rates, 2.7535; with capacity fixed there is no threshold pair that gives all three together, and the best attempt leaves a gap of 0.2727. The previous lesson assumed that when one subgroup's number came out lower than another's it was a harm, and reported the gap directly. The question of which number's gap was left open. Equalizing the **rate** of flagged subscribers in two subgroups, equalizing the **rate** of suspicious cases caught, and equalizing how much of what is flagged is actually suspicious are three separate demands. All three are reasonable, all three are measurable and all three cannot be met at the same time on the same model. This lesson builds those three criteria and counts the conflict between them. The result is not a tuning error: as long as the base rates of the two groups differ, the definitions of the criteria exclude each other. This is the second claim of the course, and here it is shown with an identity. - **ER9.** The set, split, model and subgroup definitions carry over unchanged from the previous lesson: a training set of 756 rows, a held-out set of 504 subscribers, a twelve-column linear model, **thirteen subgroups**. - **ER10.** The two groups in which the conflict is measured are taken from the **region axis**: A is the east region, B is the remaining four regions. The reason for the choice is the base rate difference; the west region cannot be taken alone, because it has a single suspicious case in the held-out set. - **ER11.** **Demographic parity** is the equality of the rates of flagged subscribers in the two groups. - **ER12.** **Equal opportunity** is the equality of recall in the two groups: how much of the suspicious cases is flagged. - **ER13.** **Predictive parity** is the equality of precision in the two groups: how much of what is flagged is actually suspicious. - **ER14.** The means of satisfying a criterion is giving **the group a separate threshold**. The model, its weights and its scores do not change at all; only the score above which a subscriber is flagged changes. - **ER15.** Threshold pairs are searched on a grid from 0.02 to 0.98 in steps of 0.01 and **the total number of flagged subscribers is held fixed** (63 subscribers, a shift of at most three). Without this constraint every criterion is satisfied by flagging no one and the comparison becomes meaningless. - **ER16.** A gap is the absolute difference between the numbers of the two groups; rather than how close it gets to zero, what is read is **what the other two gaps do**. #### Three Criteria, Three Different Equalities The setup reproduces the previous lesson's model, gives each subscriber in the held-out set a score and puts the five regions side by side at a single threshold. ```python # fairness.py — MODEL: The same FICTIONAL subscriber table, the same seed, the same split and # the previous lesson's twelve-column linear model; nothing changes except the threshold. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "region_name": k["region"][0], "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, HELD_OUT = K[:756], K[756:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} def scaled(x): return [1.0] + [(x[a] - MEAN[a]) / STD[a] for a in FIELDS] def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def train_model(rows, label): # MODEL: linear model, gradient descent X = [scaled(x) for x in rows] Y = [x[label] for x in rows] w = [0.0] * len(X[0]) for _ in range(200): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= 0.3 * g[j] / len(X) return w def probability(w, x): return squash(sum(a * b for a, b in zip(w, scaled(x)))) def groups(x): t = 10 if x["mean_consumption"] <= 10 else 25 if x["mean_consumption"] <= 25 else 40 return ["region " + x["region_name"], "tariff %d" % t, "household " + ("1-2" if x["household"] <= 2 else "3-4" if x["household"] <= 4 else "5+"), "readings " + ("few" if x["period"] <= 2 else "full")] W = train_model(TRAIN, "suspicious") SCORES = [(probability(W, x), x["suspicious"], x["region_name"]) for x in HELD_OUT] A = [p for p in SCORES if p[2] == "east"] B = [p for p in SCORES if p[2] != "east"] def measure(s, e): # selection, recall, precision, correct, flagged susp = sum(y for _, y, _ in s) n = sum(1 for p, _, _ in s if p > e) caught = sum(y for p, y, _ in s if p > e) return (n / len(s), caught / susp, caught / n if n else float("nan"), sum(1 for p, y, _ in s if (p > e) == bool(y)), n) print(f"train {len(TRAIN)}, held-out {len(HELD_OUT)}, subgroups examined " f"{len({g for x in HELD_OUT for g in groups(x)})}, baseline " f"{sum(x['suspicious'] == 0 for x in HELD_OUT) / len(HELD_OUT):.4f}") print(f"{'region':<10}{'n':>5}{'suspicious':>11}{'base rate':>12}{'selection':>10}" f"{'recall':>12}{'precision':>10}") for b in ("north", "south", "east", "west", "central"): s = [p for p in SCORES if p[2] == b] m = measure(s, 0.5) print(f"{b:<10}{len(s):>5}{sum(y for _, y, _ in s):>11}" f"{sum(y for _, y, _ in s) / len(s):>12.4f}{m[0]:>10.4f}{m[1]:>12.4f}{m[2]:>10.4f}") base_a = sum(y for _, y, _ in A) / len(A) base_b = sum(y for _, y, _ in B) / len(B) a0, b0 = measure(A, 0.5), measure(B, 0.5) print(f"\nA = east: n {len(A)}, base rate {base_a:.4f} | B = other four regions: " f"n {len(B)}, base rate {base_b:.4f}, ratio of rates {base_a / base_b:.4f}") print(f"single threshold 0.50, flagged {a0[4] + b0[4]}: parity gap " f"{abs(a0[0] - b0[0]):.4f}, opportunity gap {abs(a0[1] - b0[1]):.4f}, " f"predictive gap {abs(a0[2] - b0[2]):.4f}") ``` ``` train 756, held-out 504, subgroups examined 13, baseline 0.7857 region n suspicious base rate selection recall precision north 144 23 0.1597 0.0694 0.1739 0.4000 south 117 17 0.1453 0.0256 0.1176 0.6667 east 83 38 0.4578 0.4578 0.6842 0.6842 west 63 1 0.0159 0.0000 0.0000 nan central 97 29 0.2990 0.1237 0.2759 0.6667 A = east: n 83, base rate 0.4578 | B = other four regions: n 421, base rate 0.1663, ratio of rates 2.7535 single threshold 0.50, flagged 63: parity gap 0.3984, opportunity gap 0.4842, predictive gap 0.1242 ``` The column of the table that needs to be read is the **base rate**: the share of suspicious cases is 0.4578 in the east and 0.0159 in the west. This difference is not something the model produced but something that sits inside the fiction; the hidden score of the east region is 0.30 higher. The model sees this difference and flags 0.4578 of the subscribers in the east and 0.0256 in the south. The bottom line is the starting point of the lesson. With a single threshold all three gaps are far from zero: parity gap 0.3984, opportunity gap 0.4842, predictive gap 0.1242. Which one to fix becomes a choice, because the three look from separate sides: demographic parity asks that a subscriber not be flagged more often than the neighbors, equal opportunity asks that in no region suspicious cases be missed more than in another, and predictive parity asks that being flagged mean the same thing in every region. #### Fixing One Breaks the Others The way to satisfy a criterion is to give the two groups separate thresholds. The search below, holding the total number of flagged subscribers fixed, pulls each criterion one at a time to the point nearest zero and writes the other two gaps next to it. ```python GRID = [i / 100 for i in range(2, 99)] CA = {e: measure(A, e) for e in GRID} CB = {e: measure(B, e) for e in GRID} TARGET = a0[4] + b0[4] def equalize(k): # k: 0 selection, 1 recall, 2 precision best = None for ea in GRID: u = CA[ea] for eb in GRID: v = CB[eb] if abs(u[4] + v[4] - TARGET) > 3 or v[2] != v[2] or u[2] != u[2]: continue gap, acc = abs(u[k] - v[k]), (u[3] + v[3]) / len(HELD_OUT) if best is None or gap < best[0] - 1e-12 or (abs(gap - best[0]) < 1e-12 and acc > best[3]): best = (gap, ea, eb, acc) return best def show_row(name, ea, eb): u, v = CA[ea], CB[eb] print(f"{name:<22}{ea:>12.2f}{eb:>12.2f}{u[4] + v[4]:>12}" f"{abs(u[0] - v[0]):>12.4f}{abs(u[1] - v[1]):>16.4f}{abs(u[2] - v[2]):>15.4f}" f"{(u[3] + v[3]) / len(HELD_OUT):>10.4f}") print(f"{'configuration':<22}{'threshold A':>12}{'threshold B':>12}{'flagged':>12}" f"{'parity gap':>12}{'opportunity gap':>16}{'predictive gap':>15}{'accuracy':>10}") show_row("single threshold", 0.5, 0.5) for k, name in ((0, "demographic parity"), (1, "equal opportunity"), (2, "predictive parity")): gap, ea, eb, acc = equalize(k) show_row(name, ea, eb) ``` ``` configuration threshold A threshold B flagged parity gap opportunity gap predictive gap accuracy single threshold 0.50 0.50 63 0.3984 0.4842 0.1242 0.8194 demographic parity 0.81 0.40 60 0.0017 0.0632 0.4800 0.7857 equal opportunity 0.76 0.40 63 0.0379 0.0105 0.4262 0.7877 predictive parity 0.34 0.65 66 0.6365 0.7038 0.0000 0.7976 ``` In each row the number on the diagonal falls to zero and the others rise. If demographic parity is wanted, the east is given a threshold of 0.81 and the rest 0.40; the rates of flagged subscribers are equalized with a difference of 0.0017, but the predictive gap rises from 0.1242 to 0.4800 — 0.9000 of the subscribers flagged in the east are actually suspicious, while in the rest 0.4200 are. If equal opportunity is wanted, the thresholds become 0.76 and 0.40, the recalls meet with a difference of 0.0105, and the predictive gap stays at 0.4262. If predictive parity is wanted, the east is given 0.34 and the rest 0.65; the precisions are equalized exactly, but the opportunity gap shoots up to 0.7038, because 0.7895 of the suspicious cases are caught in the east and 0.0857 in the rest. Part of the cost comes from accuracy: overall accuracy falls to 0.7857, 0.7877 and 0.7976 in the three configurations, and under demographic parity the model sits exactly on the baseline. But the real cost is not in accuracy but in the other two gaps; no configuration can shrink all three columns at once. #### This Is Not a Tuning Error The pattern in the table is not fixed by a better search or a finer grid, because its source is the definitions themselves. The three criteria are not independent; there is an identity among them. If in a group the selection rate is $s$, recall $d$, precision $k$ and base rate $t$, the number of correct flags can be written in two ways and $$s = \frac{d \cdot t}{k}$$ follows. The block below first verifies this identity in the two groups, then tries to equalize recall and precision **together**. ```python print(f"\nidentity: selection = recall * base rate / precision (threshold 0.50)") for name, m, t in (("A", a0, base_a), ("B", b0, base_b)): print(f" {name}: {m[1] * t / m[2]:.6f} = {m[0]:.6f}") def jointly(tol): best = None for ea in GRID: u = CA[ea] for eb in GRID: v = CB[eb] if v[2] != v[2] or u[2] != u[2]: continue if tol and abs(u[4] + v[4] - TARGET) > tol: continue gap = max(abs(u[1] - v[1]), abs(u[2] - v[2])) if best is None or gap < best[0] - 1e-12: best = (gap, ea, eb) return best SUSP = sum(y for _, y, _ in SCORES) print(f"\nattempt to equalize recall and precision jointly ({SUSP} suspicious)") print(f"{'capacity constraint':<20}{'threshold A':>12}{'threshold B':>12}{'largest gap':>18}" f"{'flagged':>10}{'caught':>11}{'selection ratio':>16}") for tol, name in ((0, "none"), (3, "63 shift 3"), (15, "63 shift 15")): gap, ea, eb = jointly(tol) u, v = CA[ea], CB[eb] caught = round(u[1] * sum(y for _, y, _ in A)) + round(v[1] * sum(y for _, y, _ in B)) print(f"{name:<20}{ea:>12.2f}{eb:>12.2f}{gap:>18.4f}{u[4] + v[4]:>10}{caught:>11}" f"{u[0] / v[0]:>16.4f}") print(f"ratio of base rates {base_a / base_b:.4f}") ``` ``` identity: selection = recall * base rate / precision (threshold 0.50) A: 0.457831 = 0.457831 B: 0.059382 = 0.059382 attempt to equalize recall and precision jointly (108 suspicious) capacity constraint threshold A threshold B largest gap flagged caught selection ratio none 0.86 0.82 0.0120 2 2 5.0723 63 shift 3 0.74 0.43 0.2727 60 33 1.8445 63 shift 15 0.72 0.46 0.2486 48 29 2.7816 ratio of base rates 2.7535 ``` The identity holds in both groups to the sixth decimal place. Its consequence is this: if recall and precision are equalized in the two groups, then $s_A / s_B = t_A / t_B$, that is, the ratio of selection rates **locks** to the ratio of base rates, here 2.7535. Demographic parity asks for this ratio to be 1. The two can be true together only if the base rates are equal or if no one is flagged. The bottom table shows this by experiment. When the capacity constraint is removed, the search does find a threshold pair that makes all three gaps small at once — 0.86 and 0.82 — but at that point only two of the 504 subscribers are flagged and two of the 108 suspicious cases are caught. The equality has been achieved by doing nothing. When capacity is fixed at 63, the best attempt leaves a gap of 0.2727; under a looser constraint of 48 subscribers the gap falls to 0.2486 and the ratio of selection rates settles at 2.7816, next to the ratio of base rates, 2.7535 — where the identity predicts. The rule that follows from this is not a recommendation but a limit: **as long as the base rates differ, the three criteria cannot be satisfied at the same time.** The work to be done is choosing a criterion, writing the choice down and reporting the other two gaps together with their numbers; there is no single number called "a fair model". #### Summary - The share of suspicious cases is 0.4578 in the east region and 0.1663 in the remaining four regions; this base rate difference determines all the results of the lesson, and at a single threshold all three gaps are large: parity 0.3984, opportunity 0.4842, predictive 0.1242. - When the number of flagged subscribers is held fixed, demographic parity raises the predictive gap to 0.4800, equal opportunity raises it to 0.4262 and predictive parity raises the opportunity gap to 0.7038; overall accuracy falls to between 0.7857 and 0.7976. - The selection rate equals the product of recall and base rate divided by precision; when two criteria are equalized, the ratio of selection rates locks to the ratio of base rates, 2.7535. - With capacity fixed there is no threshold pair that gives all three together: the best attempt leaves a gap of 0.2727, and when the constraint is removed the equality is achieved only by flagging two subscribers. #### Next Step In two lessons, what was measured was **what the model did to whom**. What the model **says** about the records it was trained on was never asked. How much does a single subscriber entering or leaving the training table move the output, and does deleting the subscriber ID really make that subscriber invisible? The next lesson counts these: the trace a single record leaves on the output is measured, what adding noise takes from accuracy is swept, and how many subscribers stay alone in their own field combination in a anonymized table is counted. ### Privacy-Preserving Learning Source: https://academia.sh/en/courses/model-ethics-and-interpretation/privacy-preserving-learning Counting from the defense side what the model says about its own training records: in a 1,260-row table with the subscriber ID deleted, four subgroup axes produce 72 combinations and 6 records stand alone, and when the sixth field is added this number rises to 1,242, that is, to 0.9857. Removing a single training record moves the output of the linear model by at most 0.0313 and changes two predictions; the same record moves the output by 0.7778 and changes 14 predictions in the model built on the cell table, because 13 of the 66 cells hold a single record. When noise is swept over ten levels, masking the trace of the linear model asks for 0.04 noise and 0.0022 of accuracy, while that of the cell model asks for 0.80 noise and falling 0.1669 below the baseline. In two lessons, what was measured was what the model did to the outside, that is, to the subscribers it is applied to. What the model leaks to the **inside**, that is, toward the records it was trained on, was not asked. If the output of the model changes when a subscriber in the training set enters the table, that output carries something about that subscriber. The size of what it carries can be measured. This lesson makes that measurement and is limited to two questions. The first: does deleting the subscriber ID from a table make that subscriber invisible. The second: how much does removing a single record move the output of the model, and what is the cost in accuracy of masking that movement. What personal data is, the retention period and the limit of the analysis were measured in the Detection, Response and Compliance course and in The Structure of Data topic of the Introduction to Data Analytics course; they are not repeated. What is measured here is only the behavior of **the model** and **the table** themselves. - **ER17.** The set, split, model and subgroup definitions carry over unchanged from the previous two lessons: a training set of 756 rows, a held-out set of 504 subscribers, **thirteen subgroups**, baseline 0.7857. - **ER18.** The table is taken to be **anonymized**: there is no subscriber ID column. The remaining fields are ordinary fields that come from the measurement and from the subscriber record. - **ER19.** The measure of joinability is **how many records stand alone in their own field combination**. This is not an identification procedure but a count made on the table itself. - **ER20.** A second model is built for comparison: the **cell table**. The four subgroup axes give one cell, and the share of suspicious cases in the cell is the output. It is a small and weak model; its job here is to show how an output that by its structure rests on few records behaves. - **ER21.** **Record sensitivity** is the largest movement seen in the output of the model on the held-out set when a single training record is removed. Since every measurement on the linear model asks for a retraining, **sixteen records** are sampled; on the cell model all 756 records are computed exactly. - **ER22.** **Adding noise** is adding a number to the published output with equal probability in the negative and positive direction. The noise scale is swept over ten levels and at each level it is drawn twenty times and averaged. - **ER23.** A "record above the noise" is a record whose trace exceeds the largest value the noise scale can take. As the number falls, the share of individual records in the output stays inside the noise. - **ER24.** The measurement is entirely on the defense side: no inference procedure is built, only the quantities to be protected are counted. #### What Deleting the ID Deletes The first question is asked of the table before the model. When the subscriber ID is deleted, how many records does the combination of the remaining fields leave alone. The block below adds the fields one by one and counts at each step how many separate combinations form and how many records stand alone in their combination. ```python # privacy.py — MODEL: The same FICTIONAL subscriber table, the same seed and the same split; # what is measured is on the defense side: the behavior of the table and the model themselves. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "region_name": k["region"][0], "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, HELD_OUT = K[:756], K[756:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] MEAN = {a: statistics.fmean(x[a] for x in TRAIN) for a in FIELDS} STD = {a: max(1e-9, statistics.pstdev([x[a] for x in TRAIN])) for a in FIELDS} def scaled(x): return [1.0] + [(x[a] - MEAN[a]) / STD[a] for a in FIELDS] def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def train_model(rows, label): # MODEL: linear model, gradient descent X = [scaled(x) for x in rows] Y = [x[label] for x in rows] w = [0.0] * len(X[0]) for _ in range(200): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= 0.3 * g[j] / len(X) return w def probability(w, x): return squash(sum(a * b for a, b in zip(w, scaled(x)))) from collections import Counter, defaultdict def groups(x): t = 10 if x["mean_consumption"] <= 10 else 25 if x["mean_consumption"] <= 25 else 40 return ["region " + x["region_name"], "tariff %d" % t, "household " + ("1-2" if x["household"] <= 2 else "3-4" if x["household"] <= 4 else "5+"), "readings " + ("few" if x["period"] <= 2 else "full")] FIELD_NAMES = ["region", "tariff tier", "household size", "reading count", "satisfaction", "mean consumption"] def key_of(x, k): # the first k fields of the anonymized record return tuple((groups(x) + [x["satisfaction"], x["mean_consumption"]])[:k]) print(f"records {len(DATA)}, subscriber ID not in the table, subgroups examined " f"{len({g for x in HELD_OUT for g in groups(x)})}") print(f"{'field':>5} {'added field':<18}{'combinations':>14}{'lone records':>14}{'share':>9}") for k in range(1, 7): c = Counter(key_of(x, k) for x in DATA) single = sum(1 for x in DATA if c[key_of(x, k)] == 1) print(f"{k:>5} {FIELD_NAMES[k - 1]:<18}{len(c):>14}{single:>14}{single / len(DATA):>9.4f}") ``` ``` records 1260, subscriber ID not in the table, subgroups examined 13 field added field combinations lone records share 1 region 5 0 0.0000 2 tariff tier 15 1 0.0008 3 household size 41 4 0.0032 4 reading count 72 6 0.0048 5 satisfaction 247 69 0.0548 6 mean consumption 1251 1242 0.9857 ``` The first four rows are the subgroup axes of the course and are coarse fields: 1,260 records fall into only 72 combinations, and the lone records number 6. The fifth field raises this number to 69. The sixth field, that is, mean consumption, which carries two digits after the decimal point, splits the table into 1,251 combinations in a single step, and **0.9857** of the records stand alone in their own combination. What is to be read is not the size of the last row but the jump from the fourth row to the sixth. De-identification deletes a column; it does not delete joinability. **The number and the resolution** of the fields in a table determine how anonymized that table is more than the presence of the ID does. Rounding a continuous field to a tier lowers this number directly, and the cost of that is the detail the model sees. #### The Trace a Single Record Leaves The second question is asked of the model. When a training record is removed and the model is retrained, how much does the output on the held-out set move. The block below measures this on two models: the twelve-column linear model and the cell table given by the four axes. For the second model the measurement can be made exactly, because removing a record changes only the rate of its own cell. ```python def cell(x): return tuple(groups(x)) COUNT = defaultdict(lambda: [0, 0]) # MODEL: cell table, states the rate of the cell for x in TRAIN: h = COUNT[cell(x)] h[0], h[1] = h[0] + 1, h[1] + x["suspicious"] TRAIN_RATE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN) HELD_CELLS = Counter(cell(x) for x in HELD_OUT) def cell_rate(x): v = COUNT.get(cell(x)) return TRAIN_RATE if v is None else v[1] / v[0] D, flipped = [], 0 for x in TRAIN: n, k = COUNT[cell(x)] q = (k - x["suspicious"]) / (n - 1) if n > 1 else TRAIN_RATE D.append(abs(k / n - q)) if (k / n > 0.5) != (q > 0.5): flipped = max(flipped, HELD_CELLS[cell(x)]) D.sort(reverse=True) W = train_model(TRAIN, "suspicious") P0 = [probability(W, x) for x in HELD_OUT] r = rng(SEED + 71000) TRACE = [] for _ in range(16): x = TRAIN[int(r() * len(TRAIN))] w = train_model([y for y in TRAIN if y is not x], "suspicious") P1 = [probability(w, y) for y in HELD_OUT] TRACE.append((max(abs(a - b) for a, b in zip(P0, P1)), sum((a > 0.5) != (b > 0.5) for a, b in zip(P0, P1)))) TRACE.sort(reverse=True) held_base = sum(x["suspicious"] == 0 for x in HELD_OUT) / len(HELD_OUT) print(f"cells {len(COUNT)}, smallest cell size {min(v[0] for v in COUNT.values())}, " f"single-record cells {sum(1 for v in COUNT.values() if v[0] == 1)}, " f"two or fewer {sum(1 for v in COUNT.values() if v[0] <= 2)}") print(f"baseline {held_base:.4f}, left-out records {len(TRACE)} (linear), {len(D)} (cell)") print(f"\n{'model':<22}{'accuracy':>10}{'largest trace':>15}{'median trace':>14}" f"{'changed predictions':>21}") print(f"{'linear model':<22}" f"{sum((p > 0.5) == x['suspicious'] for p, x in zip(P0, HELD_OUT)) / len(HELD_OUT):>10.4f}" f"{TRACE[0][0]:>15.4f}{TRACE[len(TRACE) // 2][0]:>14.4f}{max(v[1] for v in TRACE):>21}") print(f"{'cell model':<22}" f"{sum((cell_rate(x) > 0.5) == x['suspicious'] for x in HELD_OUT) / len(HELD_OUT):>10.4f}" f"{D[0]:>15.4f}{D[len(D) // 2]:>14.4f}{flipped:>21}") ``` ``` cells 66, smallest cell size 1, single-record cells 13, two or fewer 22 baseline 0.7857, left-out records 16 (linear), 756 (cell) model accuracy largest trace median trace changed predictions linear model 0.8194 0.0313 0.0030 2 cell model 0.7917 0.7778 0.0074 14 ``` The difference between the two rows approaches twenty-five times. In the linear model, removing a record moves the probability by no more than 0.0313 for any of the 504 subscribers and changes at most two predictions; the median trace is 0.0030, that is, more than half of the records leave almost no trace in the output. The reason is that the model produces every output from **all** 756 records; the share of a single row is spread over thirteen weights. On the cell model the same measurement gives 0.7778. The reason is the numbers at the start of the second line: 13 of the 66 cells hold a single record, and 22 hold two or fewer. In a single-record cell, the published rate is the label of that single record itself. When the decision of that cell is flipped, the prediction of 14 subscribers in the held-out set changes. The rule follows from here and is independent of the model: **the fewer records an output rests on, the more exposed those records are in that output.** The same number is the other face of the small subgroup warning in the previous lesson — a subgroup too small to be measured is also a subgroup too small to be hidden. #### The Cost of Noise The direct way to shrink the trace is to add noise to the published output. Noise masks the trace but also takes accuracy away. The sweep below prints side by side, at ten noise levels, the accuracy of the two models and the number of records that leave a trace above that level. ```python print(f"\n{'noise':>8}{'linear accuracy':>17}{'cell accuracy':>15}" f"{'linear above':>14}{'cell above':>12}") for b in (0.00, 0.02, 0.04, 0.08, 0.12, 0.20, 0.30, 0.45, 0.60, 0.80): lin_acc = cell_acc = 0.0 for t in range(20): # the noise is drawn twenty times, the mean is printed rd = rng(SEED + 74000 + t * 13) lin_acc += sum(((p + b * (2 * rd() - 1)) > 0.5) == x["suspicious"] for p, x in zip(P0, HELD_OUT)) / len(HELD_OUT) rc = rng(SEED + 75000 + t * 13) noisy = {h: min(1.0, max(0.0, v[1] / v[0] + b * (2 * rc() - 1))) for h, v in COUNT.items()} cell_acc += sum((noisy.get(cell(x), TRAIN_RATE) > 0.5) == x["suspicious"] for x in HELD_OUT) / len(HELD_OUT) print(f"{b:>8.2f}{lin_acc / 20:>17.4f}{cell_acc / 20:>15.4f}" f"{sum(1 for v, _ in TRACE if v > b):>14}{sum(1 for v in D if v > b):>12}") ``` ``` noise linear accuracy cell accuracy linear above cell above 0.00 0.8194 0.7917 16 679 0.02 0.8184 0.7792 1 246 0.04 0.8172 0.7792 0 140 0.08 0.8151 0.7792 0 59 0.12 0.8126 0.7780 0 34 0.20 0.8043 0.7696 0 25 0.30 0.7910 0.7584 0 12 0.45 0.7560 0.7265 0 11 0.60 0.6908 0.6779 0 1 0.80 0.6436 0.6188 0 0 ``` For the linear model the account is cheap. None of the sixteen records leaves a trace above 0.04 noise, and at that level accuracy falls from 0.8194 to 0.8172, that is, by 0.0022 — still 0.0315 above the baseline. When the noise scale is raised to 0.30, accuracy drops to 0.7910, just above the baseline, and at 0.45 to 0.7560, below the baseline. That is, the whole of the cost comes not from the masking but from masking more than necessary. On the cell model the same account cannot be paid. To mask the largest trace, the noise scale has to be raised to 0.80; at that level accuracy is 0.6188 and **0.1669 below** the baseline. Worse, the model falls to 0.7792 already at 0.02 noise and drops below the baseline — because its noiseless form was exceeding the baseline by only 0.0060. A model with no margin to spend cannot cover the cost of the noise. Read together, the two columns give the result of the lesson. The cost of privacy is not a fixed number; **it depends on the structure of the model and on how far above the baseline it stands.** A model that produces its output from many records and clears the baseline comfortably can mask the trace of individual records almost for free. A model that produces its output from few records and sticks to the baseline can buy the same protection only by making itself useless. #### Summary - In a 1,260-row table with the subscriber ID deleted, four coarse fields produce 72 combinations and 6 records stand alone; when the sixth field is added, the records standing alone rise to 1,242, that is, to 0.9857. - De-identification deletes a column, it does not delete joinability; what decides is the number and the resolution of the fields. - Removing a single training record moves the output of the linear model by at most 0.0313 and changes two predictions; the median trace is 0.0030. - The same measurement gives 0.7778 on the cell model and changes 14 predictions, because 13 of the 66 cells hold a single record: the fewer records an output rests on, the more exposed those records are in that output. - The cost of noise varies with the model: on the linear model masking the trace asks for 0.04 noise and 0.0022 of accuracy, while on the cell model it asks for 0.80 noise and falling 0.1669 below the baseline. #### Next Step In three lessons the harm was measured in three separate forms and in each one an item to be reduced was found: the source of the bias was separated, a fairness criterion was chosen and the cost of the choice was counted, the record trace was measured and masked with noise. None of what was measured has so far reached **the person who will use the model**. How will someone who has not seen the three tables in this lesson know, when they take over the model, which subgroups were not measured, which fairness criterion was chosen with which threshold, and whether noise was added to the output. The next lesson turns this question into a number: of the set of questions the person taking over the model can ask, how many are closed by a written statement and which remain unanswered. ### Model Documentation Source: https://academia.sh/en/courses/model-ethics-and-interpretation/model-documentation Counting which of the questions of the person who takes over the model are closed by a written statement: the same model gives 0.7778 accuracy overall on the test set and 0.0198 above the baseline, while in 12 measurable subgroups of the 14 built on four axes the best reads 0.9688 and the worst 0.6111, the subgroup gap is 0.3576 and three subgroups fall below their own baseline. Of the handover question set of twenty-two questions, the statement that carries only the number measured overall closes 5, the statement that adds the subgroup table closes 12, and the statement that also writes the usage limit closes 18; the remaining 4 questions close in no version because none of them is a measurement question. In five usage decisions the first version produces 4 wrong decisions and the second 1, while the third produces none because it can say unknown for the unmeasured subgroup. Three lessons measured harm and tried to reduce it: the three sources of bias were counted one by one, the three fairness criteria were shown not to hold at the same time, and how much a single record moves the output and what the noise takes from accuracy were printed side by side. None of these measurements reaches **the person who will use the model**. The measurement stays in the notebook; the decision is made at another desk. This lesson measures what reaches that desk. What reaches it is **the model document**: a written statement that says what the model was measured on, for whom it was measured, and where it will not be used. The quality of the document is counted not by its character but by **how many questions it closes**. The Visualization, Reporting and Analysis at Scale course measured what a report makes its reader read back; the reader here is not a report reader but the person who takes over the model. - **ER31.** The set, label and model are inherited from the first seven courses of the Machine Learning and Data Science curriculum and are **fictional**: 1,260 subscribers, a 756/252/252 split, a baseline of 0.7579 on the test set. - **ER32.** The measure is accuracy and the baseline is the most common class of the training set. Both are computed **again** in every subgroup. - **ER33.** Subgroups are defined on four axes: region, tariff tier, household size, reading count. **This lesson splits the tariff axis into four tiers instead of the course's three and pulls the reading threshold down to one period; the number of subgroups is therefore fourteen, not the course's fixed thirteen.** The definition is a decision and **how many subgroups were looked at** is written down. - **ER34.** A subgroup of fewer than twenty subscribers is **not measured**; the threshold is a decision and is separate from the course's criterion of twenty-five. - **ER35.** The question set of the person who takes over is **fictional**: twenty-two questions. Each question is tied to a single statement item, and it counts as closed if that item is in the statement. - **ER36.** The three statement versions are nested: A carries only the number measured overall, B adds the subgroup table, C adds the usage limit as well. None of them is a template; each is a list of items. - **ER37.** The usage decision procedure reads only the statement: if there is a subgroup row it uses that, otherwise it uses the number measured overall. For a subgroup that was not measured, the right answer is "unknown". #### The Same Number, Twelve Subgroups ```python # documentation.py — MODEL: The same FICTIONAL subscriber table of the first seven courses is built with # the same seed (20260218), the FICTIONAL illicit-use or fault suspicion label is added on top, and the # same number is measured first overall, then in subgroups defined on four axes. import math SEED, RAW, M32, MIN_SIZE = 20260218, 1400, 0xFFFFFFFF, 20 REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue num = 10001 + i r, v = rng(SEED + 7000 + num), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + num) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (num - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "region": b[0], "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for q in REGION: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) def shuffle(items, seed): r, s = rng(seed), list(items) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return s K = shuffle(DATA, SEED + 90000) TRAIN, HELD_OUT = K[:756], K[1008:] def tree(s, depth, min_size=2): def impurity(u): p = sum(x["suspicious"] for x in u) / len(u) if u else 0.0 return 2 * p * (1 - p) p, best = sum(x["suspicious"] for x in s) / len(s), None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for a in FIELDS: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(s) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, min_size), "right": tree(best[4], depth - 1, min_size)} M = tree(TRAIN, 6) MOST_COMMON = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN)) def measure(s): """Baseline (most common class) and accuracy of the model on the same set.""" hits = 0 for x in s: u = M while "prediction" not in u: u = u["left"] if x[u["field"]] < u["threshold"] else u["right"] hits += u["prediction"] == x["suspicious"] return sum(x["suspicious"] == MOST_COMMON for x in s) / len(s), hits / len(s) AXES = [("region", lambda x: x["region"]), ("tariff tier", lambda x: "tier %d" % ( 1 + (x["mean_consumption"] >= 10) + (x["mean_consumption"] >= 25) + (x["mean_consumption"] >= 40))), ("household size", lambda x: "1-2 people" if x["household"] <= 2 else ("3-4 people" if x["household"] <= 4 else "5+ people")), ("reading count", lambda x: "few readings" if x["period"] <= 1 else "full readings")] t, m = measure(HELD_OUT) print(f"test set {len(HELD_OUT)} subscribers; overall baseline {t:.4f}, model {m:.4f}, " f"difference {m - t:+.4f}") print(f"\n{'axis':<16}{'subgroup':<14}{'subscribers':>12}{'baseline':>10}{'model':>8}{'difference':>12}") SUBGROUPS, UNMEASURED = [], [] for name, f in AXES: for g in sorted({f(x) for x in HELD_OUT}): sub = [x for x in HELD_OUT if f(x) == g] if len(sub) < MIN_SIZE: UNMEASURED.append((g, len(sub))) continue bt, bm = measure(sub) SUBGROUPS.append((g, len(sub), bt, bm)) print(f"{name:<16}{g:<14}{len(sub):>12}{bt:>10.4f}{bm:>8.4f}{bm - bt:>+12.4f}") best, worst = max(SUBGROUPS, key=lambda u: u[3]), min(SUBGROUPS, key=lambda u: u[3]) print(f"\n{len(AXES)} axes built {len(SUBGROUPS) + len(UNMEASURED)} subgroups; " f"{len(UNMEASURED)} of them with fewer than {MIN_SIZE} subscribers were not measured " f"({', '.join(g + ' ' + str(n) for g, n in UNMEASURED)})") print(f"in the {len(SUBGROUPS)} measured subgroups best {best[3]:.4f} ({best[0]}), worst " f"{worst[3]:.4f} ({worst[0]}), subgroup gap {best[3] - worst[3]:.4f}; subgroups below " f"their own baseline: {sum(1 for u in SUBGROUPS if u[3] < u[2])}") ``` ``` test set 252 subscribers; overall baseline 0.7579, model 0.7778, difference +0.0198 axis subgroup subscribers baseline model difference region central 58 0.6552 0.6724 +0.0172 region east 36 0.4722 0.6111 +0.1389 region north 68 0.8235 0.7794 -0.0441 region south 58 0.8276 0.8793 +0.0517 region west 32 1.0000 0.9688 -0.0312 tariff tier tier 2 194 0.8402 0.8299 -0.0103 tariff tier tier 3 46 0.4565 0.6304 +0.1739 household size 1-2 people 73 0.7534 0.7945 +0.0411 household size 3-4 people 143 0.7692 0.7762 +0.0070 household size 5+ people 36 0.7222 0.7500 +0.0278 reading count few readings 22 0.8636 0.8636 +0.0000 reading count full readings 230 0.7478 0.7696 +0.0217 4 axes built 14 subgroups; 2 of them with fewer than 20 subscribers were not measured (tier 1 11, tier 4 1) in the 12 measured subgroups best 0.9688 (west), worst 0.6111 (east), subgroup gap 0.3576; subgroups below their own baseline: 3 ``` What is read overall is a single line: the model gives 0.7778 accuracy and beats the baseline by 0.0198. The subgroup table measures the same model in twelve separate places, and the single line splits into twelve sentences. The **subgroup gap between the best and the worst is 0.3576**; eighteen times the difference read overall. In three subgroups the model falls **below its own baseline**: it trails by 0.0441 in the north, by 0.0312 in the west, and by 0.0103 in tariff tier 2. In the west the baseline is 1.0000, that is, there are no suspicious subscribers at all there, and the model flags one subscriber for nothing. In the "few readings" subgroup the difference is exactly **0.0000**: in those twenty-two subscribers the model does nothing different from saying the most common class. Two of the fourteen subgroups, on the other hand, were not measured at all, because they come to eleven and one subscribers in the test set — but this does not mean the model will not be used there. #### The Twenty-Two Questions of the Person Who Takes Over ```python # questions.py — the question set of the person who takes over the model is FICTIONAL and has twenty-two # questions. Each question is tied to the single statement item that closes it; a document version # is a set of items, and the number of closed questions is counted by looking at the set. QUESTIONS = [ ("what does the model predict", "measure"), ("which number was reported", "overall"), ("on which set was this number measured", "set"), ("what is the baseline and does the model beat it", "overall"), ("which periods does the data cover", "set"), ("which fields go into the model", "feature"), ("how was the label produced", "label"), ("on which subgroups was the number measured", "subgroup_definition"), ("how many subgroups on how many axes were examined", "subgroup_definition"), ("what is the number in the worst subgroup", "subgroup_table"), ("how large is the subgroup gap", "subgroup_table"), ("in which subgroup is the model below its own baseline", "subgroup_table"), ("was any subgroup left unmeasured", "unmeasured"), ("why was it not measured", "unmeasured"), ("where is the model not used", "limit"), ("does the output decide by itself", "decision"), ("at what threshold is the output turned into a decision", "decision"), ("when is the number measured again", "remeasure"), ("how does a subscriber appeal against the output", "appeal"), ("what does a wrong output do to the subscriber", "consequence"), ("who verified these numbers", "verifier"), ("who rolls the model back when it breaks", "rollback"), ] A = {"measure", "overall", "set"} B = A | {"feature", "label", "subgroup_definition", "subgroup_table"} C = B | {"unmeasured", "limit", "decision", "remeasure"} VERSIONS = [("A: only the number measured overall", A), ("B: A + subgroup table", B), ("C: B + usage limit statement", C)] print(f"{'document version':<38}{'items':>6}{'closed':>9}{'open':>6}{'closed share':>14}") for name, s in VERSIONS: k = sum(1 for _, field in QUESTIONS if field in s) print(f"{name:<38}{len(s):>6}{k:>9}{len(QUESTIONS) - k:>6}" f"{k / len(QUESTIONS):>14.4f}") NEVER_CLOSED = [q for q, field in QUESTIONS if field not in C] print(f"\nthe {len(NEVER_CLOSED)} questions that none of the three versions closes:") for q in NEVER_CLOSED: print(f" - {q}") ``` ``` document version items closed open closed share A: only the number measured overall 3 5 17 0.2273 B: A + subgroup table 7 12 10 0.5455 C: B + usage limit statement 11 18 4 0.8182 the 4 questions that none of the three versions closes: - how does a subscriber appeal against the output - what does a wrong output do to the subscriber - who verified these numbers - who rolls the model back when it breaks ``` A statement of three items closes five of the twenty-two questions. Adding four items brings the closed questions to twelve, and adding four more brings them to eighteen. The number of closed questions does not rise evenly with the number of items: the subgroup table alone is enough for three questions at once. The place to read is the last list. Four questions close in none of the three versions, and none of the four is a measurement question. The appeal route, what a wrong output does to the subscriber, who verified the number and who will roll the model back: the answers to these are neither in the test set nor in the subgroup table. A statement **cannot close** these four, but it **can write that they are open**. Counting what is left open is also a measure of the statement. #### The Decision the Statement Changed ```python # decisions.py — the person who takes over makes five usage decisions and reads only the document. # The procedure first looks for the subgroup row; if it cannot find one, it falls back to the number # measured overall. Version C can say "unknown" because it recorded the unmeasured subgroups. ROWS = {u[0]: u for u in SUBGROUPS} MISSING = dict(UNMEASURED) OVERALL = "yes" if m > t else "no" def answer(version, group): if "subgroup_table" in version and group in ROWS: return "yes" if ROWS[group][3] > ROWS[group][2] else "no" if "unmeasured" in version and group in MISSING: return "unknown" return OVERALL def actual(group): if group in ROWS: return "yes" if ROWS[group][3] > ROWS[group][2] else "no" return "unknown" DECISIONS = ["north", "east", "west", "few readings", "tier 4"] print(f"{'subgroup':<14}{'measured':>12}" + "".join(f"{'version ' + name.split(':')[0]:>12}" for name, _ in VERSIONS)) for g in DECISIONS: print(f"{g:<14}{actual(g):>12}" + "".join(f"{answer(s, g):>12}" for _, s in VERSIONS)) print() for name, s in VERSIONS: y = sum(1 for g in DECISIONS if answer(s, g) != actual(g)) print(f"{name:<38}{len(DECISIONS) - y} correct, {y} wrong decisions") print(f"\noverall the model beats the baseline by {m - t:+.4f}; in the north " f"{ROWS['north'][3] - ROWS['north'][2]:+.4f}, in the west " f"{ROWS['west'][3] - ROWS['west'][2]:+.4f}, in few readings " f"{ROWS['few readings'][3] - ROWS['few readings'][2]:+.4f}") ``` ``` subgroup measured version A version B version C north no yes no no east yes yes yes yes west no yes no no few readings no yes no no tier 4 unknown yes yes unknown A: only the number measured overall 1 correct, 4 wrong decisions B: A + subgroup table 4 correct, 1 wrong decisions C: B + usage limit statement 5 correct, 0 wrong decisions overall the model beats the baseline by +0.0198; in the north -0.0441, in the west -0.0312, in few readings +0.0000 ``` The question is "does using the model in this subgroup add anything over the baseline", and it is asked five times. The person who reads version A says "yes" to all five, because they hold a single number and that number is right for the whole; four come out wrong. Version B corrects four decisions but errs on the fifth: tariff tier 4 is not in the table, the procedure falls back to the number measured overall, and it answers **as if an unmeasured subgroup had been measured**. The fifth row is the number of the lesson. The only difference between C and B is that it records the two unmeasured subgroups and why they were not measured. That record brings no new performance; the only thing it brings is the authority to say "unknown", and it takes the wrong decisions down to zero. The rule of the course is at work inside this table. The 0.7778 in version A is not miscalculated; it is a number that cannot be read because it was not written down for whom it was measured. #### Summary - The same model gives 0.7778 accuracy overall on the test set and beats the baseline by 0.0198, while in 12 measurable subgroups of the 14 built on four axes the best reads 0.9688 and the worst 0.6111; the subgroup gap is 0.3576, eighteen times the difference read overall. - In three subgroups the model falls below its own baseline (north 0.0441, west 0.0312, tariff tier 2 trailing by 0.0103), and in the "few readings" subgroup the difference is exactly 0.0000. - Of the handover question set of twenty-two questions, version A closes 5, version B 12 and version C 18; the number of closed questions does not rise evenly as items are added. - Four questions close in no version and none of the four is a measurement question; a statement cannot close them but can write that they are open, and counting what is left open is also a measure of the statement. - In five usage decisions version A produces 4 wrong decisions and version B 1; the one advantage of version C is that it records the unmeasured subgroup and can say "unknown", and this takes the wrong decisions down to zero. #### Next Step The statement of this lesson carries the measured number, but nowhere does it say that the number **can be produced a second time**. The numbers of fourteen lessons were produced with a single seed, a single split and a single threshold; if one of these items is not fixed, how much which number will move was not asked. The next lesson collects the items accumulated from the preparation notebook of the Data Collection and Preparation course to the training loop of the Deep Learning course, measures by how many subscribers a single unfixed item shifts the test count, and gives the full item list of the Machine Learning and Data Science curriculum. That lesson also closes the course and the curriculum. ### Reproducibility Source: https://academia.sh/en/courses/model-ethics-and-interpretation/reproducibility Counting the items that must be fixed for the same number to be produced a second time, and how much a single unfixed item moves it: 6 of the evaluation's own 9 items each move the test count on their own, the largest shift being 0.0370 or 7 subscribers; 8 of the same items move the subgroup gap, and the largest shift is 0.0989, 2.7 times what is read overall. When the minimum subgroup size is released, neither headline number changes at all but the measured subgroups fall from 12 to 11. On top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course, the 11 of the Visualization, Reporting and Analysis at Scale course, the 4 of the Data Preparation and Feature Engineering course and the 8 of the Deep Learning course, the evaluation's 9 items are added and the full list of the curriculum reaches 80. The course wrap-up gathers the fifteen lessons overall and in the worst subgroup, and the curriculum wrap-up gathers the eight courses and 121 lessons of the Machine Learning and Data Science curriculum in a single table. The previous lesson counted that a statement closes eighteen of twenty-two questions. The number the statement carried came from a single run: one seed, one split, one depth, one threshold. Nowhere was it shown that the same number would come out when that run is made a second time. This lesson prints two numbers. The first is **the total number of items**: how many decisions must be written down for the same test count to be reproduced. The second is **the number a single unfixed item moves**. The item count is **inherited** from the Data Collection and Preparation, Data Analysis with Python, Visualization, Reporting and Analysis at Scale, Data Preparation and Feature Engineering and Deep Learning courses; this lesson only adds **the evaluation's own** items and gives the total. - **ER38.** The set, label and model are inherited from the first seven courses of the Machine Learning and Data Science curriculum and are **fictional**; the preparation items were fixed in the Data Collection and Preparation course and stay fixed in this lesson. - **ER39.** **Releasing** an item does not mean deleting it but moving it to another **equally defensible** value. The alternatives are fictional and none of them is wrong. - **ER40.** Two numbers are read in every run: the accuracy measured overall on the test set and the **subgroup gap** measured on the four axes of the previous lesson. - **ER41.** Items are released one at a time; while one is released, the other eight stay fixed. - **ER42.** When the shift is converted to a number of subscribers, **that run's own** test set size is used, because the set size also changes when the split item is released. - **ER43.** Floating-point comparison is made with a threshold of `1e-9`; this threshold is far below an accuracy difference rounded to four decimals. #### The Evaluation's Own Items ```python # reproducibility.py — MODEL: The FICTIONAL subscriber table is built with the same seed (20260218); # the preparation items were fixed in the Data Collection and Preparation course and stay fixed here. # The evaluation's own items are collected in a single dictionary, and the run reads it. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] DEFAULTS = {"split_seed": 90000, "train_share": 0.60, "depth": 6, "leaf_min": 2, "leaf_threshold": 0.5, "candidate_count": 10, "subgroup_min": 20, "tiers": (10, 25, 40), "field_order": "natural"} def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 DATA = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() < 0.046: continue num = 10001 + i r, v = rng(SEED + 7000 + num), [] for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( b[2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + num) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3] + 0.9 * (num - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": household, "satisfaction": rating, "period": len(v), "region": b[0], "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for q in REGION: x["area_" + q[0]] = int(b[0] == q[0]) DATA.append(x) def run(**released): """Reads the item dictionary and returns the test accuracy and the subgroup gap.""" a = dict(DEFAULTS, **released) r, s = rng(SEED + a["split_seed"]), list(DATA) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] n = int(len(s) * a["train_share"]) train, held = s[:n], s[n + (len(s) - n) // 2:] fields = FIELDS if a["field_order"] == "natural" else FIELDS[::-1] def tree(u, depth): p, best = sum(x["suspicious"] for x in u) / len(u), None if depth and len(u) >= 2 * a["leaf_min"] and 0.0 < p < 1.0: g = 2 * p * (1 - p) for field in fields: d = sorted({x[field] for x in u}) for v in (d[1:] if len(d) < a["candidate_count"] else [d[int(i * len(d) / a["candidate_count"])] for i in range(1, a["candidate_count"])]): left = [x for x in u if x[field] < v] right = [x for x in u if x[field] >= v] if min(len(left), len(right)) < a["leaf_min"]: continue q = 0.0 for y in (left, right): w = sum(x["suspicious"] for x in y) / len(y) q += len(y) * 2 * w * (1 - w) / len(u) if best is None or g - q > best[0]: best = (g - q, field, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > a["leaf_threshold"])} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1), "right": tree(best[4], depth - 1)} M = tree(train, a["depth"]) def accuracy(u): hits = 0 for x in u: k = M while "prediction" not in k: k = k["left"] if x[k["field"]] < k["threshold"] else k["right"] hits += k["prediction"] == x["suspicious"] return hits / len(u) t1, t2, t3 = a["tiers"] axes = [lambda x: x["region"], lambda x: "tier %d" % (1 + (x["mean_consumption"] >= t1) + (x["mean_consumption"] >= t2) + (x["mean_consumption"] >= t3)), lambda x: "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+"), lambda x: "few" if x["period"] <= 1 else "full"] measured = [] for f in axes: for g in sorted({f(x) for x in held}): sub = [x for x in held if f(x) == g] if len(sub) >= a["subgroup_min"]: measured.append(accuracy(sub)) return (accuracy(held), max(measured) - min(measured), len(measured), len(held)) BASE, GAP, COUNT, N = run() print(f"with all items fixed, test {BASE:.4f}, subgroup gap {GAP:.4f}; " f"{COUNT} measured subgroups in the test set of {N} subscribers") ``` ``` with all items fixed, test 0.7778, subgroup gap 0.3576; 12 measured subgroups in the test set of 252 subscribers ``` None of the nine items is hidden inside a line of code; all of them sit in a single dictionary that the run function reads. This is the first condition of fixing: whether an item buried in the code is fixed or released cannot be asked. #### A Single Unfixed Item ```python # sweep.py — each item is released ALONE: it is moved to another equally defensible value # while the others stay fixed. Two numbers are read at once: the test accuracy # measured overall and the subgroup gap. RELEASED = [("split seed", "split_seed", 90001), ("train share", "train_share", 0.70), ("tree depth", "depth", 5), ("minimum subscribers in a leaf", "leaf_min", 5), ("leaf threshold", "leaf_threshold", 0.45), ("candidate threshold count", "candidate_count", 8), ("minimum subgroup size", "subgroup_min", 30), ("tariff tier boundaries", "tiers", (12, 28, 45)), ("feature list order", "field_order", "reversed")] print(f"{'released item':<32}{'new value':<14}{'test':>10}" f"{'subscribers':>13}{'gap':>8}{'shift':>9}{'subgroups':>11}") SHIFTS = [] for label, key, value in RELEASED: s, c, k, n = run(**{key: value}) SHIFTS.append((label, abs(s - BASE), abs(c - GAP), n)) print(f"{label:<32}{str(value):<14}{s:>10.4f}{round(abs(s - BASE) * n):>13}" f"{c:>8.4f}{c - GAP:>+9.4f}{k:>11}") moved = sum(1 for _, u, _, _ in SHIFTS if u > 1e-9) top_acc = max(SHIFTS, key=lambda t: t[1]) top_gap = max(SHIFTS, key=lambda t: t[2]) print(f"\nof {len(RELEASED)} items, {moved} move the test count on their own; " f"largest shift {top_acc[1]:.4f} ({top_acc[0]}), {round(top_acc[1] * top_acc[3])} subscribers") print(f"{sum(1 for _, _, u, _ in SHIFTS if u > 1e-9)} of the same items move the subgroup " f"gap; largest shift {top_gap[2]:.4f} ({top_gap[0]}), " f"{top_gap[2] / top_acc[1]:.1f} times the largest shift read overall") ``` ``` released item new value test subscribers gap shift subgroups split seed 90001 0.8056 7 0.3373 -0.0204 12 train share 0.7 0.8148 7 0.4286 +0.0709 11 tree depth 5 0.7897 3 0.4565 +0.0989 12 minimum subscribers in a leaf 5 0.7778 0 0.3601 +0.0024 12 leaf threshold 0.45 0.7817 1 0.3723 +0.0146 12 candidate threshold count 8 0.7579 5 0.4470 +0.0894 12 minimum subgroup size 30 0.7778 0 0.3576 +0.0000 11 tariff tier boundaries (12, 28, 45) 0.7778 0 0.3854 +0.0278 13 feature list order reversed 0.7817 1 0.3601 +0.0024 12 of 9 items, 6 move the test count on their own; largest shift 0.0370 (train share), 7 subscribers 8 of the same items move the subgroup gap; largest shift 0.0989 (tree depth), 2.7 times the largest shift read overall ``` Six of the nine items move the test count on their own, and the largest shift is seven subscribers. Seven subscribers may look small; yet the model's whole difference above the baseline is five subscribers. The sixth row writes this most sharply: when the candidate threshold count is 8 instead of 10, the test accuracy is **0.7579**, that is, exactly the baseline. A single unjustified item turns a model that beats the baseline into a model that does not. The two columns on the right are the course's own measure, and a harsher one. The largest shift in the gap is 0.0989, **2.7 times** what is read overall. The subgroup gap is more fragile, because it is measured on small sets and read from the extremes. The last column is the invisible face of fixing. In the "minimum subgroup size" row neither headline number moves at all, but the measured subgroups fall from 12 to 11; when the tariff tier boundaries are released they rise to 13. An item can change **what is measured** without changing the reported number, and the previous lesson counted how that spoils the decision. #### The Full Item List of the Machine Learning and Data Science Curriculum ```python # total.py — the full item list of the curriculum. The first five sources were COUNTED IN EARLIER COURSES # and are not recomputed here; this lesson only adds the evaluation's items # and gives the total. SOURCES = [("preparation notebook (Data Collection and Preparation)", 39), ("working environment (Data Analysis with Python)", 9), ("the report itself (Visualization, Reporting and Analysis at Scale)", 11), ("preparation chain (Data Preparation and Feature Engineering)", 4), ("training itself (Deep Learning)", 8), ("evaluation itself (this lesson)", len(RELEASED))] for label, n in SOURCES: print(f" {label:<68}{n:>3} items") print(f" {'total':<68}{sum(n for _, n in SOURCES):>3} items") print(f"the moment one of the eighty items is not fixed, the number read overall " f"can drift by {round(top_acc[1] * top_acc[3])} subscribers and the subgroup gap by {top_gap[2]:.4f}") ``` ``` preparation notebook (Data Collection and Preparation) 39 items working environment (Data Analysis with Python) 9 items the report itself (Visualization, Reporting and Analysis at Scale) 11 items preparation chain (Data Preparation and Feature Engineering) 4 items training itself (Deep Learning) 8 items evaluation itself (this lesson) 9 items total 80 items the moment one of the eighty items is not fixed, the number read overall can drift by 7 subscribers and the subgroup gap by 0.0989 ``` Eighty items make a single list, and they were counted piece by piece in six separate courses. Its length is not a flaw but a measure, and none of these decisions writes itself. #### Summary - The evaluation's own items number nine and all of them sit in a single dictionary; whether an item buried in the code is fixed or released cannot be asked. - Six of the nine items move the test count on their own; the largest shift is seven subscribers with 0.0370, and this is larger than the model's five-subscriber difference above the baseline. When the candidate threshold count is 8, the test accuracy falls to exactly the baseline, 0.7579. - Eight of the same items move the subgroup gap; the largest shift is 0.0989, 2.7 times what is read overall, because the gap is read on small sets and from the extremes. - When the minimum subgroup size is released, neither headline number moves but the measured subgroups fall from 12 to 11: an item can change what is measured without changing the reported number. - On top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course, the 11 of the Visualization, Reporting and Analysis at Scale course, the 4 of the Data Preparation and Feature Engineering course and the 8 of the Deep Learning course, the evaluation's 9 items are added and the full list of the curriculum comes to 80. #### Course Wrap-Up In each of the fifteen lessons the same number was read twice: once overall, once in subgroups. Because the rows do not share a unit, the cells state by which criterion they were read, and every number is taken from its own lesson. | Lesson | Overall | In the worst subgroup | Gap | |---|---|---|---| | Evaluation 01 Validation Strategies | accuracy 0.7778, baseline +0.0198 | 0.6111 | 0.3576 | | Evaluation 02 Metric Selection | cost gain +0.0198 or +0.1984 | −0.4468 | 0.8606 | | Evaluation 03 Baseline Model | against eight baselines −0.0119 … +0.5357 | −0.0833 in the east | 0.5476 moved by the choice | | Evaluation 04 Statistical Significance | difference +0.0198, its margin includes zero | −0.0471 in few readings | 4 of 13 differences exclude zero | | Evaluation 05 Distribution Shift | difference above the baseline 0.0198 → 0.0066 | none below the baseline | 0.3576 → 0.2573 | | Interpretability 01 The Need for Interpretability | closed questions 8.58/12 | 7.25 questions | 1.64 questions | | Interpretability 02 Feature Importance | the ranking diverges in 10 of 12 columns | 0.0162 in the most important column | 0.1720 | | Interpretability 03 Local Explanation Methods | faithfulness 0.9417 | 0.9027 | 0.0637 | | Interpretability 04 Partial Dependence and Interaction | household slope −0.0069 (flat) | −0.1891 in the east | 0.3192 | | Interpretability 05 Inherently Interpretable Models | rule list 0.7897, 0.0159 to the family | 0.0862 away in the central region | 6 subgroups cannot get within 0.02 | | Ethics 01 Sources of Bias | accuracy 0.8194, recall 0.3704 | 0.7108 in the east | 0.2733 | | Ethics 02 Fairness Metrics | the best attempt leaves a gap of 0.2727 | base rate 0.4578 in the east | recall 0.4842, 0.7038 under predictive parity | | Ethics 03 Privacy-Preserving Learning | trace of a single record 0.0313 / 0.7778 | records standing alone with the sixth field 0.9857 | cost of noise 0.0022 against 0.1669 | | Ethics 04 Model Documentation | accuracy 0.7778; the statement closes 18/22 questions | 0.6111 (east) | 0.3576 | | Ethics 05 Reproducibility | 0.7778; a single item moves 7 subscribers | 0.7579 at candidate threshold count 8 | 0.3576; a single item moves it by 0.0989 | The third column is the rule of the course: **a number cannot be read without writing for whom it was measured; a performance whose subgroup is not written is counted as unmeasured.** In no row did the gap come out zero. The eight courses of the Machine Learning and Data Science curriculum read the same axis in eight separate places. | Course | Lessons | Measure axis | Unit | |---|---|---|---| | Introduction to Machine Learning | 11 | separation | in training and in the held-out set | | Data Preparation and Feature Engineering | 12 | the difference a feature adds | contribution | | Supervised Learning | 20 | what the family adds and what the budget inflates | candidate tried | | Unsupervised Learning | 11 | what the baseline is when there are no labels | baseline built | | Deep Learning | 24 | what capacity adds and the share of training | second run | | Natural Language Processing and Computer Vision | 14 | the domain's own baseline | the domain's baseline | | Reinforcement Learning and Recommender Systems | 14 | the decision changing its own data | data collected | | Model Evaluation, Interpretability and Ethics | 15 | baseline for whom | subgroup gap | | Total | 121 | | | One hundred twenty-one lessons established a single rule eight times: **a model is as good as how far it beats a dumber procedure; a performance whose baseline is not written is counted as unmeasured.** The rule was paid in the same form in every course: next to the number of the model went the number of a dumber procedure, and where the difference came from was questioned. The second claim was paid in every course too: **the difference does not always come from learning.** Leakage, selection share, tuning budget, imbalanced classes and, in the last course, the definition of the subgroup itself produced a difference. In one hundred twenty-one lessons a model was built, measured, explained, and what it does to whom was counted. But all of it happened **in a single run, on a single machine, inside a notebook**. The model becoming a **system** — taking a version, gaining an interface that others call, being rolled back when it breaks and having its cost counted — was never taken up. The next curriculum, AI Engineering, opens with this question.