--- title: 'The Need for Interpretability' source: 'https://academia.sh/en/courses/model-ethics-and-interpretation/need-for-interpretability' course: 'Model Evaluation, Interpretability and Ethics' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Splits the questions that can be asked about a model decision into three: those the output closes, those an explanation closes, and those closed by neither' - 'Searches for the one-step change at which the decision flips to count whether an explanation can actually be produced for a given subscriber' - 'Measures the number of closed questions separately in thirteen subgroups and writes the gap between the groups in units of questions' --- # The Need for Interpretability Counting how many of the twelve questions that can be asked about a decision are closed by the model's output, how many by an explanation, and how many by neither: the output alone closes 4 questions, the explanation raises the average to 8.58, and three questions are closed by no explanation. The one-step change at which the decision flips exists for 159 subscribers of the test set and for 2 of the thirty-two subscribers in the west region; thirteen subgroups were examined, and among the twelve that pass the threshold the number of closed questions ranges from 8.89 at best to 7.88 at worst, a gap of 1.02 questions. The previous topic measured in five lessons **what** the number is and **for whom** it is: the share the split inflated, the operating point the cost matrix moved, the choice of baseline, the margin of the difference between two models, and distribution shift. In all five, what was measured was the model's **output**. What the model **based** that output on was never asked. This question is usually presented as a matter of preference and is generally backed by a list of requirements. No list of requirements is written here; what is measured is a **function**: that the rationale for a decision made about a subscriber can be **documented**. Documentability comes down to a concrete number. There is a set of questions that can be asked about a decision; how many of that set are answered by looking at the model's output, how many by looking at an explanation, and how many by looking at neither. That is the measure of this lesson, and it is read separately in the subgroups. - **IN1.** The set, the label, the seed and the split are the **fictional** setup used throughout the first seven courses of the Machine Learning and Data Science curriculum: the subscriber table derived from the regional measurement network, the label for suspected illicit use or a fault, seed 20260218, split 756/252/252, baseline 0.7579 on the test set. - **IN2.** The model to be explained is the linear model chosen in the Supervised Learning course: twelve scaled columns, learning rate 0.3, 300 epochs, 0.8016 on the test set. In this topic the model is **not chosen but taken as given**; what is measured is not the model's performance but the readability of its decision. - **IN3.** The subgroups are fixed throughout this topic and follow the four axes set up in the Evaluation Design topic: five regions, three tariff tiers (10, 25 and 40 cubic meters), three household size ranges, and two reading levels (a subscriber with readings in all three periods and one with fewer). **The number of subgroups examined is 13.** - **IN4.** A subgroup's number is read only if it has at least 25 records in the test set. A group that falls below the threshold enters the table, but its number does not count as a finding. - **IN5.** A counterfactual is a **one-step** change in a single column. The step is one standard deviation in the training set for numeric columns, a flip of the value for two-valued columns, and a move to another region for the region. These three steps are not the same size, and that is what makes the comparison hard. - **IN6.** Consumption per person is derived from mean consumption and household size; when its source is changed it is recomputed, and it is not changed on its own. - **IN7.** A question counts as closed if its answer can be read **mechanically** from the model's output or from an explanation. A question that requires expert judgment, field verification or a policy decision is open. ## Questions That Can Be Asked About a Decision When the subscriber in the setup is flagged as suspicious, the questions of the person who will listen to them on the phone, of the team that will go to the field, and of the person who will later audit the decision are not the same. The twelve questions are the union of the questions of these three people. What the model's output closes is four: did the subscriber receive the suspicious flag, what is the score and how close is it to the threshold, at what threshold is the decision made, and when the same row is submitted again does the same flag come out. What an explanation can close is five: which column pushes the score most, in which direction does that column push, which single column's one-step change flips the flag, do the other subscribers who receive the same flag rely on the same column, and in this subscriber's subgroup is the flag rate different from the overall one. What neither closes is three: is the flag correct, does the suspicion come from illicit use or from a fault, and what should be done after this flag. The first requires field verification, the second is already not separated in the setup's label, and the third is a policy decision. ```python # interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed; # the twelve-column feature set and the split stay fixed throughout the course. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def scale(train, fields): return ({a: statistics.fmean(x[a] for x in train) for a in fields}, {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields}) def train_model(train, fields, o, s, rate=0.3, epoch=300): # MODEL: logistic regression X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train] Y = [x["suspicious"] for x in train] w = [0.0] * len(X[0]) for _ in range(epoch): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= rate * g[j] / len(X) return w def score(w, fields, o, s, x): return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a] for j, a in enumerate(fields))) def decision(w, fields, o, s, x): return int(score(w, fields, o, s, x) > 0.5) O, S = scale(TRAIN, FIELDS) W = train_model(TRAIN, FIELDS, O, S) ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}") print(f"baseline {BASELINE:.4f}, test accuracy of the linear model {ACCURACY:.4f}") ``` ``` train 756, validation 252, test 252 baseline 0.7579, test accuracy of the linear model 0.8016 ``` ## The One Step at Which the Decision Flips Three of the five explanation questions close for every subscriber, because the linear model's score is the sum of one term per column, and those terms can always be read. The fourth — **which single column's one-step change flips the flag** — may not close. For a subscriber who stands far enough from the decision boundary, no one-step change flips the flag, and that question goes unanswered. This is not a shortcoming of the explanation method but the **certainty** of that subscriber's decision; but the result leads to the same place: that row stays empty in the rationale document. ```python def tariff_tier(x): return ("tariff 10" if x["mean_consumption"] <= 10 else "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40") def household_group(x): return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4 else "household 5+") def reading_group(x): return "readings few" if x["period"] <= 2 else "readings all" def equals(f, value): return lambda x: f(x) == value SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION] + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")] + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")] + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")]) MIN_SIZE, SIZES = 25, {} for name, f in SUBGROUPS: SIZES[name] = sum(1 for x in TEST if f(x)) SHORT = [name for name in SIZES if SIZES[name] < MIN_SIZE] print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, " f"below the threshold {SHORT}") SINGLE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "region_name"] def change(x, a, v): d = dict(x) d[a] = v if a in ("mean_consumption", "household"): d["per_person"] = round(d["mean_consumption"] / d["household"], 3) if a == "region_name": for b in REGION: d["area_" + b[0]] = int(v == b[0]) return d def counterfactual(x): # one-step change in a single column k = decision(W, FIELDS, O, S, x) for a in SINGLE: if a == "zero_readings": candidates = [1 - x[a]] elif a == "region_name": candidates = [b[0] for b in REGION if b[0] != x["region_name"]] else: candidates = [x[a] + S[a], x[a] - S[a]] for v in candidates: if decision(W, FIELDS, O, S, change(x, a, v)) != k: return a return None COUNTERFACTUALS = [counterfactual(x) for x in TEST] EXAMPLE = TEST[4] print(f"\nexample subscriber: region {EXAMPLE['region_name']}, household {EXAMPLE['household']}, " f"readings {EXAMPLE['period']}, mean consumption {EXAMPLE['mean_consumption']}") print(f" score {score(W, FIELDS, O, S, EXAMPLE):.4f}, decision " f"{decision(W, FIELDS, O, S, EXAMPLE)}, counterfactual column {COUNTERFACTUALS[4]}") print(f"subscribers with a counterfactual in the whole test set " f"{sum(1 for k in COUNTERFACTUALS if k)}/{len(TEST)} " f"({sum(1 for k in COUNTERFACTUALS if k) / len(TEST):.4f})") ``` ``` subgroup count 13, threshold 25 records, below the threshold ['tariff 10'] example subscriber: region central, household 4, readings 2, mean consumption 34.16 score 0.4768, decision 0, counterfactual column mean_consumption subscribers with a counterfactual in the whole test set 159/252 (0.6310) ``` The example subscriber's score of 0.4768 is just below the threshold, and when mean consumption rises by one standard deviation the flag flips. For this subscriber, the row in the rationale document can be written. For 159 of the test set's 252 subscribers such a row can be written; for the remaining 93 subscribers, no one-step change flips the flag. ## Number of Closed Questions in the Subgroups The 0.6310 measured overall is not read on its own. The same number is measured in thirteen subgroups separately and converted into a number of closed questions: for every subscriber, four questions are closed by the output, three by the explanation that can always be produced, the fourth explanation question closes if a counterfactual exists, and the fifth closes if all of the subscriber's subgroups are above the threshold size. ```python FIXED = 4 + 3 # output closes four, three explanation questions always close def closed(i, x): readable = all(SIZES[name] >= MIN_SIZE for name, f in SUBGROUPS if f(x)) return FIXED + (1 if COUNTERFACTUALS[i] else 0) + (1 if readable else 0) ALL = sum(closed(i, x) for i, x in enumerate(TEST)) / len(TEST) print(f"{'subgroup':<16}{'records':>8}{'counterfactual rate':>21}{'closed questions':>18}") ROWS = [] for name, f in SUBGROUPS: g = [(i, x) for i, x in enumerate(TEST) if f(x)] cf_rate = sum(1 for i, x in g if COUNTERFACTUALS[i]) / len(g) closed_mean = sum(closed(i, x) for i, x in g) / len(g) ROWS.append((name, closed_mean)) print(f"{name:<16}{len(g):>8}{cf_rate:>21.4f}{closed_mean:>18.2f}") lowest = min(ROWS, key=lambda t: t[1]) highest = max(ROWS, key=lambda t: t[1]) print(f"\nclosed questions overall {ALL:.2f}/12; output alone 4/12; " f"closed by no explanation 3/12") print(f"best subgroup {highest[0]} {highest[1]:.2f}, worst subgroup {lowest[0]} " f"{lowest[1]:.2f}, gap {highest[1] - lowest[1]:.2f} questions") ``` ``` subgroup records counterfactual rate closed questions region north 68 0.8235 8.82 region south 58 0.3793 8.31 region east 36 0.8333 8.81 region west 32 0.0625 7.88 region central 58 0.8448 8.83 tariff 10 12 0.2500 7.25 tariff 25 193 0.5907 8.59 tariff 40 47 0.8936 8.89 household 1-2 73 0.5753 8.49 household 3-4 143 0.6503 8.62 household 5+ 36 0.6667 8.64 readings few 85 0.5647 8.49 readings all 167 0.6647 8.63 closed questions overall 8.58/12; output alone 4/12; closed by no explanation 3/12 best subgroup tariff 40 8.89, worst subgroup tariff 10 7.25, gap 1.64 questions ``` Overall, the explanation raises the number of closed questions from 4 to 8.58. In the subgroups, the same number is spread between 7.88 and 8.89 across the twelve groups that pass the threshold, and the gap is 1.02 questions; if tariff 10, which falls below the threshold, is also counted, the lower end drops to 7.25. The counterfactual rate, on the other hand, separates far more sharply: 0.8448 in the central region, 0.0625 in the west region. In the west region, the one step at which the decision flips can be shown for two of the thirty-two subscribers. The reason can be read in the setup: the west's base consumption of 14 cubic meters is the lowest of the five regions, so the subscribers there stand far from the decision boundary and no movement of one standard deviation takes them to the other side. Whether the decision changes is read here. Someone looking at the overall number writes the sentence "the rationale for the decisions can be documented." Someone looking at the subgroup cannot write the same sentence for the west region; in the rationale document of the subscribers there, the counterfactual row is empty. The number of the lowest tariff tier is not even read, because it has 12 records in the test set and the threshold is 25. The number of subgroups examined is 13, and the number read is 12. That three questions are closed by no explanation also sits in the same table, and it is the most important row for this lesson. An explanation tells **how** the decision was produced; it does not tell whether the decision is **correct**, what the **source** of the suspicion is, or **what to do next**. Closing these three questions takes not an explanation but field data, a finer label, and a written policy. ## Summary - Of the twelve questions that can be asked about a decision, 4 can be closed by the model's output, 5 by an explanation, and 3 by neither; the three that stay open are correctness, the source of the suspicion, and what to do next. - Overall, the average number of closed questions is 8.58/12; without an explanation this number stays at 4/12. - The one-step change at which the decision flips exists for 159 subscribers of the test set, a rate of 0.6310. - Thirteen subgroups were examined, and the 12 that pass the 25-record threshold were read; among the groups read, the number of closed questions ranges from 7.88 to 8.89, and the gap is 1.02 questions. - The counterfactual rate is 0.8448 in the central region and 0.0625 in the west region; because the west has the lowest base consumption, the subscribers there stand far from the decision boundary. ## Next Step In this lesson, three of the five explanation questions were passed over as "always closes," and the reason was that the linear model's score can be decomposed into one term per column. The ranking of those terms is read as the **global** answer to which column the model relies on, and the Data Preparation and Feature Engineering course and the Supervised Learning course had already measured that ranking. But a ranking is not produced in only one way. Measuring a column by **permuting** it instead of reading it from the model — shuffling its values among themselves and looking at how much performance drops — gives a second ranking, and there is no reason for the two rankings to come out the same. The next lesson puts the two rankings side by side, counts in how many ranks they disagree, and measures how mean consumption and consumption per person, two forms of the same quantity, split the importance.