--- title: 'Metric Selection' source: 'https://academia.sh/en/courses/model-ethics-and-interpretation/metric-selection' course: 'Model Evaluation, Interpretability and Ethics' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Shows by sweeping ten candidates that the same model produces two separate thresholds and two separate decisions under two cost matrices' - 'Computes that the cost matrix changes the baseline too: in one structure flagging nobody, in the other flagging everybody is the baseline' - 'Counts how different a cost a single operating point produces across subgroups and measures the selection share of a subgroup-specific threshold' --- # Metric Selection The same model taking two separate operating points and making two separate decisions under two cost matrices: in the structure where inspection is costly the selected threshold is 0.95 and 17 of 252 subscribers are flagged, in the structure where illicit use is costly the threshold is 0.25 and 77 are flagged; the decision of 60 subscribers flips between the two structures. The baseline moves with the cost as well — flagging nobody is 0.2421 in one, flagging everybody is 0.7579 in the other. A single threshold spreads the gain across subgroups from +0.4138 to -0.4468, the gap is 0.8606 and in four of twelve subgroups the model stays below its own baseline; choosing a subgroup-specific threshold lowers the test cost in only four of them. The previous lesson took the metric to be accuracy and never questioned it. Accuracy counts the two error types with the same weight: sending a subscriber who is not suspicious to inspection and missing a suspicious subscriber are equally wrong. In operations they are not equal. An inspection crew's travel, its time and the mark a wasted visit leaves on the subscriber are one cost; an illicit use that slipped through or a fault left unrepaired is another cost, and the ratio of the two is a business decision. This lesson writes that ratio as a **cost matrix** and measures one thing: the same model, the same scores, two separate cost matrices — how many separate operating points, how many separate decisions. Metric definitions, the threshold and the curves were set up in the Evaluation and Tuning topic of the Supervised Learning course and are not repeated here. The only thing added is reading the threshold from the **business objective**. The subgroups and the triple carry over unchanged from the previous lesson. - **ED8.** The setup, the split (**756/252/252**), the seed and the subgroup definitions carry over unchanged from the previous lesson: four axes, **13 subgroups**, an n written next to every number, and **a group whose n is below 25 does not enter the gap**. - **ED9.** The model is the same depth-6 decision tree, with one difference: the leaf holds not the class but the **suspicious share**. This share is the model's score; a subscriber above the threshold is sent to inspection. - **ED10.** The metric is the **cost per subscriber** and **small is good**. A false alarm (a flagged subscriber who is not suspicious) costs $a$ units, a miss (an unflagged subscriber who is suspicious) costs $m$ units; correct decisions cost zero. The unit is abstract and is not money. - **ED11.** Two cost matrices are tested. **Inspection costly**: $a = 2$, $m = 1$. **Illicit use costly**: $a = 1$, $m = 4$. Both are fictional and stand for a business decision. - **ED12.** The threshold is swept over **10 candidates** (from 0.05 to 0.95 in steps of one tenth) and is selected on the **validation set**; the test set is read only for the report. - **ED13.** The baseline is computed separately for each cost matrix and is **the cheaper one** of two procedures that learn nothing: flagging nobody, or flagging everybody. - **ED14.** A subgroup-specific threshold is selected on that subgroup's **validation** rows and reported on that subgroup's test rows. Subgroup validation sets are small, and the selection share comes from there. - **ED15.** Accuracy is never printed in this lesson. The 0.7778 of the previous lesson and the costs of this lesson are not on the same axis and are not compared. ## Two Cost Structures, Two Thresholds The first block sets up the scenario, trains the tree so that it leaves a share at the leaf, and sweeps ten thresholds under the two cost matrices. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): # the leaf holds the SHARE, not the class p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def score(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from lesson 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] def cost(rows, t, fa, miss): # cost per subscriber; small is good total = 0 for x in rows: total += fa * (1 - x["suspicious"]) if score(M, x) >= t else miss * x["suspicious"] return total / len(rows) def flagged(rows, t): return sum(score(M, x) >= t for x in rows) / len(rows) SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M = tree(TRAIN, 6, FIELDS) THRESHOLDS = [0.05 + 0.10 * i for i in range(10)] STRUCTURES = (("inspection costly", 2, 1), ("illicit use costly", 1, 4)) print(f"{'threshold':>9} | {'A val':>7} {'A test':>7} | {'B val':>7} {'B test':>7} " f"| {'flagged':>7}") for t in THRESHOLDS: print(f"{t:>9.2f} | {cost(VAL, t, 2, 1):>7.4f} {cost(TEST, t, 2, 1):>7.4f} " f"| {cost(VAL, t, 1, 4):>7.4f} {cost(TEST, t, 1, 4):>7.4f} " f"| {flagged(TEST, t):>7.4f}") CHOSEN = {} for name, fa, miss in STRUCTURES: t = CHOSEN[name] = min(THRESHOLDS, key=lambda c: (cost(VAL, c, fa, miss), c)) flag_none, flag_all = cost(TEST, 1.1, fa, miss), cost(TEST, -0.1, fa, miss) print(f"\n{name:<18} threshold {t:.2f} test cost {cost(TEST, t, fa, miss):.4f} baseline " f"{min(flag_none, flag_all):.4f} ({'none' if flag_none < flag_all else 'all'}) gain " f"{min(flag_none, flag_all) - cost(TEST, t, fa, miss):+.4f} flagged " f"{sum(score(M, x) >= t for x in TEST)}/{len(TEST)}") ``` ``` threshold | A val A test | B val B test | flagged 0.05 | 1.0952 0.9762 | 0.6032 0.5437 | 0.7063 0.15 | 0.7659 0.7659 | 0.4802 0.5913 | 0.5357 0.25 | 0.4802 0.4246 | 0.4762 0.5595 | 0.3056 0.35 | 0.3889 0.3214 | 0.5278 0.5913 | 0.2183 0.45 | 0.3492 0.2976 | 0.5079 0.6349 | 0.1825 0.55 | 0.2778 0.2579 | 0.5833 0.7262 | 0.1151 0.65 | 0.2540 0.2460 | 0.6548 0.7341 | 0.1032 0.75 | 0.2222 0.2302 | 0.6667 0.7540 | 0.0833 0.85 | 0.2222 0.2302 | 0.6667 0.7540 | 0.0833 0.95 | 0.2183 0.2222 | 0.6786 0.7778 | 0.0675 inspection costly threshold 0.95 test cost 0.2222 baseline 0.2421 (none) gain +0.0198 flagged 17/252 illicit use costly threshold 0.25 test cost 0.5595 baseline 0.7579 (all) gain +0.1984 flagged 77/252 ``` The two cost columns fall in opposite directions. In the structure where inspection is costly, the cost drops as the threshold rises and the cheapest point is **0.95**; in the structure where illicit use is costly the cheapest point is **0.25**, after which the cost starts to climb. The model is a single model, the scores are a single set of scores, and the two structures pick two separate rows from the same table. In one case **17** of 252 subscribers go to inspection, in the other **77**. The baseline moves with the cost as well. When inspection is costly, the cheapest procedure that learns nothing is **flagging nobody**, and it costs 0.2421; when illicit use is costly it is **flagging everybody**, and it costs 0.7579. Both are computed on the same set with the same labels. The model's gain changes accordingly: in the first structure it can drop only **0.0198** below the baseline, in the second **0.1984**. The same model, the same test set — and one is reported as "helps little", the other as "helps clearly". ## Who the Decision Changes For The difference between the two operating points is two rows in a table, but on the subscriber side it is something concrete: some subscribers go to inspection under one structure and not under the other. ```python tA, tB = CHOSEN["inspection costly"], CHOSEN["illicit use costly"] changed = [x for x in TEST if (score(M, x) >= tA) != (score(M, x) >= tB)] shares = sorted(((sum(g in subgroups(x) for x in changed) / sum(g in subgroups(x) for x in TEST), g) for g in SUBGROUPS), reverse=True) print(f"decision changed for {len(changed)}/{len(TEST)} subscribers, {sum(x['suspicious'] for x in changed)}" f" of them suspicious; share by subgroup:") for o, g in shares[:2] + shares[-1:]: print(f" {' '.join(g):<14} {o:.4f}") ``` ``` decision changed for 60/252 subscribers, 23 of them suspicious; share by subgroup: tariff 10 0.7500 region east 0.5556 region west 0.0938 ``` The decision of sixty subscribers flips, and 23 of them are truly suspicious. Choosing the cost structure is not a metric preference; it is whether sixty households get a knock on the door or not. The distribution is not even either: **0.5556** of the subscribers in the east region move between the two structures, while in the west the share is **0.0938**. Once a cost matrix is chosen, the subgroups do not carry its load equally. ## The Cost Is Not the Same Across Subgroups The single threshold selected in the structure where illicit use is costly is 0.25, and it is applied to all thirteen subgroups in the same way. The block below prints, side by side, the cost of that single threshold in each subgroup, that subgroup's own baseline, and what would happen if the subgroup selected its own threshold. ```python fa, miss = 1, 4 print(f"{'subgroup':<14} {'n':>4} {'flagged':>7} {'cost':>7} {'baseline':>8} {'gain':>8}" f" | {'own':>5} {'cost':>7} {'extra':>8}") gains, extras = [], [] for g in SUBGROUPS: val_g = [x for x in VAL if g in subgroups(x)] test_g = [x for x in TEST if g in subgroups(x)] c, base = cost(test_g, tB, fa, miss), min(cost(test_g, 1.1, fa, miss), cost(test_g, -0.1, fa, miss)) own_t = min(THRESHOLDS, key=lambda t: (cost(val_g, t, fa, miss), t)) own_cost = cost(test_g, own_t, fa, miss) if len(test_g) >= 25: gains.append(base - c) extras.append(c - own_cost) print(f"{' '.join(g):<14} {len(test_g):>4} {flagged(test_g, tB):>7.4f} {c:>7.4f} {base:>8.4f} " f"{base - c:>+8.4f} | {own_t:>5.2f} {own_cost:>7.4f} {c - own_cost:>+8.4f}" f"{' n<25' if len(test_g) < 25 else ''}") print(f"\ngain gap ({len(gains)} subgroups): best {max(gains):+.4f}, worst " f"{min(gains):+.4f}, gap {max(gains) - min(gains):.4f}; below baseline " f"{sum(1 for v in gains if v < 0)}/{len(gains)}") print(f"subgroups whose own threshold lowers the test cost: {sum(1 for v in extras if v > 0)}" f"/{len(extras)}, unchanged {sum(1 for v in extras if v == 0)}, raised " f"{sum(1 for v in extras if v < 0)}") ``` ``` subgroup n flagged cost baseline gain | own cost extra household 1-2 73 0.3562 0.5205 0.7534 +0.2329 | 0.15 0.6301 -0.1096 household 3-4 143 0.2657 0.5944 0.7692 +0.1748 | 0.25 0.5944 +0.0000 household 5+ 36 0.3611 0.5000 0.7222 +0.2222 | 0.55 0.7500 -0.2500 reading few 85 0.2706 0.7059 0.7882 +0.0824 | 0.45 0.8471 -0.1412 reading full 167 0.3234 0.4850 0.7425 +0.2575 | 0.15 0.5509 -0.0659 region central 58 0.3276 0.8448 0.6552 -0.1897 | 0.25 0.8448 +0.0000 region east 36 0.7500 0.7778 0.4722 -0.3056 | 0.45 0.7222 +0.0556 region north 68 0.2500 0.6618 0.7059 +0.0441 | 0.15 0.6765 -0.0147 region south 58 0.1897 0.2759 0.6897 +0.4138 | 0.15 0.5517 -0.2759 region west 32 0.0938 0.0938 0.0000 -0.0938 | 0.35 0.0625 +0.0312 tariff 10 12 0.7500 0.4167 0.6667 +0.2500 | 0.05 0.4167 +0.0000 n<25 tariff 25 193 0.1813 0.4870 0.6425 +0.1554 | 0.45 0.4560 +0.0311 tariff 40 47 0.7021 0.8936 0.4468 -0.4468 | 0.05 0.5957 +0.2979 gain gap (12 subgroups): best +0.4138, worst -0.4468, gap 0.8606; below baseline 4/12 subgroups whose own threshold lowers the test cost: 4/12, unchanged 2, raised 6 ``` The gain, 0.1984 overall, spreads across the subgroups between **+0.4138 and −0.4468**; the **gap is 0.8606**. In four subgroups the model is **below** its own baseline: west, east, central and the tariff 40 tier. The baseline of the tariff 40 tier is 0.4468 and the model's cost is 0.8936 — in that tier the model is twice as expensive as saying "flag everybody". The reason is in the flagged column: the 0.25 threshold flags 0.7021 of the subscribers in that tier, but because the suspicious share of the tier is 0.553, the suspicious subscribers left among the unflagged pay the four-unit cost of a miss. The last three columns suggest a correction and show the cost of the correction too. When each subgroup selects its own threshold from its own validation rows, the tariff 40 tier drops to 0.05 and its cost falls from 0.8936 to 0.5957. But the same procedure lowers the test cost in only **four** of the twelve subgroups; in two it changes nothing, and in **six it raises it** — by 0.2759 units in the south and 0.2500 in household size 5 and above. The reason is the smallness of the subgroup validation sets: a threshold selected from forty to seventy rows selects the noise of those rows. Tuning a threshold per subgroup requires a validation set per subgroup, and those sets come from the design, not from the split. ## Summary - A cost matrix turns the business objective into a number: the ratio between a false alarm and a miss is a business decision, and the threshold is read from that ratio. The metric is the **cost per subscriber** and small is good. - The same model, the same scores, two cost matrices: when inspection is costly the selected threshold is 0.95 and 17 of 252 subscribers are flagged; when illicit use is costly the threshold is 0.25 and 77 are flagged. **The decision of 60 subscribers flips**, and 23 of them are truly suspicious. - The baseline moves with the cost too: in the first structure flagging nobody (0.2421), in the second flagging everybody (0.7579) is the baseline. The model's gain is accordingly +0.0198 or +0.1984. - A single threshold does not give a single result across subgroups: the gain spreads between +0.4138 and −0.4468, the **gap is 0.8606**, and **in 4 of 12 subgroups the model stays below its own baseline**. - Selecting a subgroup-specific threshold lowers the test cost in only 4 of 12 subgroups and raises it in six; the smaller the subgroup validation set, the larger the selection share. ## Next Step In this lesson the baseline turned into a question of its own for the second time. When the cost structure changed, the baseline changed too: in one, flagging nobody, in the other, flagging everybody. Across the subgroups every group turned out to have its own baseline, which was 0.0000 in one group and 0.7882 in another. Over seven courses the baseline was a **tool** — a fixed mark that makes the model's number readable. Here it was seen not to be fixed. The next lesson takes the baseline up directly: how many baselines a problem has, which one is chosen, which two separate ways the choice lets the same model be read, and what shows when the baselines chosen in the seven courses of this curriculum are collected in a single table.