--- title: 'Statistical Significance' source: 'https://academia.sh/en/courses/model-ethics-and-interpretation/statistical-significance' course: 'Model Evaluation, Interpretability and Ethics' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures the difference between two models as a paired comparison on the same samples and compares it with the independent measurement: prints the widths of the two margins side by side' - 'Sweeps the held-out set size and computes how the margin narrows with set size' - 'Counts how many of the differences measured in the subgroups have a margin that excludes zero and shows that a contribution that looks large can be unmeasurable' --- # Statistical Significance Measuring the margin of the difference between two models and counting how much narrower a margin the paired comparison gives than the independent comparison: the margins of all the differences between the depth-6 decision tree and its four rivals contain zero, and only the 0.5357 difference stays outside. On the same 252 subscribers the paired margin runs from 0.0476 to 0.0794 and the independent margin from 0.1190 to 0.1310; the more the two models agree, the narrower the margin. As the set grows from 63 to 504, the paired margin falls from 0.1111 to 0.0556. Among the subgroups, the margin excludes zero for only four of thirteen differences, and +0.1389, one of the largest subgroup contributions of the course, is not one of them. Over three lessons fourteen separate number pairs were compared, and in all of them the difference was read directly: +0.0198 in one place, −0.0119 in another, +0.1702 in one subgroup. What was never asked in any of them is whether these differences can be told apart from **the noise of a set of 252 subscribers**. A contribution of 0.1389 read from thirty-six subscribers could vanish had the class of a few subscribers flipped. Hypothesis testing, permutation testing, the bootstrap and the margin of variation were measured in the Statistical Analysis and Exploratory Data Analysis course; the testing theory is not repeated here, and the same tool is applied only to model performance. This lesson measures three things: the margin of the difference between two models, how much measuring on the same samples (the **paired comparison**) narrows that margin, and how the size of the held-out set determines the margin. - **ED23.** The setup, the split (**756/252/252**), the seed, the metric (**accuracy**) and the subgroup definitions carry over unchanged from the earlier lessons: four axes, **13 subgroups**. - **ED24.** The model being compared is the **depth-6 decision tree** (0.7778 on the test set). The rivals: the most frequent class, the depth-2 tree, the depth-8 tree, the region and tier lookup table, and calling everyone suspicious. All of them were built in the earlier lessons. - **ED25.** The margin is computed by **bootstrap**: **400 repeats**, the 5% and 95% points. The method was measured in the Statistical Analysis and Exploratory Data Analysis course and is not justified here. - **ED26.** In the **paired comparison** the quantity that is resampled is the per-subscriber **difference** indicator: the same subscriber is right or wrong under both models, and the shared margin of error drops out. - **ED27.** In the **independent comparison** the numbers of the two models are resampled separately; this corresponds to the case where the two models were measured on two separate sets. - **ED28.** A margin that contains zero does not mean "the two models are the same"; it means **this difference could not be measured on this set**. A shortfall in measurement and equality are separate things. - **ED29.** In the set size sweep, the 504 row is built by combining the validation and test sets and exists **only to show how the margin narrows with size**; it is not a reportable performance number, because the validation set has been looked at before. - **ED30.** Subgroup margins are computed by the same procedure. In a small subgroup the margin is wide, and this is not a flaw but a direct consequence of sample size. ## The Difference Between Two Models and Its Share The first block builds the setup, compares five rivals with the tree on the same test set and prints two margins for each difference. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) MAJORITY = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN)) M, DEPTH2, DEPTH8 = tree(TRAIN, 6, FIELDS), tree(TRAIN, 2, FIELDS), tree(TRAIN, 8, FIELDS) REPEATS = 400 counts = {} for x in TRAIN: # region and tier lookup table baseline, carried over from 03 k = (x["region_name"], dict(subgroups(x))["tariff"]) counts.setdefault(k, [0, 0])[x["suspicious"]] += 1 votes = {k: int(v[1] > v[0]) for k, v in counts.items()} LOOKUP = (lambda x: votes.get((x["region_name"], dict(subgroups(x))["tariff"]), MAJORITY)) def paired_compare(rows, f, seed): # bootstrap was measured in the Statistical Analysis and Exploratory Data Analysis course; # here it is applied only to the DIFFERENCE hits_m = [int(predict(M, x) == x["suspicious"]) for x in rows] hits_f = [int(f(x) == x["suspicious"]) for x in rows] diff, n, paired, independent = [a - b for a, b in zip(hits_m, hits_f)], len(rows), [], [] for j in range(REPEATS): r, r1, r2 = rng(seed + j), rng(seed + 5000 + j), rng(seed + 9000 + j) paired.append(sum(diff[int(r() * n)] for _ in range(n)) / n) independent.append(sum(hits_m[int(r1() * n)] for _ in range(n)) / n - sum(hits_f[int(r2() * n)] for _ in range(n)) / n) paired.sort() independent.sort() low, high = int(0.05 * REPEATS), int(0.95 * REPEATS) - 1 return (sum(diff) / n, paired[low], paired[high], independent[low], independent[high], sum(1 for v in diff if v)) RIVALS = (("most frequent class", lambda x: MAJORITY), ("depth 2", lambda x: predict(DEPTH2, x)), ("depth 8", lambda x: predict(DEPTH8, x)), ("lookup table", LOOKUP), ("all suspicious", lambda x: 1)) print(f"{'rival':<19} {'disagree':>8} {'diff':>8} | {'paired':>21} {'width':>7}" f" | {'independent':>21} {'width':>7} | zero") for name, f in RIVALS: d, lo, hi, ilo, ihi, u = paired_compare(TEST, f, 71000) print(f"{name:<19} {u:>8} {d:>+8.4f} | [{lo:>+8.4f}, {hi:>+8.4f}] {hi - lo:>7.4f}" f" | [{ilo:>+8.4f}, {ihi:>+8.4f}] {ihi - ilo:>7.4f} | " f"{'outside' if lo > 0 or hi < 0 else 'inside'}") ``` ``` rival disagree diff | paired width | independent width | zero most frequent class 37 +0.0198 | [ -0.0198, +0.0556] 0.0754 | [ -0.0516, +0.0794] 0.1310 | inside depth 2 24 -0.0079 | [ -0.0437, +0.0238] 0.0675 | [ -0.0714, +0.0516] 0.1230 | inside depth 8 12 +0.0159 | [ -0.0079, +0.0397] 0.0476 | [ -0.0437, +0.0754] 0.1190 | inside lookup table 33 -0.0119 | [ -0.0556, +0.0238] 0.0794 | [ -0.0754, +0.0476] 0.1230 | inside all suspicious 215 +0.5357 | [ +0.4603, +0.6190] 0.1587 | [ +0.4683, +0.5992] 0.1310 | outside ``` The last column is the first result of the lesson. In **four** of the five comparisons the margin contains zero. The model's 0.0198 lead over the most frequent class, its 0.0119 loss to the lookup table, the 0.0079 between it and depth 2, the 0.0159 between it and depth 8 — all four cannot be measured on this set of 252 subscribers. Only the "call everyone suspicious" difference stays outside the margin, at 0.5357. The numbers in the tables of the previous three lessons were not computed wrongly; they are **below the measurement threshold**, and these are two separate things. The second result is the comparison of the two margin columns. The paired margin is narrower in every row: 0.0754 against 0.1310, 0.0675 against 0.1230, 0.0476 against 0.1190. The reason stands in the disagreement column. The depth-8 tree and the depth-6 tree part ways on only **12** of the 252 subscribers; on the remaining 240 both give the same answer, and the difficulty of those subscribers drops out of the computation of the difference entirely. In the independent measurement it does not drop out, because there the two models are assumed to have been measured on **separate sets** and each set's own variability of difficulty enters the margin. Measuring the two models on the same samples narrows the margin by **up to two and a half times**, without collecting any data. The last row reads in the opposite direction: because "call everyone suspicious" and the tree part ways on 215 subscribers, the paired margin there is **wider** than the independent one. Pairing does not always narrow; the more the two models agree, the more it narrows. ## The Set Size Determines the Share The width of the margin comes not from the model but from the set. The same comparison is repeated at different sizes taken from the head of the set. ```python POOL = VAL + TEST print(f"{'size':>6} | {'diff':>8} {'paired':>21} {'width':>7} | {'independent width':>17}") for n in (63, 126, 189, 252, 504): d, lo, hi, ilo, ihi, u = paired_compare(POOL[:n] if n > 252 else TEST[:n], LOOKUP, 74000) print(f"{n:>6} | {d:>+8.4f} [{lo:>+8.4f}, {hi:>+8.4f}] {hi - lo:>7.4f} | " f"{ihi - ilo:>17.4f}") ``` ``` size | diff paired width | independent width 63 | +0.0000 [ -0.0635, +0.0476] 0.1111 | 0.2540 126 | +0.0079 [ -0.0397, +0.0556] 0.0952 | 0.1667 189 | -0.0106 [ -0.0529, +0.0317] 0.0847 | 0.1376 252 | -0.0119 [ -0.0476, +0.0278] 0.0754 | 0.1270 504 | -0.0159 [ -0.0437, +0.0119] 0.0556 | 0.0794 ``` The difference column is unstable: 0.0000 from 63 subscribers, +0.0079 from 126, −0.0159 from 504. Even the sign changes. The margin column, on the other hand, narrows steadily — from 0.1111 to 0.0556, which is to say that as the set grows eightfold the margin is halved. This means the smallest measurable difference depends on the set size: on 252 subscribers no difference below 0.04 can be read, and most of the model differences reported throughout the Machine Learning and Data Science curriculum are below that threshold. The last row carries a warning. The set of 504 is the union of the validation and test sets, and the validation set was already looked at while the model was being chosen; the −0.0159 there is not a reportable performance number. The row stands only to show how the margin narrows with size. The size of the held-out set is therefore not a split ratio but a consequence of **the smallest difference one wants to measure**. ## In the Subgroups the Difference Disappears Inside the Share The subgroup gap was measured over three lessons. Now the margin of the difference in each subgroup is computed. ```python print(f"{'subgroup':<14} {'n':>4} {'disagree':>8} {'diff':>8} " f"{'paired margin':>22} {'zero':>8}") outside = 0 for g in SUBGROUPS: rows = [x for x in TEST if g in subgroups(x)] d, lo, hi, ilo, ihi, u = paired_compare(rows, lambda x: MAJORITY, 76000) outside += lo > 0 or hi < 0 print(f"{' '.join(g):<14} {len(rows):>4} {u:>8} {d:>+8.4f} [{lo:>+8.4f}, {hi:>+8.4f}]" f" {'outside' if lo > 0 or hi < 0 else 'inside':>8}") print(f"\nsubgroups whose margin excludes zero: {outside}/{len(SUBGROUPS)}") ``` ``` subgroup n disagree diff paired margin zero household 1-2 73 9 +0.0411 [ -0.0274, +0.1096] inside household 3-4 143 21 +0.0070 [ -0.0420, +0.0629] inside household 5+ 36 7 +0.0278 [ -0.0833, +0.1389] inside reading few 85 4 -0.0471 [ -0.0824, -0.0118] outside reading full 167 33 +0.0539 [ +0.0060, +0.1018] outside region central 58 7 +0.0172 [ -0.0690, +0.1034] inside region east 36 19 +0.1389 [ -0.0556, +0.3611] inside region north 68 7 -0.0441 [ -0.1176, +0.0147] inside region south 58 3 +0.0517 [ +0.0172, +0.1034] outside region west 32 1 -0.0312 [ -0.0938, +0.0000] inside tariff 10 12 1 -0.0833 [ -0.2500, +0.0000] inside tariff 25 193 22 -0.0104 [ -0.0466, +0.0311] inside tariff 40 47 14 +0.1702 [ +0.0426, +0.2766] outside subgroups whose margin excludes zero: 4/13 ``` The east region row is the harshest result of this lesson. The model's contribution there is **+0.1389**, and this was one of the largest subgroup contributions the course had measured so far. Its margin is **[−0.0556, +0.3611]** and contains zero. Nineteen disagreements among thirty-six subscribers spread the difference over an interval a third of a point wide. The same number was read in two lessons as "what the model adds in the east"; here it turns out that it cannot be read that way. Four subgroups leave zero outside, and one of them is in the opposite direction: among the 85 subscribers with few readings the difference is **−0.0471** and its margin is **[−0.0824, −0.0118]**. In that subgroup the model is measurably **worse** than stating the most frequent class. The remaining three are in the forward direction: south +0.0517, full readings +0.0539, the tariff 40 tier +0.1702. Two rules follow from this. The subgroup gap is the result of a design decision and **the gap itself can be measured**, but the individual differences that make it up cannot be measured in most subgroups. The second is that looking at thirteen subgroups and choosing the largest difference among them reproduces the multiple-comparison share counted in the Statistical Analysis and Exploratory Data Analysis course: stepping outside the margin in one of thirteen independent tests is not a finding unless it is written that thirteen looks were taken. ## Summary - In four of the five comparisons the margin contains zero: the model's 0.0198 lead over the most frequent class, its 0.0119 loss to the lookup table and the two depth differences **cannot be measured** on 252 subscribers. A margin that contains zero means a shortfall in measurement, not equality. - The paired comparison gives a narrower margin than the independent one: 0.0754 against 0.1310, 0.0476 against 0.1190. The narrowing comes from the agreement rate of the two models, and for a pair that parts ways on 215 subscribers pairing **widens** the margin. - The margin narrows with set size: 0.1111 at 63 subscribers, 0.0556 at 504. The size of the held-out set is not a split ratio but a consequence of the smallest difference one wants to measure. - Among the subgroups, the margin excludes zero for only **4** of the 13 differences. The +0.1389 in the east region, one of the largest subgroup contributions of the course, is not one of them: its margin is [−0.0556, +0.3611]. - Of the four differences that leave zero outside, one is in the opposite direction: among the subscribers with few readings the model is measurably below the baseline, at **−0.0471**. ## Next Step Over four lessons the measurement itself was designed: according to whom the set was split, from which cost the threshold was read, how the baseline was chosen, with which margin the difference was stated. The assumption they all share is that the test set comes from **the same distribution** as the data the model will meet. A measurement network changes over time: new regions join, tariff tiers shift, the rate of zero readings falls as meters are replaced. The next lesson measures that assumption: it reduces the difference between training and production data to a single number, places that number next to the drop in performance, and separates **covariate shift** from **label shift** to show why the two call for different solutions.