Statistical Significance

Over three lessons fourteen separate number pairs were compared, and in all of them the difference was read directly: +0.0198 in one place, −0.0119 in another, +0.1702 in one subgroup. What was never asked in any of them is whether these differences can be told apart from the noise of a set of 252 subscribers. A contribution of 0.1389 read from thirty-six subscribers could vanish had the class of a few subscribers flipped.

Hypothesis testing, permutation testing, the bootstrap and the margin of variation were measured in the Statistical Analysis and Exploratory Data Analysis course; the testing theory is not repeated here, and the same tool is applied only to model performance. This lesson measures three things: the margin of the difference between two models, how much measuring on the same samples (the paired comparison) narrows that margin, and how the size of the held-out set determines the margin.

• ED23. The setup, the split (756/252/252), the seed, the metric (accuracy) and the subgroup definitions carry over unchanged from the earlier lessons: four axes, 13 subgroups.
• ED24. The model being compared is the depth-6 decision tree (0.7778 on the test set). The rivals: the most frequent class, the depth-2 tree, the depth-8 tree, the region and tier lookup table, and calling everyone suspicious. All of them were built in the earlier lessons.
• ED25. The margin is computed by bootstrap: 400 repeats, the 5% and 95% points. The method was measured in the Statistical Analysis and Exploratory Data Analysis course and is not justified here.
• ED26. In the paired comparison the quantity that is resampled is the per-subscriber difference indicator: the same subscriber is right or wrong under both models, and the shared margin of error drops out.
• ED27. In the independent comparison the numbers of the two models are resampled separately; this corresponds to the case where the two models were measured on two separate sets.
• ED28. A margin that contains zero does not mean “the two models are the same”; it means this difference could not be measured on this set. A shortfall in measurement and equality are separate things.
• ED29. In the set size sweep, the 504 row is built by combining the validation and test sets and exists only to show how the margin narrows with size; it is not a reportable performance number, because the validation set has been looked at before.
• ED30. Subgroup margins are computed by the same procedure. In a small subgroup the margin is wide, and this is not a flaw but a direct consequence of sample size.

The Difference Between Two Models and Its Share

The first block builds the setup, compares five rivals with the tree on the same test set and prints two margins for each difference.

# MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the
# same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS, DATA = [], {}, []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


def three_way_split(rows):
    a, b = int(len(rows) * 0.6), int(len(rows) * 0.8)
    return rows[:a], rows[a:b], rows[b:]


def impurity(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):
    p, best = sum(x["suspicious"] for x in rows) / len(rows), None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields),
            "right": tree(best[4], depth - 1, fields)}


def predict(node, x):
    while "prediction" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["prediction"]


def subgroups(x):                  # four axes, thirteen subgroups; carried over unchanged from 01
    return [("region", x["region_name"]),
            ("tariff", "10" if x["mean_consumption"] <= 10 else
             ("25" if x["mean_consumption"] <= 25 else "40")),
            ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")),
            ("reading", "few" if x["period"] <= 2 else "full")]


SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)})
TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000))
MAJORITY = int(sum(x["suspicious"] for x in TRAIN) * 2 > len(TRAIN))
M, DEPTH2, DEPTH8 = tree(TRAIN, 6, FIELDS), tree(TRAIN, 2, FIELDS), tree(TRAIN, 8, FIELDS)
REPEATS = 400
counts = {}
for x in TRAIN:                    # region and tier lookup table baseline, carried over from 03
    k = (x["region_name"], dict(subgroups(x))["tariff"])
    counts.setdefault(k, [0, 0])[x["suspicious"]] += 1
votes = {k: int(v[1] > v[0]) for k, v in counts.items()}
LOOKUP = (lambda x: votes.get((x["region_name"], dict(subgroups(x))["tariff"]), MAJORITY))


def paired_compare(rows, f, seed):
    # bootstrap was measured in the Statistical Analysis and Exploratory Data Analysis course;
    # here it is applied only to the DIFFERENCE
    hits_m = [int(predict(M, x) == x["suspicious"]) for x in rows]
    hits_f = [int(f(x) == x["suspicious"]) for x in rows]
    diff, n, paired, independent = [a - b for a, b in zip(hits_m, hits_f)], len(rows), [], []
    for j in range(REPEATS):
        r, r1, r2 = rng(seed + j), rng(seed + 5000 + j), rng(seed + 9000 + j)
        paired.append(sum(diff[int(r() * n)] for _ in range(n)) / n)
        independent.append(sum(hits_m[int(r1() * n)] for _ in range(n)) / n
                           - sum(hits_f[int(r2() * n)] for _ in range(n)) / n)
    paired.sort()
    independent.sort()
    low, high = int(0.05 * REPEATS), int(0.95 * REPEATS) - 1
    return (sum(diff) / n, paired[low], paired[high], independent[low], independent[high],
            sum(1 for v in diff if v))


RIVALS = (("most frequent class", lambda x: MAJORITY), ("depth 2", lambda x: predict(DEPTH2, x)),
          ("depth 8", lambda x: predict(DEPTH8, x)), ("lookup table", LOOKUP),
          ("all suspicious", lambda x: 1))
print(f"{'rival':<19} {'disagree':>8} {'diff':>8} | {'paired':>21} {'width':>7}"
      f" | {'independent':>21} {'width':>7} | zero")
for name, f in RIVALS:
    d, lo, hi, ilo, ihi, u = paired_compare(TEST, f, 71000)
    print(f"{name:<19} {u:>8} {d:>+8.4f} | [{lo:>+8.4f}, {hi:>+8.4f}] {hi - lo:>7.4f}"
          f" | [{ilo:>+8.4f}, {ihi:>+8.4f}] {ihi - ilo:>7.4f} | "
          f"{'outside' if lo > 0 or hi < 0 else 'inside'}")

rival               disagree     diff |                paired   width |           independent   width | zero
most frequent class       37  +0.0198 | [ -0.0198,  +0.0556]  0.0754 | [ -0.0516,  +0.0794]  0.1310 | inside
depth 2                   24  -0.0079 | [ -0.0437,  +0.0238]  0.0675 | [ -0.0714,  +0.0516]  0.1230 | inside
depth 8                   12  +0.0159 | [ -0.0079,  +0.0397]  0.0476 | [ -0.0437,  +0.0754]  0.1190 | inside
lookup table              33  -0.0119 | [ -0.0556,  +0.0238]  0.0794 | [ -0.0754,  +0.0476]  0.1230 | inside
all suspicious           215  +0.5357 | [ +0.4603,  +0.6190]  0.1587 | [ +0.4683,  +0.5992]  0.1310 | outside

The last column is the first result of the lesson. In four of the five comparisons the margin contains zero. The model’s 0.0198 lead over the most frequent class, its 0.0119 loss to the lookup table, the 0.0079 between it and depth 2, the 0.0159 between it and depth 8 — all four cannot be measured on this set of 252 subscribers. Only the “call everyone suspicious” difference stays outside the margin, at 0.5357. The numbers in the tables of the previous three lessons were not computed wrongly; they are below the measurement threshold, and these are two separate things.

The second result is the comparison of the two margin columns. The paired margin is narrower in every row: 0.0754 against 0.1310, 0.0675 against 0.1230, 0.0476 against 0.1190. The reason stands in the disagreement column. The depth-8 tree and the depth-6 tree part ways on only 12 of the 252 subscribers; on the remaining 240 both give the same answer, and the difficulty of those subscribers drops out of the computation of the difference entirely. In the independent measurement it does not drop out, because there the two models are assumed to have been measured on separate sets and each set’s own variability of difficulty enters the margin. Measuring the two models on the same samples narrows the margin by up to two and a half times, without collecting any data.

The last row reads in the opposite direction: because “call everyone suspicious” and the tree part ways on 215 subscribers, the paired margin there is wider than the independent one. Pairing does not always narrow; the more the two models agree, the more it narrows.

The Set Size Determines the Share

The width of the margin comes not from the model but from the set. The same comparison is repeated at different sizes taken from the head of the set.

POOL = VAL + TEST
print(f"{'size':>6} | {'diff':>8} {'paired':>21} {'width':>7} | {'independent width':>17}")
for n in (63, 126, 189, 252, 504):
    d, lo, hi, ilo, ihi, u = paired_compare(POOL[:n] if n > 252 else TEST[:n], LOOKUP, 74000)
    print(f"{n:>6} | {d:>+8.4f} [{lo:>+8.4f}, {hi:>+8.4f}] {hi - lo:>7.4f} | "
          f"{ihi - ilo:>17.4f}")

  size |     diff                paired   width | independent width
    63 |  +0.0000 [ -0.0635,  +0.0476]  0.1111 |            0.2540
   126 |  +0.0079 [ -0.0397,  +0.0556]  0.0952 |            0.1667
   189 |  -0.0106 [ -0.0529,  +0.0317]  0.0847 |            0.1376
   252 |  -0.0119 [ -0.0476,  +0.0278]  0.0754 |            0.1270
   504 |  -0.0159 [ -0.0437,  +0.0119]  0.0556 |            0.0794

The difference column is unstable: 0.0000 from 63 subscribers, +0.0079 from 126, −0.0159 from 504. Even the sign changes. The margin column, on the other hand, narrows steadily — from 0.1111 to 0.0556, which is to say that as the set grows eightfold the margin is halved. This means the smallest measurable difference depends on the set size: on 252 subscribers no difference below 0.04 can be read, and most of the model differences reported throughout the Machine Learning and Data Science curriculum are below that threshold.

The last row carries a warning. The set of 504 is the union of the validation and test sets, and the validation set was already looked at while the model was being chosen; the −0.0159 there is not a reportable performance number. The row stands only to show how the margin narrows with size. The size of the held-out set is therefore not a split ratio but a consequence of the smallest difference one wants to measure.

In the Subgroups the Difference Disappears Inside the Share

The subgroup gap was measured over three lessons. Now the margin of the difference in each subgroup is computed.

print(f"{'subgroup':<14} {'n':>4} {'disagree':>8} {'diff':>8} "
      f"{'paired margin':>22} {'zero':>8}")
outside = 0
for g in SUBGROUPS:
    rows = [x for x in TEST if g in subgroups(x)]
    d, lo, hi, ilo, ihi, u = paired_compare(rows, lambda x: MAJORITY, 76000)
    outside += lo > 0 or hi < 0
    print(f"{' '.join(g):<14} {len(rows):>4} {u:>8} {d:>+8.4f} [{lo:>+8.4f}, {hi:>+8.4f}]"
          f" {'outside' if lo > 0 or hi < 0 else 'inside':>8}")
print(f"\nsubgroups whose margin excludes zero: {outside}/{len(SUBGROUPS)}")

subgroup          n disagree     diff          paired margin     zero
household 1-2    73        9  +0.0411 [ -0.0274,  +0.1096]   inside
household 3-4   143       21  +0.0070 [ -0.0420,  +0.0629]   inside
household 5+     36        7  +0.0278 [ -0.0833,  +0.1389]   inside
reading few      85        4  -0.0471 [ -0.0824,  -0.0118]  outside
reading full    167       33  +0.0539 [ +0.0060,  +0.1018]  outside
region central   58        7  +0.0172 [ -0.0690,  +0.1034]   inside
region east      36       19  +0.1389 [ -0.0556,  +0.3611]   inside
region north     68        7  -0.0441 [ -0.1176,  +0.0147]   inside
region south     58        3  +0.0517 [ +0.0172,  +0.1034]  outside
region west      32        1  -0.0312 [ -0.0938,  +0.0000]   inside
tariff 10        12        1  -0.0833 [ -0.2500,  +0.0000]   inside
tariff 25       193       22  -0.0104 [ -0.0466,  +0.0311]   inside
tariff 40        47       14  +0.1702 [ +0.0426,  +0.2766]  outside

subgroups whose margin excludes zero: 4/13

The east region row is the harshest result of this lesson. The model’s contribution there is +0.1389, and this was one of the largest subgroup contributions the course had measured so far. Its margin is [−0.0556, +0.3611] and contains zero. Nineteen disagreements among thirty-six subscribers spread the difference over an interval a third of a point wide. The same number was read in two lessons as “what the model adds in the east”; here it turns out that it cannot be read that way.

Four subgroups leave zero outside, and one of them is in the opposite direction: among the 85 subscribers with few readings the difference is −0.0471 and its margin is [−0.0824, −0.0118]. In that subgroup the model is measurably worse than stating the most frequent class. The remaining three are in the forward direction: south +0.0517, full readings +0.0539, the tariff 40 tier +0.1702.

Two rules follow from this. The subgroup gap is the result of a design decision and the gap itself can be measured, but the individual differences that make it up cannot be measured in most subgroups. The second is that looking at thirteen subgroups and choosing the largest difference among them reproduces the multiple-comparison share counted in the Statistical Analysis and Exploratory Data Analysis course: stepping outside the margin in one of thirteen independent tests is not a finding unless it is written that thirteen looks were taken.

Summary

• In four of the five comparisons the margin contains zero: the model’s 0.0198 lead over the most frequent class, its 0.0119 loss to the lookup table and the two depth differences cannot be measured on 252 subscribers. A margin that contains zero means a shortfall in measurement, not equality.
• The paired comparison gives a narrower margin than the independent one: 0.0754 against 0.1310, 0.0476 against 0.1190. The narrowing comes from the agreement rate of the two models, and for a pair that parts ways on 215 subscribers pairing widens the margin.
• The margin narrows with set size: 0.1111 at 63 subscribers, 0.0556 at 504. The size of the held-out set is not a split ratio but a consequence of the smallest difference one wants to measure.
• Among the subgroups, the margin excludes zero for only 4 of the 13 differences. The +0.1389 in the east region, one of the largest subgroup contributions of the course, is not one of them: its margin is [−0.0556, +0.3611].
• Of the four differences that leave zero outside, one is in the opposite direction: among the subscribers with few readings the model is measurably below the baseline, at −0.0471.

Next Step

Over four lessons the measurement itself was designed: according to whom the set was split, from which cost the threshold was read, how the baseline was chosen, with which margin the difference was stated. The assumption they all share is that the test set comes from the same distribution as the data the model will meet. A measurement network changes over time: new regions join, tariff tiers shift, the rate of zero readings falls as meters are replaced. The next lesson measures that assumption: it reduces the difference between training and production data to a single number, places that number next to the drop in performance, and separates covariate shift from label shift to show why the two call for different solutions.