Validation Strategies

Across seven courses the baseline was written next to every number: a model was never read on its own, it was put side by side with the same-metric number of a procedure dumber than itself, and in the last course the fact that the system making the decision changes its own data was counted too. The one thing never asked is for whom the number was measured. All of those numbers were averages over subscribers, and an average may belong to none of the groups beneath it.

The rule of this course is this: a number cannot be read until it says for whom it was measured; performance whose subgroup is not written down has not been measured. The rule is paid for in every lesson with three numbers — the number measured overall, the same number measured on subgroups, the subgroup gap between them — and, when a gap is seen, with the sentence that changes in the report. This lesson sets up the triple and the subgroups, but first it answers a question that takes priority: whether the held-out set was truly held out.

• ED1. The dataset carries over unchanged from the seven courses and it is fictional: a 3,199-row reading table, 1,329 subscribers, three periods, five regions, seed 20260218. The label is fictional too — a subscriber carrying a suspicion of illicit use or a fault — and is not a real audit record.
• ED2. At subscriber level 1,260 rows remain and are split 756/252/252. The split, the metric (accuracy) and the baseline (the most frequent class of the training set) are inherited from the Introduction to Machine Learning course; the baseline on the test set is 0.7579. The model is a depth-6 decision tree, written with the standard library.
• ED3. The separation of the three sets and the time-ordered split were measured in the generalization topic of the Introduction to Machine Learning course and are not repeated here; the only thing added is the group-based split.
• ED4. The subgroups are defined on four axes and stay fixed throughout the course: region (five values), tariff tier (the tier the mean consumption falls in: 10, 25, 40 m³), household size (1–2, 3–4, 5 and above), reading count (few: one or two periods; full: three periods). 13 subgroups in total.
• ED5. The subgroup definition is a decision, not a finding. How many subgroups were looked at is written in every lesson, because looking at many produces findings out of noise — this was measured in the Statistical Analysis and Exploratory Data Analysis course and is not repeated.
• ED6. Next to every subgroup number is written how many subscribers it was read from; a group whose n is below 25 does not enter the gap calculation, because in 25 subscribers a single wrong count moves the number by 0.04.
• ED7. In the row-level table every row is one (subscriber, period) reading, the label belongs to the subscriber, and the subscriber-level summaries are written onto the rows too. Depth is swept over six candidates.

Thirteen Readings of the Same Number

The first block generates the setup, defines the subgroups and prints the course’s triple in a single table. The tree is a model: at each node it picks a threshold, and at the leaf it states the majority class.

# MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the
# same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS, DATA = [], {}, []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


def three_way_split(rows):
    a, b = int(len(rows) * 0.6), int(len(rows) * 0.8)
    return rows[:a], rows[a:b], rows[b:]


def impurity(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):
    p, best = sum(x["suspicious"] for x in rows) / len(rows), None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields),
            "right": tree(best[4], depth - 1, fields)}


def predict(node, x):
    while "prediction" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["prediction"]


def accuracy(node, rows):
    return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows)


def baseline(train, rows):
    c = int(sum(x["suspicious"] for x in train) * 2 > len(train))
    return sum(x["suspicious"] == c for x in rows) / len(rows)


def subgroups(x):                  # four axes, thirteen subgroups; fixed throughout the course
    return [("region", x["region_name"]),
            ("tariff", "10" if x["mean_consumption"] <= 10 else
             ("25" if x["mean_consumption"] <= 25 else "40")),
            ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")),
            ("reading", "few" if x["period"] <= 2 else "full")]


SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)})
TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000))
M = tree(TRAIN, 6, FIELDS)


def report_row(name, rows):
    base, acc = baseline(TRAIN, rows), accuracy(M, rows)
    print(f"{name:<14} {len(rows):>4} {base:>8.4f} {acc:>7.4f} {acc - base:>+8.4f}"
          f"{'   n<25, not in gap' if len(rows) < 25 else ''}")
    return len(rows), base, acc


print(f"subscribers {len(DATA)} -- 756/252/252 -- subgroups {len(SUBGROUPS)} (four axes)\n"
      f"{'subgroup':<14} {'n':>4} {'baseline':>8} {'tree':>7} {'diff':>8}")
report_row("TOTAL", TEST)
measured = {g: report_row(" ".join(g), [x for x in TEST if g in subgroups(x)]) for g in SUBGROUPS}
scored = [v[2] for v in measured.values() if v[0] >= 25]
below = [" ".join(g) for g, v in measured.items() if v[2] < v[1]]
print(f"\nsubgroup gap ({len(scored)} subgroups): best {max(scored):.4f}, worst {min(scored):.4f}, "
      f"gap {max(scored) - min(scored):.4f}\nbelow baseline: {len(below)}/"
      f"{len(SUBGROUPS)} -- " + ", ".join(below))

subscribers 1260 -- 756/252/252 -- subgroups 13 (four axes)
subgroup          n baseline    tree     diff
TOTAL           252   0.7579  0.7778  +0.0198
household 1-2    73   0.7534  0.7945  +0.0411
household 3-4   143   0.7692  0.7762  +0.0070
household 5+     36   0.7222  0.7500  +0.0278
reading few      85   0.7882  0.7412  -0.0471
reading full    167   0.7425  0.7964  +0.0539
region central   58   0.6552  0.6724  +0.0172
region east      36   0.4722  0.6111  +0.1389
region north     68   0.8235  0.7794  -0.0441
region south     58   0.8276  0.8793  +0.0517
region west      32   1.0000  0.9688  -0.0312
tariff 10        12   0.6667  0.5833  -0.0833   n<25, not in gap
tariff 25       193   0.8394  0.8290  -0.0104
tariff 40        47   0.4468  0.6170  +0.1702

subgroup gap (12 subgroups): best 0.9688, worst 0.6111, gap 0.3576
below baseline: 5/13 -- reading few, region north, region west, tariff 10, tariff 25

The first row is the row that seven courses read: the model beats the baseline by 0.0198. The thirteen rows beneath it are other numbers of the same metric for the same model, and the gap between the best and the worst is 0.3576 — eighteen times the difference read overall. The 0.7778 reported as performance means 0.6111 for a subscriber in the east region and 0.9688 for one in the west.

The harsher number is on the last line: the model beats the baseline overall but falls below the baseline in five of the thirteen subgroups. In the 85 subscribers with few readings the baseline is 0.7882 and the model 0.7412 — the model is 4.7 points worse than the procedure that learns nothing. The difference measured overall contains all of these groups and shows none of them.

The baseline column says why it is this way. None of the west’s 32 test subscribers is suspicious; there, saying “none is suspicious” gives 1.0000, and no model can beat that. In the east and in the tariff 40 tier the suspicious share exceeds half, so the baseline drops below 0.50 and the model beats it most there. The subgroup gap also carries the group’s own difficulty, which is why the baseline stands next to every number. The tariff 10 tier does not enter the gap because it holds 12 subscribers.

The Same Subscriber in Two Sets

This table assumes the held-out sets were truly separated. The raw form of the setup is not at subscriber level but at reading level: 3,199 rows for 1,260 subscribers, and a subscriber who produced three periods of readings has three rows, all with the same label. The moment the rows are shuffled and split, one row of the same subscriber falls into the training set and another into the test set.

ROWS = []
for x in DATA:
    for i, o in enumerate(READINGS[x["id"]]):
        ROWS.append(dict(x, reading=o, position=i))
ROW_FIELDS = ["reading", "position"] + FIELDS

ROW_TRAIN, ROW_VAL, ROW_TEST = three_way_split(shuffle(ROWS, SEED + 93000))     # row split
ids = shuffle([x["id"] for x in DATA], SEED + 93000)                             # group-based split
a, b = int(len(ids) * 0.6), int(len(ids) * 0.8)
ASSIGN = {n: (0 if i < a else (1 if i < b else 2)) for i, n in enumerate(ids)}
GROUP_TRAIN = [y for y in ROWS if ASSIGN[y["id"]] == 0]
GROUP_TEST = [y for y in ROWS if ASSIGN[y["id"]] == 2]

print(f"rows {len(ROWS)} -- row split: train {len(ROW_TRAIN)} test {len(ROW_TEST)}, subscribers "
      f"seen in both sets {len({y['id'] for y in ROW_TEST} & {y['id'] for y in ROW_TRAIN})}"
      f"\ngroup-based split: train {len(GROUP_TRAIN)} test {len(GROUP_TEST)}, shared subscribers "
      f"{len({y['id'] for y in GROUP_TEST} & {y['id'] for y in GROUP_TRAIN})}\nbaseline: row split "
      f"{baseline(ROW_TRAIN, ROW_TEST):.4f}, group-based split {baseline(GROUP_TRAIN, GROUP_TEST):.4f}\n\n"
      f"{'depth':>8} | {'row':>7} {'diff':>8} | {'group':>7} {'diff':>8} "
      f"| {'inflation':>9}")
for d in (2, 4, 6, 8, 10, 12):
    m_row, m_group = tree(ROW_TRAIN, d, ROW_FIELDS), tree(GROUP_TRAIN, d, ROW_FIELDS)
    d_row = accuracy(m_row, ROW_TEST) - baseline(ROW_TRAIN, ROW_TEST)
    d_group = accuracy(m_group, GROUP_TEST) - baseline(GROUP_TRAIN, GROUP_TEST)
    print(f"{d:>8} | {accuracy(m_row, ROW_TEST):>7.4f} {d_row:>+8.4f} | {accuracy(m_group, GROUP_TEST):>7.4f} "
          f"{d_group:>+8.4f} | {d_row - d_group:>+9.4f}")

rows 3199 -- row split: train 1919 test 640, subscribers seen in both sets 416
group-based split: train 1921 test 649, shared subscribers 0
baseline: row split 0.7594, group-based split 0.7319

   depth |     row     diff |   group     diff | inflation
       2 |  0.7812  +0.0219 |  0.7411  +0.0092 |   +0.0126
       4 |  0.7781  +0.0187 |  0.7689  +0.0370 |   -0.0182
       6 |  0.8078  +0.0484 |  0.7535  +0.0216 |   +0.0269
       8 |  0.8266  +0.0672 |  0.7458  +0.0139 |   +0.0533
      10 |  0.8375  +0.0781 |  0.7396  +0.0077 |   +0.0704
      12 |  0.8594  +0.1000 |  0.7427  +0.0108 |   +0.0892

In the row split, 416 of the subscribers who own the 640 test rows are also in the training set; in the group-based split this number is zero. The training sets of the two splits are nearly the same size (1,921 rows against 1,919), so the difference does not come from the amount of data.

The inflation grows with capacity. At depth 2 the tree has no room to memorize; at depth 4 the inflation is temporarily negative, because in a shallow tree the split noise is larger than the leakage. At depth 12 the table opens up: the row split shows 0.1000 above the baseline, the group-based split 0.0108. The 0.0892 between them is not something the model learned — the tree recognizes a subscriber from its feature values and recalls the label it saw from the other row.

The time-ordered split was measured in the Introduction to Machine Learning course, and there the sets were separated by when they were measured. The group-based split works on another axis: the sets are separated by whose data it is. If a record produces more than one row, the unit of the split is that record, not the row.

Inflation Shrinks the Gap

That leakage inflates the overall number is a known result. The question of this course is different: what happens to the gap. The block below takes the depth-12 models of the two splits down to the subgroups.

row_acc, group_acc, drops = [], [], []
for g in SUBGROUPS:
    row_sub = [y for y in ROW_TEST if g in subgroups(y)]
    group_sub = [y for y in GROUP_TEST if g in subgroups(y)]
    if min(len(row_sub), len(group_sub)) >= 60:     # 60 rows ~ 25 subscribers; the row equivalent of ED6
        row_acc.append(accuracy(m_row, row_sub))
        group_acc.append(accuracy(m_group, group_sub))
        drops.append((row_acc[-1] - group_acc[-1], " ".join(g), row_acc[-1], group_acc[-1]))
for f, name, a1, a2 in sorted(drops, reverse=True)[:2] + sorted(drops)[:1]:
    print(f"{name:<14} row {a1:.4f} | group {a2:.4f} | drop {f:+.4f}")
print(f"\nsubgroup gap -- row split {max(row_acc) - min(row_acc):.4f} ({min(row_acc):.4f} .. "
      f"{max(row_acc):.4f}), group-based split {max(group_acc) - min(group_acc):.4f} "
      f"({min(group_acc):.4f} .. {max(group_acc):.4f}), {len(row_acc)} subgroups")

tariff 40      row 0.7623 | group 0.5413 | drop +0.2210
region central row 0.8364 | group 0.6279 | drop +0.2085
region west    row 0.9341 | group 0.9259 | drop +0.0081

subgroup gap -- row split 0.1718 (0.7623 .. 0.9341), group-based split 0.3846 (0.5413 .. 0.9259), 12 subgroups

Leakage does not inflate the subgroups equally. The best subgroup is in the same place in both splits and has almost the same number: west, 0.9341 and 0.9259. What drops is the bad end — the tariff 40 tier from 0.7623 to 0.5413, the central region from 0.8364 to 0.6279. In the row split the subgroup gap reads 0.1718, in the group-based split 0.3846: leakage shrinks the difference between groups by more than half.

A badly designed split thus makes two errors at once: it raises the overall number and suppresses the difference between subgroups. The second is quieter, because whoever looks at the table reads “the groups are close to each other” and finds the subgroup measurement unnecessary.

Summary

• The triple of the course was set up: the number measured overall, the same number measured on 13 subgroups, and the subgroup gap between them. The axes are region, tariff tier, household size and reading count; the definition is a decision, and how many subgroups were looked at is written in every lesson.
• The depth-6 decision tree beats the baseline on the test set by 0.0198 but varies between 0.6111 and 0.9688 across the subgroups: a gap of 0.3576, eighteen times the difference overall. In five of the thirteen subgroups the model falls below the baseline.
• A subgroup number cannot be read without writing how many subscribers it was read from; the tariff 10 tier does not enter the gap because it holds 12 subscribers.
• When the 3,199 reading-level rows are shuffled, 416 subscribers fall into both sets, and at depth 12 the difference above the baseline reads 0.1000 instead of 0.0108; the group-based split undoes this 0.0892 inflation, and the inflation grows with capacity.
• Leakage corrupts the subgroup gap too: the gap that reads 0.1718 in the row split is 0.3846 in the group-based split. A bad split does not stop at raising the number, it also suppresses the difference between groups.

Next Step

Throughout, the metric was taken as accuracy and never questioned. Yet accuracy counts both error types with the same weight: the cost of sending a subscriber who is not suspicious to inspection is not the same as the cost of missing a suspicious one, and when the ratio between the two changes, the threshold of the same model changes. Metric definitions, threshold and curves were set up in the Supervised Learning course and will not be repeated. The next lesson measures only this: the same model gives two different operating points under two different cost matrices — and when the cost is not the same across subgroups, what a single threshold does to which group is counted.