Distribution Shift

In four lessons the measurement itself was designed: which way the sets were split, which cost the threshold was read from, how the baseline was chosen, which margin the difference was stated with. All four assume that the test set comes from the same distribution as the data the model will meet. A measurement network changes over time: new regions join, tariff tiers move, and the zero-reading rate falls as meters are replaced.

This lesson measures that assumption. Distribution shift comes in two forms, and when the two are gathered under one name the wrong fix gets chosen. In covariate shift the distribution of the features changes, but the probability of being suspicious given a subscriber’s features stays the same. In label shift the rate of suspicion changes. Data drift was mentioned in the Introduction to Machine Learning course and the time-ordered split was measured there; what is added here is reducing the shift to a single number and separating the two kinds.

• ED31. The setup, the split (756/252/252), the seed, the metric (accuracy), the model (depth-6 decision tree, suspicious rate in the leaf) and the subgroup definitions carry over from the earlier lessons: four axes, 13 subgroups. The suspicious rate of the training set is 0.2222.
• ED32. The shift score is measured with a discriminator model: a depth-4 tree that predicts whether a row came from the training set or the production set, trained with an equal number of rows from the two sets and measured on a held-out half. The number is accuracy minus 0.5; a value near zero means “cannot be told apart”. The mean of five runs is taken, and the label column is not given to the discriminator.
• ED33. The shift score sees only the input distribution. It cannot see a shift in the label distribution, and this is one of the things the lesson measures.
• ED34. The three constructed production sets are derived from the union of the validation and test sets (504 subscribers). Covariate shift: the probability of selecting subscribers with a high mean consumption is raised, and the label rule is not touched. Label shift: the suspicious subscribers are kept and only 30% of the non-suspicious ones are retained. Time-ordered production: the last 20% of the set sorted by subscriber number; the subscriber number is the order of joining the network.
• ED35. Prior correction multiplies the model’s leaf rate by the ratio of the odds of suspicion in production and in training; it requires knowing labels in production, which is a cost. Retraining builds the same tree on a training set of 439 rows sampled with the covariate shift rule; 317 of the 756 rows are dropped, and the lost data is a cost.
• ED36. The model is trained once and the same model is measured on four sets; the difference comes from the data, not from the model. The sizes of the production sets are not equal (302, 222, 252), and the margin calculations were measured in the previous lesson and are not repeated.

The Size of the Shift Is a Number

The first block sets up the scenario, produces the three production sets, measures the shift score for each, and prints on one row the baseline, the model’s number and the results of the two fixes.

# MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the
# same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS, DATA = [], {}, []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


def three_way_split(rows):
    a, b = int(len(rows) * 0.6), int(len(rows) * 0.8)
    return rows[:a], rows[a:b], rows[b:]


def impurity(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):
    p, best = sum(x["suspicious"] for x in rows) / len(rows), None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"p": p}
    return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields),
            "right": tree(best[4], depth - 1, fields)}


def score(node, x):
    while "p" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["p"]


def subgroups(x):                  # four axes, thirteen subgroups; carried over unchanged from lesson 01
    return [("region", x["region_name"]),
            ("tariff", "10" if x["mean_consumption"] <= 10 else
             ("25" if x["mean_consumption"] <= 25 else "40")),
            ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")),
            ("reading", "few" if x["period"] <= 2 else "full")]


SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)})
TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000))
M = tree(TRAIN, 6, FIELDS)
RATE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN)
MAJORITY = int(RATE > 0.5)


def hit_rate(f, rows):
    return sum(f(x) == x["suspicious"] for x in rows) / len(rows)


def shift_score(a, b, seed, repeats=5):
    # DISCRIMINATOR MODEL: a depth-4 tree that predicts which set a row came from
    t = 0.0
    for j in range(repeats):
        n = min(len(a), len(b))
        ka = [dict(x, suspicious=0) for x in shuffle(a, seed + j)[:n]]
        kb = [dict(x, suspicious=1) for x in shuffle(b, seed + 500 + j)[:n]]
        d = tree(shuffle(ka[:n // 2] + kb[:n // 2], seed + 900 + j), 4, FIELDS)
        s = ka[n // 2:] + kb[n // 2:]
        t += sum((score(d, x) > 0.5) == x["suspicious"] for x in s) / len(s)
    return t / repeats - 0.5


def covariate_shift(rows, seed, power=0.12):      # high consumption weighs more; the label rule is the SAME
    r = rng(seed)
    return [x for x in rows if r() < min(0.98, 0.03 + power * max(0.0, x["mean_consumption"] - 14))]


def label_shift(rows, seed, keep=0.30):           # suspicious rows are kept, 30% of the non-suspicious remain
    r = rng(seed)
    return [x for x in rows if x["suspicious"] or r() < keep]


POOL = VAL + TEST
TIME_SET = three_way_split(sorted(DATA, key=lambda x: x["id"]))[2]
COVARIATE_SET, LABEL_SET = covariate_shift(POOL, 83000), label_shift(POOL, 84000)
M_RETRAINED = tree(covariate_shift(TRAIN, 86000), 6, FIELDS)


def prior_correction(rows):                       # prior correction: with the ratio of odds
    p1 = sum(x["suspicious"] for x in rows) / len(rows)
    o = (p1 / (1 - p1)) / (RATE / (1 - RATE))
    return lambda x: int(score(M, x) * o / (score(M, x) * o + 1 - score(M, x)) >= 0.5)


print(f"{'set':<18} {'n':>4} {'suspicious':>10} {'shift':>8} {'base':>7} {'model':>7} "
      f"{'diff':>8} {'prior':>7} {'retrain':>7}")
for name, s in (("same distribution", TEST), ("covariate shift", COVARIATE_SET),
                ("label shift", LABEL_SET), ("time ordered", TIME_SET)):
    base = hit_rate(lambda x: MAJORITY, s)
    print(f"{name:<18} {len(s):>4} {sum(x['suspicious'] for x in s)/len(s):>10.4f} "
          f"{shift_score(TRAIN, s, 85000):>+8.4f} {base:>7.4f} "
          f"{hit_rate(lambda x: int(score(M, x) > 0.5), s):>7.4f} "
          f"{hit_rate(lambda x: int(score(M, x) > 0.5), s)-base:>+8.4f} "
          f"{hit_rate(prior_correction(s), s):>7.4f} {hit_rate(lambda x: int(score(M_RETRAINED, x) > 0.5), s):>7.4f}")

set                   n suspicious    shift    base   model     diff   prior retrain
same distribution   252     0.2421  -0.0127  0.7579  0.7778  +0.0198  0.7817  0.7698
covariate shift     302     0.2748  +0.1172  0.7252  0.7318  +0.0066  0.7318  0.7384
label shift         222     0.4865  +0.0063  0.5135  0.6306  +0.1171  0.6622  0.6261
time ordered        252     0.3532  -0.0056  0.6468  0.7500  +0.1032  0.7579  0.7381

The first row validates the measurement. Training and test sets came from the same split, so the discriminator model cannot separate them and the shift score is −0.0127; the minus sign is not a direction, it is run noise on a number indistinguishable from zero.

In the second row the shift score is +0.1172: the discriminator model predicts which set a row came from 11.7 points better than random guessing, so there is a real feature-level difference. On the performance side the counterpart is this: the model drops from 0.7778 to 0.7318, and its difference above the baseline falls from 0.0198 to 0.0066. The size of the shift and the erosion of the contribution are read on one row.

The third and fourth rows give the distinction of the lesson. Under label shift the suspicious rate climbs from 0.2421 to 0.4865, and under time-ordered production to 0.3532; the sets are unrecognizably different. Yet the shift score is +0.0063 and −0.0056, which is zero. The indicator that measures covariate shift sees nothing even when half the labels have changed, because it looks only at the distribution of the features. The silence of the shift indicator does not mean there is no shift.

Two Shifts Call for Two Separate Fixes

The last two columns carry the two fixes, and each works only on its own shift.

Prior correction rescales the leaf rate of the model to the new suspicion rate. Under label shift it raises accuracy from 0.6306 to 0.6622, and under time-ordered production from 0.7500 to 0.7579. Under covariate shift it does not move the number at all: 0.7318 and 0.7318. The reason is arithmetic: there the suspicious rate has gone only to 0.2748, the ratio of the odds is close to one, and no decision flips. It has a cost too: knowing the suspicion rate in production requires collecting labels there.

Retraining works in the opposite direction. The tree built on a 439-row training set sampled with the covariate shift rule rises from 0.7318 to 0.7384 under covariate shift; on the other three sets it lowers the number (0.7698, 0.6261, 0.7381). A third of the training set was thrown away, so the model learns from less data, and this loss is made up only on the targeted shift.

The rule is this: a fix chosen without determining the type of the shift does harm. Collecting labels for covariate shift, throwing data away for label shift: both pull the number down here.

The Shift Is Not Equal in the Subgroups

The time-ordered production set is the most realistic of the four: as the network grows, later subscribers enter the set. The second block compares the share and the suspicious rate of the subgroups in that set with those in the training set.

print(f"{'subgroup':<14} | {'train share':>11} {'prod share':>11} | {'train susp':>10} "
      f"{'prod susp':>10} | {'n':>4} {'base':>7} {'model':>7} {'diff':>8}")
accs, gaps = [], []
for g in SUBGROUPS:
    e = [x for x in TRAIN if g in subgroups(x)]
    u = [x for x in TIME_SET if g in subgroups(x)]
    base, mod = hit_rate(lambda x: MAJORITY, u), hit_rate(lambda x: int(score(M, x) > 0.5), u)
    if len(u) >= 25:
        accs.append(mod)
        gaps.append(mod - base)
    print(f"{' '.join(g):<14} | {len(e)/len(TRAIN):>11.4f} {len(u)/len(TIME_SET):>11.4f} | "
          f"{sum(x['suspicious'] for x in e)/len(e):>10.4f} "
          f"{sum(x['suspicious'] for x in u)/len(u):>10.4f} | {len(u):>4} {base:>7.4f} "
          f"{mod:>7.4f} {mod-base:>+8.4f}{'  n<25' if len(u) < 25 else ''}")
print(f"\nproduction set gap ({len(accs)} subgroups): best {max(accs):.4f}, worst "
      f"{min(accs):.4f}, gap {max(accs)-min(accs):.4f}; below baseline "
      f"{sum(1 for v in gaps if v < 0)}/{len(gaps)}")

subgroup       | train share  prod share | train susp  prod susp |    n    base   model     diff
household 1-2  |      0.3148      0.3135 |     0.2227     0.4177 |   79  0.5823  0.6962  +0.1139
household 3-4  |      0.5265      0.5476 |     0.2060     0.3188 |  138  0.6812  0.7754  +0.0942
household 5+   |      0.1587      0.1389 |     0.2750     0.3429 |   35  0.6571  0.7714  +0.1143
reading few    |      0.3373      0.3294 |     0.1725     0.2892 |   83  0.7108  0.7831  +0.0723
reading full   |      0.6627      0.6706 |     0.2475     0.3846 |  169  0.6154  0.7337  +0.1183
region central |      0.1772      0.1905 |     0.2761     0.4375 |   48  0.5625  0.6458  +0.0833
region east    |      0.1799      0.1548 |     0.4706     0.6667 |   39  0.3333  0.7692  +0.4359
region north   |      0.2897      0.2937 |     0.1963     0.2973 |   74  0.7027  0.7027  +0.0000
region south   |      0.1865      0.2183 |     0.1064     0.2727 |   55  0.7273  0.8000  +0.0727
region west    |      0.1667      0.1429 |     0.0714     0.1389 |   36  0.8611  0.8889  +0.0278
tariff 10      |      0.0384      0.0476 |     0.2069     0.4167 |   12  0.5833  0.6667  +0.0833  n<25
tariff 25      |      0.7804      0.7262 |     0.1712     0.2732 |  183  0.7268  0.7923  +0.0656
tariff 40      |      0.1812      0.2262 |     0.4453     0.5965 |   57  0.4035  0.6316  +0.2281

production set gap (12 subgroups): best 0.8889, worst 0.6316, gap 0.2573; below baseline 0/12

The first two columns are nearly stationary: the largest change is in tariff 25, from 0.7804 to 0.7262. The composition of the subgroups has not changed, so covariate shift is small here, and the shift score of −0.0056 agrees. In the next two columns the suspicious rate of every subgroup rises, but not equally: in the east from 0.4706 to 0.6667, in the household 1-2 group from 0.2227 to 0.4177, in the tariff 25 tier only from 0.1712 to 0.2732. Label shift looks like a single number overall, and like thirteen separate numbers in the subgroups.

The last row completes the course’s triple and goes in an unexpected direction. The subgroup gap in the production set is 0.2573; the gap measured for the same model in the first lesson was 0.3576. The shifted set brings the groups closer together. Harsher still, in none of the twelve subgroups does the model stay below the baseline; on the unshifted set the same model with the same metric was below it in five. The reason is the baseline itself: as the suspicious rate rises, saying “none is suspicious” gets weaker in every subgroup, and the model passes it. A number measured on a shifted set shows that the baseline has worsened, not that the model has improved, and a report without the baseline beside it cannot tell the two apart.

Summary

• The shift comes down to one number: a discriminator model predicts whether a row came from training or from production, and the difference of the accuracy from 0.5 is the shift score. It is +0.1172 under covariate shift and −0.0127 on the same distribution.
• The shift score sees only the input. Under label shift the suspicious rate climbs from 0.2421 to 0.4865 while the indicator stays at +0.0063, and under time-ordered production at −0.0056: the silence of the indicator does not mean there is no shift.
• The size of the shift is read next to the drop in performance: under covariate shift the model’s difference above the baseline falls from 0.0198 to 0.0066.
• The two shifts call for two separate fixes. Prior correction raises 0.6306 to 0.6622 under label shift and changes no decision under covariate shift. Retraining on the input distribution raises 0.7318 to 0.7384 under covariate shift and lowers the number on the other three sets.
• Under time-ordered production the subgroup shares are stationary but the suspicious rates rise unequally (+0.1961 in the east, +0.1020 in tariff 25). The gap falls from 0.3576 to 0.2573, and the model is no longer below the baseline in any subgroup, because what falls is not the model but the baseline.

Next Step

In five lessons, what a number is and whom it is for were measured: which way the unit of separation was chosen, which cost the threshold was read from, in how many ways the baseline could be built, which margin the difference was stated with, how far the production data had shifted. At every step the number produced by the same tree said something else in a subgroup, and the gap was re-measured between 0.2573 and 0.8606. Throughout all of this the model stayed a closed box. What the model based that number on was never asked: which feature carries the decision, which value the decision for a single subscriber rests on, and whether that rationale, once written down, describes what the model actually does. The next topic measures these three questions.