--- title: 'Distribution Shift' source: 'https://academia.sh/en/courses/model-ethics-and-interpretation/distribution-shift' course: 'Model Evaluation, Interpretability and Ethics' language: en updated: '2026-10-21T05:03:26+00:00' license: 'CC BY-SA 4.0' objectives: - 'Reduces the difference between training and production data to a single number with a discriminator model: prints the shift score next to the drop in performance' - 'Separates covariate shift from label shift and shows that the indicators cannot see each other: the number that measures covariate shift stays near zero under label shift' - 'Measures that the two shifts call for two separate fixes and counts that the shift is not spread evenly across the subgroups' --- # Distribution Shift Reducing the difference between training and production data to a single number and placing that number next to the drop in performance: a discriminator model reads a shift score of 0.1172 under covariate shift and the model's difference above the baseline falls from 0.0198 to 0.0066, but under label shift the same indicator stays at 0.0063 while the suspicious rate climbs from 0.2421 to 0.4865. The two shifts call for two separate fixes: prior correction raises accuracy from 0.6306 to 0.6622 under label shift while changing no decision under covariate shift, and retraining on the input distribution raises 0.7318 to 0.7384 under covariate shift and drops label shift to 0.6261. On the time-ordered production set the subgroup gap falls from 0.3576 to 0.2573, and in none of the twelve subgroups does the model stay below the baseline. In four lessons the measurement itself was designed: which way the sets were split, which cost the threshold was read from, how the baseline was chosen, which margin the difference was stated with. All four assume that the test set comes from the **same distribution** as the data the model will meet. A measurement network changes over time: new regions join, tariff tiers move, and the zero-reading rate falls as meters are replaced. This lesson measures that assumption. **Distribution shift** comes in two forms, and when the two are gathered under one name the wrong fix gets chosen. In **covariate shift** the distribution of the features changes, but the probability of being suspicious given a subscriber's features stays the same. In **label shift** the rate of suspicion changes. Data drift was mentioned in the Introduction to Machine Learning course and the time-ordered split was measured there; what is added here is **reducing the shift to a single number** and separating the two kinds. - **ED31.** The setup, the split (**756/252/252**), the seed, the metric (**accuracy**), the model (**depth-6 decision tree**, suspicious rate in the leaf) and the subgroup definitions carry over from the earlier lessons: four axes, **13 subgroups**. The suspicious rate of the training set is **0.2222**. - **ED32.** The **shift score** is measured with a **discriminator model**: a depth-4 tree that predicts whether a row came from the training set or the production set, trained with an equal number of rows from the two sets and measured on a held-out half. The number is **accuracy minus 0.5**; a value near zero means "cannot be told apart". The mean of five runs is taken, and the label column is **not given** to the discriminator. - **ED33.** The shift score sees only the **input** distribution. It cannot see a shift in the label distribution, and this is one of the things the lesson measures. - **ED34.** The three constructed production sets are derived from the union of the validation and test sets (504 subscribers). **Covariate shift**: the probability of selecting subscribers with a high mean consumption is raised, and the label rule is not touched. **Label shift**: the suspicious subscribers are kept and only 30% of the non-suspicious ones are retained. **Time-ordered production**: the last 20% of the set sorted by subscriber number; the subscriber number is the order of joining the network. - **ED35.** **Prior correction** multiplies the model's leaf rate by the ratio of the odds of suspicion in production and in training; it requires knowing labels in production, which is a cost. **Retraining** builds the same tree on a training set of **439** rows sampled with the covariate shift rule; 317 of the 756 rows are dropped, and the lost data is a cost. - **ED36.** The model is trained once and the same model is measured on four sets; the difference comes from the data, not from the model. The sizes of the production sets are not equal (302, 222, 252), and the margin calculations were measured in the previous lesson and are not repeated. ## The Size of the Shift Is a Number The first block sets up the scenario, produces the three production sets, measures the shift score for each, and prints on one row the baseline, the model's number and the results of the two fixes. ```python # MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the # same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS, DATA = [], {}, [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] def three_way_split(rows): a, b = int(len(rows) * 0.6), int(len(rows) * 0.8) return rows[:a], rows[a:b], rows[b:] def impurity(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"p": p} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields), "right": tree(best[4], depth - 1, fields)} def score(node, x): while "p" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["p"] def subgroups(x): # four axes, thirteen subgroups; carried over unchanged from lesson 01 return [("region", x["region_name"]), ("tariff", "10" if x["mean_consumption"] <= 10 else ("25" if x["mean_consumption"] <= 25 else "40")), ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")), ("reading", "few" if x["period"] <= 2 else "full")] SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)}) TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000)) M = tree(TRAIN, 6, FIELDS) RATE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN) MAJORITY = int(RATE > 0.5) def hit_rate(f, rows): return sum(f(x) == x["suspicious"] for x in rows) / len(rows) def shift_score(a, b, seed, repeats=5): # DISCRIMINATOR MODEL: a depth-4 tree that predicts which set a row came from t = 0.0 for j in range(repeats): n = min(len(a), len(b)) ka = [dict(x, suspicious=0) for x in shuffle(a, seed + j)[:n]] kb = [dict(x, suspicious=1) for x in shuffle(b, seed + 500 + j)[:n]] d = tree(shuffle(ka[:n // 2] + kb[:n // 2], seed + 900 + j), 4, FIELDS) s = ka[n // 2:] + kb[n // 2:] t += sum((score(d, x) > 0.5) == x["suspicious"] for x in s) / len(s) return t / repeats - 0.5 def covariate_shift(rows, seed, power=0.12): # high consumption weighs more; the label rule is the SAME r = rng(seed) return [x for x in rows if r() < min(0.98, 0.03 + power * max(0.0, x["mean_consumption"] - 14))] def label_shift(rows, seed, keep=0.30): # suspicious rows are kept, 30% of the non-suspicious remain r = rng(seed) return [x for x in rows if x["suspicious"] or r() < keep] POOL = VAL + TEST TIME_SET = three_way_split(sorted(DATA, key=lambda x: x["id"]))[2] COVARIATE_SET, LABEL_SET = covariate_shift(POOL, 83000), label_shift(POOL, 84000) M_RETRAINED = tree(covariate_shift(TRAIN, 86000), 6, FIELDS) def prior_correction(rows): # prior correction: with the ratio of odds p1 = sum(x["suspicious"] for x in rows) / len(rows) o = (p1 / (1 - p1)) / (RATE / (1 - RATE)) return lambda x: int(score(M, x) * o / (score(M, x) * o + 1 - score(M, x)) >= 0.5) print(f"{'set':<18} {'n':>4} {'suspicious':>10} {'shift':>8} {'base':>7} {'model':>7} " f"{'diff':>8} {'prior':>7} {'retrain':>7}") for name, s in (("same distribution", TEST), ("covariate shift", COVARIATE_SET), ("label shift", LABEL_SET), ("time ordered", TIME_SET)): base = hit_rate(lambda x: MAJORITY, s) print(f"{name:<18} {len(s):>4} {sum(x['suspicious'] for x in s)/len(s):>10.4f} " f"{shift_score(TRAIN, s, 85000):>+8.4f} {base:>7.4f} " f"{hit_rate(lambda x: int(score(M, x) > 0.5), s):>7.4f} " f"{hit_rate(lambda x: int(score(M, x) > 0.5), s)-base:>+8.4f} " f"{hit_rate(prior_correction(s), s):>7.4f} {hit_rate(lambda x: int(score(M_RETRAINED, x) > 0.5), s):>7.4f}") ``` ``` set n suspicious shift base model diff prior retrain same distribution 252 0.2421 -0.0127 0.7579 0.7778 +0.0198 0.7817 0.7698 covariate shift 302 0.2748 +0.1172 0.7252 0.7318 +0.0066 0.7318 0.7384 label shift 222 0.4865 +0.0063 0.5135 0.6306 +0.1171 0.6622 0.6261 time ordered 252 0.3532 -0.0056 0.6468 0.7500 +0.1032 0.7579 0.7381 ``` The first row validates the measurement. Training and test sets came from the same split, so the discriminator model cannot separate them and the shift score is **−0.0127**; the minus sign is not a direction, it is run noise on a number indistinguishable from zero. In the second row the shift score is **+0.1172**: the discriminator model predicts which set a row came from 11.7 points better than random guessing, so there is a real feature-level difference. On the performance side the counterpart is this: the model drops from 0.7778 to 0.7318, and its difference above the baseline falls **from 0.0198 to 0.0066**. The size of the shift and the erosion of the contribution are read on one row. The third and fourth rows give the distinction of the lesson. Under label shift the suspicious rate climbs from 0.2421 to **0.4865**, and under time-ordered production to **0.3532**; the sets are unrecognizably different. Yet the shift score is **+0.0063** and **−0.0056**, which is zero. The indicator that measures covariate shift sees nothing even when half the labels have changed, because it looks only at the distribution of the features. **The silence of the shift indicator does not mean there is no shift.** ## Two Shifts Call for Two Separate Fixes The last two columns carry the two fixes, and each works only on its own shift. Prior correction rescales the leaf rate of the model to the new suspicion rate. Under label shift it raises accuracy **from 0.6306 to 0.6622**, and under time-ordered production **from 0.7500 to 0.7579**. Under covariate shift it does not move the number at all: 0.7318 and 0.7318. The reason is arithmetic: there the suspicious rate has gone only to 0.2748, the ratio of the odds is close to one, and no decision flips. It has a cost too: knowing the suspicion rate in production requires collecting labels there. Retraining works in the opposite direction. The tree built on a 439-row training set sampled with the covariate shift rule rises **from 0.7318 to 0.7384** under covariate shift; on the other three sets it **lowers** the number (0.7698, **0.6261**, 0.7381). A third of the training set was thrown away, so the model learns from less data, and this loss is made up only on the targeted shift. The rule is this: a fix chosen **without determining the type** of the shift does harm. Collecting labels for covariate shift, throwing data away for label shift: both pull the number down here. ## The Shift Is Not Equal in the Subgroups The time-ordered production set is the most realistic of the four: as the network grows, later subscribers enter the set. The second block compares the share and the suspicious rate of the subgroups in that set with those in the training set. ```python print(f"{'subgroup':<14} | {'train share':>11} {'prod share':>11} | {'train susp':>10} " f"{'prod susp':>10} | {'n':>4} {'base':>7} {'model':>7} {'diff':>8}") accs, gaps = [], [] for g in SUBGROUPS: e = [x for x in TRAIN if g in subgroups(x)] u = [x for x in TIME_SET if g in subgroups(x)] base, mod = hit_rate(lambda x: MAJORITY, u), hit_rate(lambda x: int(score(M, x) > 0.5), u) if len(u) >= 25: accs.append(mod) gaps.append(mod - base) print(f"{' '.join(g):<14} | {len(e)/len(TRAIN):>11.4f} {len(u)/len(TIME_SET):>11.4f} | " f"{sum(x['suspicious'] for x in e)/len(e):>10.4f} " f"{sum(x['suspicious'] for x in u)/len(u):>10.4f} | {len(u):>4} {base:>7.4f} " f"{mod:>7.4f} {mod-base:>+8.4f}{' n<25' if len(u) < 25 else ''}") print(f"\nproduction set gap ({len(accs)} subgroups): best {max(accs):.4f}, worst " f"{min(accs):.4f}, gap {max(accs)-min(accs):.4f}; below baseline " f"{sum(1 for v in gaps if v < 0)}/{len(gaps)}") ``` ``` subgroup | train share prod share | train susp prod susp | n base model diff household 1-2 | 0.3148 0.3135 | 0.2227 0.4177 | 79 0.5823 0.6962 +0.1139 household 3-4 | 0.5265 0.5476 | 0.2060 0.3188 | 138 0.6812 0.7754 +0.0942 household 5+ | 0.1587 0.1389 | 0.2750 0.3429 | 35 0.6571 0.7714 +0.1143 reading few | 0.3373 0.3294 | 0.1725 0.2892 | 83 0.7108 0.7831 +0.0723 reading full | 0.6627 0.6706 | 0.2475 0.3846 | 169 0.6154 0.7337 +0.1183 region central | 0.1772 0.1905 | 0.2761 0.4375 | 48 0.5625 0.6458 +0.0833 region east | 0.1799 0.1548 | 0.4706 0.6667 | 39 0.3333 0.7692 +0.4359 region north | 0.2897 0.2937 | 0.1963 0.2973 | 74 0.7027 0.7027 +0.0000 region south | 0.1865 0.2183 | 0.1064 0.2727 | 55 0.7273 0.8000 +0.0727 region west | 0.1667 0.1429 | 0.0714 0.1389 | 36 0.8611 0.8889 +0.0278 tariff 10 | 0.0384 0.0476 | 0.2069 0.4167 | 12 0.5833 0.6667 +0.0833 n<25 tariff 25 | 0.7804 0.7262 | 0.1712 0.2732 | 183 0.7268 0.7923 +0.0656 tariff 40 | 0.1812 0.2262 | 0.4453 0.5965 | 57 0.4035 0.6316 +0.2281 production set gap (12 subgroups): best 0.8889, worst 0.6316, gap 0.2573; below baseline 0/12 ``` The first two columns are nearly stationary: the largest change is in tariff 25, from 0.7804 to 0.7262. The composition of the subgroups has not changed, so covariate shift is small here, and the shift score of −0.0056 agrees. In the next two columns the suspicious rate of **every subgroup** rises, but not equally: in the east from 0.4706 to 0.6667, in the household 1-2 group from 0.2227 to 0.4177, in the tariff 25 tier only from 0.1712 to 0.2732. Label shift looks like a single number overall, and like thirteen separate numbers in the subgroups. The last row completes the course's triple and goes in an unexpected direction. The subgroup gap in the production set is **0.2573**; the gap measured for the same model in the first lesson was 0.3576. The shifted set **brings the groups closer together**. Harsher still, in **none** of the twelve subgroups does the model stay below the baseline; on the unshifted set the same model with the same metric was below it in five. The reason is the baseline itself: as the suspicious rate rises, saying "none is suspicious" gets weaker in every subgroup, and the model passes it. A number measured on a shifted set shows that the baseline has worsened, not that the model has improved, and a report without the baseline beside it cannot tell the two apart. ## Summary - The shift comes down to one number: a discriminator model predicts whether a row came from training or from production, and the difference of the accuracy from 0.5 is the **shift score**. It is +0.1172 under covariate shift and −0.0127 on the same distribution. - The shift score sees only the input. Under label shift the suspicious rate climbs from 0.2421 to 0.4865 while the indicator stays at +0.0063, and under time-ordered production at −0.0056: **the silence of the indicator does not mean there is no shift**. - The size of the shift is read next to the drop in performance: under covariate shift the model's difference above the baseline falls from 0.0198 to **0.0066**. - The two shifts call for two separate fixes. Prior correction raises 0.6306 to 0.6622 under label shift and changes no decision under covariate shift. Retraining on the input distribution raises 0.7318 to 0.7384 under covariate shift and lowers the number on the other three sets. - Under time-ordered production the subgroup shares are stationary but the suspicious rates rise unequally (+0.1961 in the east, +0.1020 in tariff 25). The gap falls from 0.3576 to **0.2573**, and the model is no longer below the baseline in any subgroup, because what falls is not the model but the baseline. ## Next Step In five lessons, what a number **is** and **whom it is for** were measured: which way the unit of separation was chosen, which cost the threshold was read from, in how many ways the baseline could be built, which margin the difference was stated with, how far the production data had shifted. At every step the number produced by the same tree said something else in a subgroup, and the gap was re-measured between 0.2573 and 0.8606. Throughout all of this the model stayed a **closed box**. What the model **based** that number on was never asked: which feature carries the decision, which value the decision for a single subscriber rests on, and whether that rationale, once written down, describes what the model actually does. The next topic measures these three questions.