--- title: 'Data Leakage' source: 'https://academia.sh/en/courses/introduction-to-machine-learning/data-leakage' course: 'Introduction to Machine Learning' language: en updated: '2026-10-07T04:31:20+00:00' license: 'CC BY-SA 4.0' objectives: - 'Applies the four kinds of leakage one by one and measures how much each of them inflates the held-out number against its own honest counterpart' - 'Separates the signature of leakage from the signature of capacity: reads the rise of the held-out number, not of the training number' - 'Shows that a model that has learned nothing can beat the baseline and separates, at the row level, where the difference comes from' --- # Data Leakage Measuring how sources other than learning inflate the held-out number: against the honest line, which gives an accuracy of 0.7778 on the test set with a baseline of 0.7579, scaling before the split inflates this number by 0.0119, a feature that sees the future by 0.0754, a duplicated record by 0.0542 and a feature derived from the target by 0.1429; all four have the same sign. A model whose features have been replaced with six random columns reads an accuracy of 0.8757 on the held-out set while the duplicates are present and beats the baseline of 0.7857 by 9 points; on the rows whose twin it has not seen, it stays 9.2 points below the baseline. At the course wrap-up, the baseline, training and held-out numbers of eleven lessons are gathered in a single table. The previous lesson silently made one decision correctly: the scale of the columns was computed from the training set only. The decision can be reversed. When the scale is computed from the whole set, the rows of the held-out set contribute to producing the numbers the model will see, and the held-out set stops being held out by that amount. This is the quietest form of **data leakage**. Leakage is the mixing of information that will not be available at prediction time into the training or measurement path, and it always pushes the same way: the held-out number rises. This lesson applies the four kinds one by one, measures the inflation of each against its own honest counterpart and counts the second claim of the curriculum — **the difference does not always come from learning.** - **GE31.** The set, the label, the split, the seed and the measure are the same throughout the course. The held-out set is the test set: 252 rows. The honest line is the model of the first lesson: the depth 6 decision tree, baseline 0.7579, 0.8466 on training, 0.7778 on test. - **GE32.** The first leak is scaling by region: the region mean and scatter are computed from the training set alone in the honest form and from the whole set in the leaky form. - **GE33.** The second leak is the fourth-period reading and is **fictional**: it is formed after the label is written, for suspicious subscribers it comes out zero with probability 0.55, and if it does not, its value is 1.45 times as much. - **GE34.** The third leak is a join error: every subscriber record is produced three times, 1,260 subscribers rise to 3,780 rows and the split is made at the **row** level. - **GE35.** The fourth leak is the audit team assignment and is **fictional**: it is one with probability 0.88 for suspicious subscribers and 0.06 for the others, and it is recorded after the label. - **GE36.** The inflation is always measured against **its own honest counterpart**; the counterpart of the first leak is the honestly computed form of the same feature. - **GE37.** In the last section the features are replaced with six random columns. These columns are unrelated to the label and identical across the three copies of a subscriber. ## Four Leaks, Four Inflations The setup is the same as in the first lesson of the course and produces the same set with the same seed. ```python # generalization.py — MODEL. The regional measurement network (a fictional setup) of the Data Analytics curriculum is generated with the same # seed; readings are summarized at the subscriber level and a FICTIONAL illicit-use/fault suspicion label # is added on top. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS = [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) DATA = [] for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["reg_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "reg_north", "reg_south", "reg_east", "reg_west", "reg_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, TEST = K[:756], K[1008:] def gini(s): p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0 return 2 * p * (1 - p) def tree(s, depth, min_size=2): p = sum(x["suspicious"] for x in s) / len(s) best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for f in FIELDS: values = sorted({x[f] for x in s}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in s if x[f] < v] right = [x for x in s if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, min_size), "right": tree(best[4], depth - 1, min_size)} def predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def accuracy(d, s): return sum(predict(d, x) == x["suspicious"] for x in s) / len(s) def baseline(train, s): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in s) / len(s) ``` The four leaks are set up in turn and each is measured against its own honest counterpart. ```python BASE, RESULTS = list(FIELDS), [] for x in DATA: x["region_name"] = [b[0] for b in REGION if x["reg_" + b[0]]][0] def measure(fields, train, held, depth=6): global FIELDS FIELDS = fields m = tree(train, depth) FIELDS = BASE return baseline(train, held), accuracy(m, train), accuracy(m, held) def region_z(source): for b in {x["region_name"] for x in DATA}: v = [x["mean_consumption"] for x in source if x["region_name"] == b] o = sum(v) / len(v) s = max(1e-6, (sum((q - o) ** 2 for q in v) / len(v)) ** 0.5) for x in DATA: if x["region_name"] == b: x["region_z"] = round((x["mean_consumption"] - o) / s, 3) HONEST = measure(BASE, TRAIN, TEST) region_z(TRAIN) S1H = measure(BASE + ["region_z"], TRAIN, TEST) region_z(DATA) RESULTS.append(("1 scaling before the split", S1H, measure(BASE + ["region_z"], TRAIN, TEST))) for x in DATA: r, t = rng(SEED + 71000 + x["id"]), 0 for b in REGION: t = b[2] if x["region_name"] == b[0] else t x["fourth"] = 0.0 if (x["suspicious"] and r() < 0.55) else round( t * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 + 0.45 * x["suspicious"]), 2) RESULTS.append(("2 feature that sees the future", HONEST, measure(BASE + ["fourth"], TRAIN, TEST))) TWINS = [] for x in DATA: for c in range(3): y = dict(x) y["identity"], y["id"] = x["id"], x["id"] + 50000 * c TWINS.append(y) D = shuffle(TWINS, SEED + 93000) D_TRAIN, D_HELD = D[:int(len(D) * 0.6)], D[int(len(D) * 0.8):] TWIN_IDS = {x["identity"] for x in D_TRAIN} WITH_TWIN = [x for x in D_HELD if x["identity"] in TWIN_IDS] NO_TWIN = [x for x in D_HELD if x["identity"] not in TWIN_IDS] RESULTS.append(("3 duplicated record", HONEST, measure(BASE, D_TRAIN, D_HELD))) for x in DATA: r = rng(SEED + 81000 + x["id"]) x["audit"] = int(r() < (0.88 if x["suspicious"] else 0.06)) RESULTS.append(("4 feature derived from the target", HONEST, measure(BASE + ["audit"], TRAIN, TEST))) print(f"{'':<34} {'baseline':>8} {'train':>9} {'honest':>8} {'leaky':>10} {'inflation':>9}") print(f"{'honest line':<34} {HONEST[0]:>8.4f} {HONEST[1]:>9.4f} {HONEST[2]:>8.4f}") for name, h, s in RESULTS: print(f"{name:<34} {s[0]:>8.4f} {s[1]:>9.4f} {h[2]:>8.4f} {s[2]:>10.4f} " f"{s[2] - h[2]:>+9.4f}") print(f"\nduplicated set {len(D)} rows; held-out {len(D_HELD)}, with a twin in train " f"{len(WITH_TWIN)}, without {len(NO_TWIN)}") print(f"inside leak 3: on rows with a twin {measure(BASE, D_TRAIN, WITH_TWIN)[2]:.4f}, " f"on rows without a twin {measure(BASE, D_TRAIN, NO_TWIN)[2]:.4f}") ``` ``` baseline train honest leaky inflation honest line 0.7579 0.8466 0.7778 1 scaling before the split 0.7579 0.8492 0.7659 0.7778 +0.0119 2 feature that sees the future 0.7579 0.9312 0.7778 0.8532 +0.0754 3 duplicated record 0.7857 0.8646 0.7778 0.8320 +0.0542 4 feature derived from the target 0.7579 0.9590 0.7778 0.9206 +0.1429 duplicated set 3780 rows; held-out 756, with a twin in train 626, without 130 inside leak 3: on rows with a twin 0.8514, on rows without a twin 0.7385 ``` All four values of the inflation column **have the same sign**: +0.0119, +0.0754, +0.0542, +0.1429. This is not a coincidence but the definition of leakage: information that mixes into the held-out set always works in the model's favor. The constancy of the sign is the first mark that tells leakage from noise. The second column points to a trap. The training number also rises from 0.8466 to 0.9590, but this says nothing on its own — the second lesson showed that raising capacity raises the same column. The signature of leakage is the rise of **the held-out number**. The ranking is the reverse of the ranking of danger. The largest inflation is produced by the feature derived from the target: with 0.1429 it carries the model to 0.9206. But the name of that column is `audit` and it is likely that someone will ask "when is this column written". The smallest inflation comes from scaling before the split: 0.0119, that is, 3 subscribers. No column has been added, the code looks right and a difference of 1.2 points is not questioned in any report. The danger of leakage is inversely proportional not to its size but to its **visibility**. The third leak can be read inside itself. In the duplicated set, 626 of the 756 rows of the held-out set have their twin in the training set; the model gives 0.8514 on these rows and 0.7385 on the 130 rows that have no twin. The split separated the rows but did not separate the **subscriber**, and to see this by looking at the split ratio is impossible. In the second leak the fourth-period reading is a real measurement; the problem is not the wrongness of the measurement but its **time**. For each column in the set a single question is enough: is this value available at the moment the prediction will be given. ## A Model That Has Learned Nothing All four leaks inflated a real model. The next question is harsher: can leakage carry a model that has **learned nothing** above the baseline. The code below discards all the features of the duplicated set and puts six random columns in their place; nothing is left to learn. ```python for x in TWINS: r = rng(SEED + 31000 + x["identity"]) for j in range(6): x["g" + str(j)] = round(r(), 4) FIELDS = ["g" + str(j) for j in range(6)] print(f"{'capacity':>11} {'train':>9} {'held-out':>9} {'with twin':>9} {'no twin':>8}") print(f"{'baseline':>11} {baseline(D_TRAIN, D_TRAIN):>9.4f} {baseline(D_TRAIN, D_HELD):>9.4f} " f"{baseline(D_TRAIN, WITH_TWIN):>9.4f} {baseline(D_TRAIN, NO_TWIN):>8.4f}") for dp in (8, 12, 16): m = tree(D_TRAIN, dp) print(f"{'depth ' + str(dp):>11} {accuracy(m, D_TRAIN):>9.4f} {accuracy(m, D_HELD):>9.4f} " f"{accuracy(m, WITH_TWIN):>9.4f} {accuracy(m, NO_TWIN):>8.4f}") FIELDS = BASE ``` ``` capacity train held-out with twin no twin baseline 0.7787 0.7857 0.7796 0.8154 depth 8 0.8589 0.8228 0.8419 0.7308 depth 12 0.9528 0.8757 0.9073 0.7231 depth 16 0.9832 0.9061 0.9377 0.7538 ``` The depth 12 row is the measured form of the second claim of the curriculum. The model reads 0.8757 on the held-out set and beats the baseline of 0.7857 by **9.0 points**. If this number stands alone in a report it looks defensible: it has beaten the baseline, it has been measured on the held-out set, both conditions the course asks for are met. Yet the six columns the model looks at are random numbers. The difference shows in the last two columns. On the 626 rows whose twin is in training the model gives 0.9073, on the 130 rows that have no twin 0.7231, and the baseline of those 130 rows is 0.8154. On the rows where it has seen the twin the model recalls its memorization, and on the ones it has not seen it stays 9.2 points below the baseline. The only thing it has learned is which label sticks to certain combinations of six random numbers. The capacity column confirms this: the held-out number is 0.8228 at depth 8, 0.8757 at 12 and 0.9061 at 16. The curve of the second lesson works here **with the opposite sign**. This is the most useful auditing rule the course can give — that overfitting is not penalized on the held-out set is the most reliable sign that the set was not held out. The rule of the course is completed here: a performance measured on a set believed to be held out may also be memorization. The split itself is a decision to be audited, and the audit is made by looking not at the split ratio but at the split **unit** and at the **time** of each column. ## Summary - The inflation of all four kinds of leakage has the same sign: scaling before the split +0.0119, a feature that sees the future +0.0754, a duplicated record +0.0542, a feature derived from the target +0.1429. - The signature of leakage is the rise of the held-out number, not of the training number; capacity also raises the training number. - The size of the inflation is inversely proportional to its visibility; and because in a duplicated record the split separates the rows but not the subscriber, the accuracy is 0.8514 on the 626 rows whose twin is in training and 0.7385 on the 130 rows whose twin is not. - A model whose features have been replaced with six random columns reads 0.8757 on the held-out set and beats the baseline of 0.7857 by 9.0 points; on the rows whose twin it has not seen it stays 9.2 points below its baseline. - As capacity rises the leaky measurement improves (0.8228, 0.8757, 0.9061); that overfitting is not penalized on the held-out set is the most reliable sign that the set was not held out. ## Course Wrap-Up The eleven lessons stand in a single table; every number was read from its own lesson. The first six lessons use the 883/377 split, the last five the 756/252/252 split; the baseline therefore changes from row to row. | Lesson | Baseline | Training | Held-out set | |---|---|---|---| | Concept 01 What Is Machine Learning | most frequent class 0.7533 | tree 0.8709 | tree 0.8594 | | Concept 02 Types of Learning | most frequent class 0.7533 | supervised 0.8709 | supervised 0.8594 | | Concept 03 Self-Supervised Learning | most frequent class 0.7533 | with representation 0.8313 | with representation 0.8037 | | Concept 04 Application Areas | most frequent class 0.7533 | tree 0.8709 | tree 0.8594 | | Concept 05 Machine Learning Workflow | random order 14 illicit cases | ranking 30 illicit cases | ranking 30 illicit cases | | Concept 06 Roles | field procedure 13 illicit cases | — | model order 30 illicit cases | | Generalization 01 Training, Validation and Test Sets | most frequent class 0.7579 | depth 6 0.8466 | depth 6 0.7778 | | Generalization 02 Overfitting and Underfitting | most frequent class 0.8135 | depth 14 0.9352 | depth 14 0.7302 | | Generalization 03 Bias–Variance Tradeoff | mean 6.419 m³ | depth 3 5.105 m³ | depth 3 5.380 m³ | | Generalization 04 Regularization Intuition | mean 6.556 m³ | penalty 100 4.920 m³ | penalty 100 5.559 m³ | | Generalization 05 Data Leakage | most frequent class 0.7579 | leaky 0.9590 | leaky 0.9206 | The row of Concept 04 is the only classification one among the four problems in that lesson. In the row of Concept 06 the training column is empty: that lesson counts not the separation of training from the held-out set but the reading of the same held-out measure against three different baselines in three job definitions — 13, 23 and 21 illicit cases, gain 17, 7 and 9. The row of Generalization 05 is deliberately leaky. The difference between the third and the fourth column is **the rule of the course**: a performance not measured on a held-out set is not performance but memorization. In no row is the training number worse than the held-out one. The second column is **the rule of the curriculum**: a model is as good as the amount by which it beats a procedure dumber than itself, and a performance whose baseline is not written counts as unmeasured. The highest accuracy, 0.9206, says nothing; the lowest, 0.7302, says something, because it stays 8.3 points below its baseline. Eleven lessons established the separation and wrote the baseline next to every number. But the **features** given to the model were always taken as ready: the region had been turned into an indicator column and three readings into a mean from the start. Which column is to be turned into what kind of number, how much that conversion moves the difference from the baseline, and which conversion carries a silent leak like the one in this lesson were never asked. The next course, Data Preparation and Feature Engineering, takes up these questions.