Data Leakage

The previous lesson silently made one decision correctly: the scale of the columns was computed from the training set only. The decision can be reversed. When the scale is computed from the whole set, the rows of the held-out set contribute to producing the numbers the model will see, and the held-out set stops being held out by that amount.

This is the quietest form of data leakage. Leakage is the mixing of information that will not be available at prediction time into the training or measurement path, and it always pushes the same way: the held-out number rises. This lesson applies the four kinds one by one, measures the inflation of each against its own honest counterpart and counts the second claim of the curriculum — the difference does not always come from learning.

• GE31. The set, the label, the split, the seed and the measure are the same throughout the course. The held-out set is the test set: 252 rows. The honest line is the model of the first lesson: the depth 6 decision tree, baseline 0.7579, 0.8466 on training, 0.7778 on test.
• GE32. The first leak is scaling by region: the region mean and scatter are computed from the training set alone in the honest form and from the whole set in the leaky form.
• GE33. The second leak is the fourth-period reading and is fictional: it is formed after the label is written, for suspicious subscribers it comes out zero with probability 0.55, and if it does not, its value is 1.45 times as much.
• GE34. The third leak is a join error: every subscriber record is produced three times, 1,260 subscribers rise to 3,780 rows and the split is made at the row level.
• GE35. The fourth leak is the audit team assignment and is fictional: it is one with probability 0.88 for suspicious subscribers and 0.06 for the others, and it is recorded after the label.
• GE36. The inflation is always measured against its own honest counterpart; the counterpart of the first leak is the honestly computed form of the same feature.
• GE37. In the last section the features are replaced with six random columns. These columns are unrelated to the label and identical across the three copies of a subscriber.

Four Leaks, Four Inflations

The setup is the same as in the first lesson of the course and produces the same set with the same seed.

# generalization.py — MODEL. The regional measurement network (a fictional setup) of the Data Analytics curriculum is generated with the same
# seed; readings are summarized at the subscriber level and a FICTIONAL illicit-use/fault suspicion label
# is added on top.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS = []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
DATA = []
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    for b in REGION:
        x["reg_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "reg_north", "reg_south", "reg_east", "reg_west", "reg_central"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, TEST = K[:756], K[1008:]


def gini(s):
    p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0
    return 2 * p * (1 - p)


def tree(s, depth, min_size=2):
    p = sum(x["suspicious"] for x in s) / len(s)
    best = None
    if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0:
        for f in FIELDS:
            values = sorted({x[f] for x in s})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in s if x[f] < v]
                right = [x for x in s if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, min_size),
            "right": tree(best[4], depth - 1, min_size)}


def predict(d, x):
    while "prediction" not in d:
        d = d["left"] if x[d["field"]] < d["threshold"] else d["right"]
    return d["prediction"]


def accuracy(d, s):
    return sum(predict(d, x) == x["suspicious"] for x in s) / len(s)


def baseline(train, s):
    c = int(sum(x["suspicious"] for x in train) * 2 > len(train))
    return sum(x["suspicious"] == c for x in s) / len(s)

The four leaks are set up in turn and each is measured against its own honest counterpart.

BASE, RESULTS = list(FIELDS), []
for x in DATA:
    x["region_name"] = [b[0] for b in REGION if x["reg_" + b[0]]][0]


def measure(fields, train, held, depth=6):
    global FIELDS
    FIELDS = fields
    m = tree(train, depth)
    FIELDS = BASE
    return baseline(train, held), accuracy(m, train), accuracy(m, held)


def region_z(source):
    for b in {x["region_name"] for x in DATA}:
        v = [x["mean_consumption"] for x in source if x["region_name"] == b]
        o = sum(v) / len(v)
        s = max(1e-6, (sum((q - o) ** 2 for q in v) / len(v)) ** 0.5)
        for x in DATA:
            if x["region_name"] == b:
                x["region_z"] = round((x["mean_consumption"] - o) / s, 3)


HONEST = measure(BASE, TRAIN, TEST)
region_z(TRAIN)
S1H = measure(BASE + ["region_z"], TRAIN, TEST)
region_z(DATA)
RESULTS.append(("1 scaling before the split", S1H, measure(BASE + ["region_z"], TRAIN, TEST)))

for x in DATA:
    r, t = rng(SEED + 71000 + x["id"]), 0
    for b in REGION:
        t = b[2] if x["region_name"] == b[0] else t
    x["fourth"] = 0.0 if (x["suspicious"] and r() < 0.55) else round(
        t * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 + 0.45 * x["suspicious"]), 2)
RESULTS.append(("2 feature that sees the future", HONEST, measure(BASE + ["fourth"], TRAIN, TEST)))

TWINS = []
for x in DATA:
    for c in range(3):
        y = dict(x)
        y["identity"], y["id"] = x["id"], x["id"] + 50000 * c
        TWINS.append(y)
D = shuffle(TWINS, SEED + 93000)
D_TRAIN, D_HELD = D[:int(len(D) * 0.6)], D[int(len(D) * 0.8):]
TWIN_IDS = {x["identity"] for x in D_TRAIN}
WITH_TWIN = [x for x in D_HELD if x["identity"] in TWIN_IDS]
NO_TWIN = [x for x in D_HELD if x["identity"] not in TWIN_IDS]
RESULTS.append(("3 duplicated record", HONEST, measure(BASE, D_TRAIN, D_HELD)))

for x in DATA:
    r = rng(SEED + 81000 + x["id"])
    x["audit"] = int(r() < (0.88 if x["suspicious"] else 0.06))
RESULTS.append(("4 feature derived from the target", HONEST,
                measure(BASE + ["audit"], TRAIN, TEST)))

print(f"{'':<34} {'baseline':>8} {'train':>9} {'honest':>8} {'leaky':>10} {'inflation':>9}")
print(f"{'honest line':<34} {HONEST[0]:>8.4f} {HONEST[1]:>9.4f} {HONEST[2]:>8.4f}")
for name, h, s in RESULTS:
    print(f"{name:<34} {s[0]:>8.4f} {s[1]:>9.4f} {h[2]:>8.4f} {s[2]:>10.4f} "
          f"{s[2] - h[2]:>+9.4f}")
print(f"\nduplicated set {len(D)} rows; held-out {len(D_HELD)}, with a twin in train "
      f"{len(WITH_TWIN)}, without {len(NO_TWIN)}")
print(f"inside leak 3: on rows with a twin {measure(BASE, D_TRAIN, WITH_TWIN)[2]:.4f}, "
      f"on rows without a twin {measure(BASE, D_TRAIN, NO_TWIN)[2]:.4f}")

                                   baseline     train   honest      leaky inflation
honest line                          0.7579    0.8466   0.7778
1 scaling before the split           0.7579    0.8492   0.7659     0.7778   +0.0119
2 feature that sees the future       0.7579    0.9312   0.7778     0.8532   +0.0754
3 duplicated record                  0.7857    0.8646   0.7778     0.8320   +0.0542
4 feature derived from the target    0.7579    0.9590   0.7778     0.9206   +0.1429

duplicated set 3780 rows; held-out 756, with a twin in train 626, without 130
inside leak 3: on rows with a twin 0.8514, on rows without a twin 0.7385

All four values of the inflation column have the same sign: +0.0119, +0.0754, +0.0542, +0.1429. This is not a coincidence but the definition of leakage: information that mixes into the held-out set always works in the model’s favor. The constancy of the sign is the first mark that tells leakage from noise.

The second column points to a trap. The training number also rises from 0.8466 to 0.9590, but this says nothing on its own — the second lesson showed that raising capacity raises the same column. The signature of leakage is the rise of the held-out number.

The ranking is the reverse of the ranking of danger. The largest inflation is produced by the feature derived from the target: with 0.1429 it carries the model to 0.9206. But the name of that column is audit and it is likely that someone will ask “when is this column written”. The smallest inflation comes from scaling before the split: 0.0119, that is, 3 subscribers. No column has been added, the code looks right and a difference of 1.2 points is not questioned in any report. The danger of leakage is inversely proportional not to its size but to its visibility.

The third leak can be read inside itself. In the duplicated set, 626 of the 756 rows of the held-out set have their twin in the training set; the model gives 0.8514 on these rows and 0.7385 on the 130 rows that have no twin. The split separated the rows but did not separate the subscriber, and to see this by looking at the split ratio is impossible.

In the second leak the fourth-period reading is a real measurement; the problem is not the wrongness of the measurement but its time. For each column in the set a single question is enough: is this value available at the moment the prediction will be given.

A Model That Has Learned Nothing

All four leaks inflated a real model. The next question is harsher: can leakage carry a model that has learned nothing above the baseline. The code below discards all the features of the duplicated set and puts six random columns in their place; nothing is left to learn.

for x in TWINS:
    r = rng(SEED + 31000 + x["identity"])
    for j in range(6):
        x["g" + str(j)] = round(r(), 4)
FIELDS = ["g" + str(j) for j in range(6)]
print(f"{'capacity':>11} {'train':>9} {'held-out':>9} {'with twin':>9} {'no twin':>8}")
print(f"{'baseline':>11} {baseline(D_TRAIN, D_TRAIN):>9.4f} {baseline(D_TRAIN, D_HELD):>9.4f} "
      f"{baseline(D_TRAIN, WITH_TWIN):>9.4f} {baseline(D_TRAIN, NO_TWIN):>8.4f}")
for dp in (8, 12, 16):
    m = tree(D_TRAIN, dp)
    print(f"{'depth ' + str(dp):>11} {accuracy(m, D_TRAIN):>9.4f} {accuracy(m, D_HELD):>9.4f} "
          f"{accuracy(m, WITH_TWIN):>9.4f} {accuracy(m, NO_TWIN):>8.4f}")
FIELDS = BASE

   capacity     train  held-out with twin  no twin
   baseline    0.7787    0.7857    0.7796   0.8154
    depth 8    0.8589    0.8228    0.8419   0.7308
   depth 12    0.9528    0.8757    0.9073   0.7231
   depth 16    0.9832    0.9061    0.9377   0.7538

The depth 12 row is the measured form of the second claim of the curriculum. The model reads 0.8757 on the held-out set and beats the baseline of 0.7857 by 9.0 points. If this number stands alone in a report it looks defensible: it has beaten the baseline, it has been measured on the held-out set, both conditions the course asks for are met. Yet the six columns the model looks at are random numbers.

The difference shows in the last two columns. On the 626 rows whose twin is in training the model gives 0.9073, on the 130 rows that have no twin 0.7231, and the baseline of those 130 rows is 0.8154. On the rows where it has seen the twin the model recalls its memorization, and on the ones it has not seen it stays 9.2 points below the baseline. The only thing it has learned is which label sticks to certain combinations of six random numbers.

The capacity column confirms this: the held-out number is 0.8228 at depth 8, 0.8757 at 12 and 0.9061 at 16. The curve of the second lesson works here with the opposite sign. This is the most useful auditing rule the course can give — that overfitting is not penalized on the held-out set is the most reliable sign that the set was not held out.

The rule of the course is completed here: a performance measured on a set believed to be held out may also be memorization. The split itself is a decision to be audited, and the audit is made by looking not at the split ratio but at the split unit and at the time of each column.

Summary

• The inflation of all four kinds of leakage has the same sign: scaling before the split +0.0119, a feature that sees the future +0.0754, a duplicated record +0.0542, a feature derived from the target +0.1429.
• The signature of leakage is the rise of the held-out number, not of the training number; capacity also raises the training number.
• The size of the inflation is inversely proportional to its visibility; and because in a duplicated record the split separates the rows but not the subscriber, the accuracy is 0.8514 on the 626 rows whose twin is in training and 0.7385 on the 130 rows whose twin is not.
• A model whose features have been replaced with six random columns reads 0.8757 on the held-out set and beats the baseline of 0.7857 by 9.0 points; on the rows whose twin it has not seen it stays 9.2 points below its baseline.
• As capacity rises the leaky measurement improves (0.8228, 0.8757, 0.9061); that overfitting is not penalized on the held-out set is the most reliable sign that the set was not held out.

Course Wrap-Up

The eleven lessons stand in a single table; every number was read from its own lesson. The first six lessons use the 883/377 split, the last five the 756/252/252 split; the baseline therefore changes from row to row.

Lesson	Baseline	Training	Held-out set
Concept 01 What Is Machine Learning	most frequent class 0.7533	tree 0.8709	tree 0.8594
Concept 02 Types of Learning	most frequent class 0.7533	supervised 0.8709	supervised 0.8594
Concept 03 Self-Supervised Learning	most frequent class 0.7533	with representation 0.8313	with representation 0.8037
Concept 04 Application Areas	most frequent class 0.7533	tree 0.8709	tree 0.8594
Concept 05 Machine Learning Workflow	random order 14 illicit cases	ranking 30 illicit cases	ranking 30 illicit cases
Concept 06 Roles	field procedure 13 illicit cases	—	model order 30 illicit cases
Generalization 01 Training, Validation and Test Sets	most frequent class 0.7579	depth 6 0.8466	depth 6 0.7778
Generalization 02 Overfitting and Underfitting	most frequent class 0.8135	depth 14 0.9352	depth 14 0.7302
Generalization 03 Bias–Variance Tradeoff	mean 6.419 m³	depth 3 5.105 m³	depth 3 5.380 m³
Generalization 04 Regularization Intuition	mean 6.556 m³	penalty 100 4.920 m³	penalty 100 5.559 m³
Generalization 05 Data Leakage	most frequent class 0.7579	leaky 0.9590	leaky 0.9206

The row of Concept 04 is the only classification one among the four problems in that lesson. In the row of Concept 06 the training column is empty: that lesson counts not the separation of training from the held-out set but the reading of the same held-out measure against three different baselines in three job definitions — 13, 23 and 21 illicit cases, gain 17, 7 and 9. The row of Generalization 05 is deliberately leaky.

The difference between the third and the fourth column is the rule of the course: a performance not measured on a held-out set is not performance but memorization. In no row is the training number worse than the held-out one.

The second column is the rule of the curriculum: a model is as good as the amount by which it beats a procedure dumber than itself, and a performance whose baseline is not written counts as unmeasured. The highest accuracy, 0.9206, says nothing; the lowest, 0.7302, says something, because it stays 8.3 points below its baseline.

Eleven lessons established the separation and wrote the baseline next to every number. But the features given to the model were always taken as ready: the region had been turned into an indicator column and three readings into a mean from the start. Which column is to be turned into what kind of number, how much that conversion moves the difference from the baseline, and which conversion carries a silent leak like the one in this lesson were never asked. The next course, Data Preparation and Feature Engineering, takes up these questions.