Training, Validation and Test Sets

Over six lessons, the number the model produced was always measured on the same data. A hand-written rule set and a procedure that extracts its rules from data were put side by side, the number of labels required by four learning types was counted, the difference added by a pretext task was measured, four different baselines were chosen for four problem types, the returns in the workflow were counted and the same problem was shown turning into three different outputs in three roles. In all of them the measure was read from data the model had seen. What would happen on data the model had not seen was never asked.

This lesson asks that question and sets the rule of the course: performance that has not been measured on a held-out set is not performance, it is memorization. The question has three faces. How many numbers does the same model give on three sets, how many times can each of these sets be looked at, and by what rule are the sets separated.

• GE1. The dataset is inherited from the regional measurement network of the Data Analytics curriculum and is fictional: a long-format readings table of 3,199 rows, 1,329 subscribers, three periods, five regions, seed 20260218.
• GE2. The label is added in this curriculum and is fictional: a subscriber suspected of illicit use or a fault. It is not a real inspection record; it is produced by applying a threshold to a hidden score.
• GE3. The modeling unit is the subscriber. Readings are summarized at the subscriber level; 69 subscribers who produced no readings at all are left out, and 1,260 subscribers remain.
• GE4. The features come from seven fields: region (five binary columns), household size, satisfaction, period count, mean consumption, volatility, zero readings.
• GE5. The measure is accuracy: the proportion of subscribers classified correctly. The classes are imbalanced, and how much accuracy hides this is not the subject of this course.
• GE6. The baseline is telling every subscriber the most frequent class in the training set. This procedure learns nothing and looks at no feature.
• GE7. The split ratio is 60/20/20 and it is a decision. The sets are split once and stay the same until the end of the lesson.
• GE8. The subscriber number is the order of joining the network. The suspicion rate rises along this order; this rise was placed in the setup on purpose and it is what the time-ordered split measures.

Three Sets, Three Numbers

The setup generates the dataset, adds the label, splits it into three and trains a single decision tree. A tree is a model: at each node it chooses a feature and a threshold, splits the set in two and states the majority class at the leaf. Depth 6 is a choice, and its justification is left to the next lesson.

# generalization.py — MODEL: the regional measurement network (a fictional setup) of the Data Analytics curriculum is generated with the same
# seed; readings are summarized at the subscriber level and a FICTIONAL illicit-use/fault suspicion label
# is added on top.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS = []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
READINGS = {}
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()

DATA = []
for k in SUBSCRIBERS:
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    for b in REGION:
        x["reg_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "reg_north", "reg_south", "reg_east", "reg_west", "reg_central"]
print(f"readings {sum(len(v) for v in READINGS.values())}, subscribers {len(SUBSCRIBERS)}, "
      f"labeled subscribers {len(DATA)}, suspicious {sum(x['suspicious'] for x in DATA)}")


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


def split_three(K):
    a, b = int(len(K) * 0.6), int(len(K) * 0.8)
    return K[:a], K[a:b], K[b:]


TRAIN, VAL, TEST = split_three(shuffle(DATA, SEED + 90000))
print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}")


def gini(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, min_size=2):
    p = sum(x["suspicious"] for x in rows) / len(rows)
    best = None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in FIELDS:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, min_size),
            "right": tree(best[4], depth - 1, min_size)}


def predict(node, x):
    while "prediction" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["prediction"]


def accuracy(node, rows):
    return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows)


def baseline(train, rows):
    c = int(sum(x["suspicious"] for x in train) * 2 > len(train))
    return sum(x["suspicious"] == c for x in rows) / len(rows)


MODEL = tree(TRAIN, 6)
print(f"\n{'procedure':<30} {'train':>8} {'validation':>10} {'test':>8}")
print(f"{'baseline (most frequent class)':<30} {baseline(TRAIN, TRAIN):>8.4f} "
      f"{baseline(TRAIN, VAL):>10.4f} {baseline(TRAIN, TEST):>8.4f}")
print(f"{'decision tree, depth 6':<30} {accuracy(MODEL, TRAIN):>8.4f} "
      f"{accuracy(MODEL, VAL):>10.4f} {accuracy(MODEL, TEST):>8.4f}")

readings 3199, subscribers 1329, labeled subscribers 1260, suspicious 276
train 756, validation 252, test 252

procedure                         train validation     test
baseline (most frequent class)   0.7778     0.8135   0.7579
decision tree, depth 6           0.8466     0.7897   0.7778

The first row of the table says something unexpected. The baseline is also three numbers. The procedure that learns nothing, looks at no feature and gives every subscriber the same answer gives 0.7778 accuracy on the training set, 0.8135 on the validation set and 0.7579 on the test set. The 0.0556 gap between them comes entirely from the randomness of the split: when the suspicious rate moves a few points in a set of 252 subscribers, the accuracy of the most frequent class moves by the same amount. Comparing a model number with the baseline of another set is therefore an error; the baseline is recalculated on every set.

The second row is the real number of the lesson. The same model, the same measure, three sets, three numbers — and two different signs. On the training set the model beats its own baseline by 6.9 points. On the validation set it falls 2.4 points below the baseline. On the test set it beats the baseline by 2.0 points. The 0.8466 read from the training set is not miscalculated; it is calculated correctly but answers the wrong question. That number measures not what the model will do in the future but how much of the past it has memorized.

What remains is this question: if one held-out set is enough, why are there two held-out sets.

How Many Times Can the Test Set Be Looked At

The difference between the validation set and the test set lies not in the measure but in the use. The validation set is set aside for selection, the test set for reporting. The difference between them can be measured. The code below produces 48 candidates from the same procedure: all are depth 6 decision trees, and their only difference is which 85% of the training set they saw. There is no real quality difference among them.

POOL = []
for a in range(48):
    sub = shuffle(TRAIN, SEED + 91000 + a * 17)[:int(len(TRAIN) * 0.85)]
    m = tree(sub, 6)
    POOL.append((accuracy(m, VAL), accuracy(m, TEST)))
rs = rng(SEED + 92000)
print(f"{len(POOL)} candidates, all the same procedure; mean validation "
      f"{sum(x[0] for x in POOL) / len(POOL):.4f}, best single candidate "
      f"{max(x[0] for x in POOL):.4f}, mean test "
      f"{sum(x[1] for x in POOL) / len(POOL):.4f}")
print(f"\n{'candidates seen':>15} {'selected validation':>19} "
      f"{'same model test':>15} {'difference':>10}")
for k in (1, 2, 4, 8, 16, 32):
    d = s = 0.0
    for _ in range(100):
        best = max([POOL[int(rs() * len(POOL))] for _ in range(k)], key=lambda t: t[0])
        d, s = d + best[0], s + best[1]
    print(f"{k:>15} {d / 100:>19.4f} {s / 100:>15.4f} {(d - s) / 100:>+10.4f}")

48 candidates, all the same procedure; mean validation 0.7875, best single candidate 0.8214, mean test 0.7657

candidates seen selected validation same model test difference
              1              0.7860          0.7650    +0.0211
              2              0.7982          0.7699    +0.0283
              4              0.8069          0.7693    +0.0375
              8              0.8113          0.7742    +0.0371
             16              0.8149          0.7756    +0.0394
             32              0.8177          0.7767    +0.0411

What is read is not the level of the columns but the growth of the difference between them. There is already a constant gap of 0.0218 between validation and test: the mean validation of the forty-eight candidates is 0.7875, the mean test is 0.7657, and this difference stands even when no selection is made. So this gap does not come from selection. What comes from selection is this: as the number of candidates looked at goes from 1 to 32, the validation number of the selected model rises from 0.7860 to 0.8177, that is, by 3.2 points; the test number of the same models rises by only 1.2 points. The 2.0 points in between are not something any model gained; they are a number produced by the selection itself.

The harshest row is in the first output. The mean validation of the forty-eight candidates is 0.7875, and this is below 0.8135, the baseline of the validation set: an average candidate is worse than saying “none is suspicious.” Yet the best single candidate reads 0.8214 and appears to have beaten the baseline. A study that tries forty-eight candidates and reports the best one can make a procedure that does not beat the baseline look as though it has. This is the first small example of the second claim of the curriculum, and its full form will be measured in the last lesson of the course.

The rule follows from this: the moment a set is looked at and a decision is made according to it, that set turns into a validation set. The test set is looked at once, at the very end, for a single model. A second look makes it a validation set and inflates the reported number by the selection’s own share.

The Split Is a Decision

The split so far was random: 1,260 subscribers were shuffled and divided into three. This assumes that every row in the dataset can replace any other. If the network has grown over time and the subscribers who joined later differ from the earlier ones, the assumption does not hold. The code below splits the same data this time by order of registration: the first 60% training, the next 20% validation, the last 20% test.

TIME_TRAIN, TIME_VAL, TIME_TEST = split_three(sorted(DATA, key=lambda x: x["id"]))
TIME_MODEL = tree(TIME_TRAIN, 6)
print(f"{'split':<12} {'suspicious rate':>15} {'baseline':>8} {'tree':>8}")
for name, s in (("train", TIME_TRAIN), ("validation", TIME_VAL), ("test", TIME_TEST)):
    print(f"{name:<12} {sum(x['suspicious'] for x in s) / len(s):>15.4f} "
          f"{baseline(TIME_TRAIN, s):>8.4f} {accuracy(TIME_MODEL, s):>8.4f}")
print(f"\nrandom split test: baseline {baseline(TRAIN, TEST):.4f}, tree "
      f"{accuracy(MODEL, TEST):.4f}, difference "
      f"{accuracy(MODEL, TEST) - baseline(TRAIN, TEST):+.4f}")
print(f"time-ordered split test: baseline {baseline(TIME_TRAIN, TIME_TEST):.4f}, tree "
      f"{accuracy(TIME_MODEL, TIME_TEST):.4f}, difference "
      f"{accuracy(TIME_MODEL, TIME_TEST) - baseline(TIME_TRAIN, TIME_TEST):+.4f}")

split        suspicious rate baseline     tree
train                 0.1561   0.8439   0.9074
validation            0.2738   0.7262   0.7381
test                  0.3532   0.6468   0.6865

random split test: baseline 0.7579, tree 0.7778, difference +0.0198
time-ordered split test: baseline 0.6468, tree 0.6865, difference +0.0397

The suspicious rate is 0.1561 on the training set and 0.3532 on the test set. In the random split these three numbers were close to each other, because shuffling distributed the later-joining subscribers equally among the three sets. The time-ordered split gathers them in a single set and sets up the real situation the model will face: learn from the past, predict the future.

The result is read in two directions at once, and both are correct. The test accuracy falls from 0.7778 to 0.6865, that is, by 9.1 points — the number given by the random split was too optimistic for a subscriber newly joining the network. But the difference above the baseline rises from 0.0198 to 0.0397, that is, it doubles. The reason is that the baseline falls too: the procedure that states the most frequent class of the training set drops to 0.6468 on a test set where the suspicious rate has risen.

Someone who looks only at the model number takes the time-ordered split for a decision that “breaks the model.” Someone who writes the baseline next to it sees the opposite: the set that got harder penalized the baseline more than the model, and the difference the model actually adds is larger there. The rule of the curriculum is at work in this row — a model is as good as how much it beats a dumber procedure, and that procedure is always remeasured on the same set.

Summary

• The same depth 6 decision tree gives 0.8466 accuracy on the training set, 0.7897 on the validation set and 0.7778 on the test set; since the baseline is also three different numbers on the three sets, 0.7778, 0.8135 and 0.7579, the baseline is recalculated on every set.
• The model beats its baseline by 6.9 points on the training set, falls 2.4 points below the baseline on the validation set and beats it by 2.0 points on the test set; a number read from the data it has seen is calculated correctly but answers the wrong question.
• Selecting the best among 48 candidates produced by the same procedure raises the validation number by 3.2 points as the candidates looked at go from 1 to 32, while raising the test number by only 1.2 points; the 2.0 points in between are the selection’s own share.
• The mean validation of the forty-eight candidates is below the baseline at 0.7875, but the best candidate reads 0.8214 and appears to have beaten the baseline; selection can make a procedure that does not beat the baseline look as though it has.
• On the time-ordered split the test accuracy falls to 0.6865, but since the baseline also falls to 0.6468 the difference the model adds rises from 0.0198 to 0.0397; the split decision moves both numbers at once.

Next Step

In this lesson depth 6 was chosen without justification and the three numbers were read on top of that choice. Yet depth is the model’s capacity: it determines how many distinct rules it can learn. When capacity is lowered, both the training number and the held-out number fall; when it is raised, the training number keeps rising but after a point the held-out number turns back. The next lesson raises capacity step by step, prints the two numbers side by side, writes down the point where the divergence begins and names the two deteriorations: underfitting and overfitting.