Hyperparameter Search

The previous lesson counted the items of the training and pinned all of them; the item that made the largest shift was the starting values. In a real training some of the items are released on purpose: a candidate list is built, all of it is run, and the best is chosen from the validation set.

The search methods were measured in the Evaluation and Tuning topic of the Supervised Learning course — grid, random and model-based search had given 0.7937, 0.7857 and 0.8016 on the test set in the same fifty-candidate budget — and are not repeated here. The only thing that changes is this: there the number of each candidate was read from a single run. The question of the lesson is this: with the number of candidates the same, when the difference between two runs of the same candidate is larger than the difference between candidates, what is the search choosing.

• TP24. The dataset, the network and the baseline are inherited from Lesson 01; in addition a validation set is used (252 rows), and the choice is read from it, the report from the test set.
• TP25. The number of candidates is ten, the budget is equal for every candidate (40 epochs) and the candidates are listed by hand; the choice of the search method is not the subject of this lesson.
• TP26. The tuning space has three axes: hidden units (4, 8, 16), learning rate (0.15, 0.30, 0.60) and batch size (20, 30, 50). Every candidate is run twice and the two runs separate only in their starting values; the other items are pinned as in Lesson 04.
• TP27. Two candidates are counted as indistinguishable if their difference is smaller than the run difference of both; the threshold comes from the measure itself. A tie goes to the first candidate on the list, and this too is a decision.

The Dataset, the Network and the Candidate List

# COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through
# Unsupervised Learning courses is generated with the same seed (20260218);
# the training set is reduced from 756 to 300 rows. The VALIDATION set is also used in this
# lesson: the choice is read from it, the report from the test set. The network comes from
# Lesson 01; the number of hidden units h changes with the candidate.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]
FIELDS = ["area_south", "area_east", "area_west", "area_central",
          "household", "satisfaction", "period", "mean"]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def mix(items, seed):
    r, s = rng(seed), list(items)
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return s


DATA = []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() < 0.046:
        continue
    subscriber_id = 10001 + i
    r, v = rng(SEED + 7000 + subscriber_id), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            b[2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id)
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3]
         + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": household, "satisfaction": rating, "period": len(v),
         "mean": round(mean, 2), "suspicious": int(z > 1.35)}
    for q in REGION[1:]:
        x["area_" + q[0]] = int(b[0] == q[0])
    DATA.append(x)

K = mix(DATA, SEED + 90000)
TRAIN, VALID, TEST = K[:300], K[756:1008], K[1008:]


def table(rows, stats=None):
    X = [[float(s[a]) for a in FIELDS] for s in rows]
    if stats is None:                     # scaling is learned from the training set only
        stats = []
        for j in range(4, 8):
            m = sum(v[j] for v in X) / len(X)
            stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X))))
    for v in X:
        for j, m, sd in stats:
            v[j] = (v[j] - m) / sd
    return X, [s["suspicious"] for s in rows], stats


X_train, y_train, STATS = table(TRAIN)
X_valid, y_valid, _ = table(VALID, STATS)
X_test, y_test, _ = table(TEST, STATS)
BASELINE = sum(v == 0 for v in y_test) / len(y_test)

G = len(FIELDS)


def init_weights(seed, h):
    r = rng(seed)
    return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(h * G)] + [0.0] * h \
        + [(r() * 2 - 1) * math.sqrt(2.0 / h) for _ in range(h)] + [0.0]


def gradient(p, X, y, idx, h):
    g = [0.0] * len(p)
    for k in idx:
        x, hh = X[k], [0.0] * h
        for j in range(h):
            s = p[h * G + j] + sum(p[j * G + i] * x[i] for i in range(G))
            hh[j] = s if s > 0 else 0.0
        s = p[-1] + sum(p[h * G + h + j] * hh[j] for j in range(h))
        d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k]
        g[-1] += d
        for j in range(h):
            g[h * G + h + j] += d * hh[j]
            if hh[j] > 0:
                g[h * G + j] += d * p[h * G + h + j]
                for i in range(G):
                    g[j * G + i] += d * p[h * G + h + j] * x[i]
    return [v / len(idx) for v in g]


def train_model(seed, h, step, batch, epochs):
    p = init_weights(seed, h)
    for t in range(epochs):
        perm = mix(range(len(X_train)), SEED + 700 + t)
        for b in range(0, len(perm) - batch + 1, batch):
            p = [u - step * v for u, v in
                 zip(p, gradient(p, X_train, y_train, perm[b:b + batch], h))]
    return p


def accuracy(p, X, y, h):
    n = 0
    for k in range(len(X)):
        x = X[k]
        o = p[-1] + sum(p[h * G + h + j] * max(0.0, p[h * G + j]
                        + sum(p[j * G + i] * x[i] for i in range(G)))
                        for j in range(h))
        n += (o >= 0.0) == (y[k] == 1)
    return n / len(X)


print(f"fictional: train {len(X_train)} rows (reduced from 756), validation {len(X_valid)}, "
      f"test {len(X_test)}; baseline {BASELINE:.4f}")

fictional: train 300 rows (reduced from 756), validation 252, test 252; baseline 0.7579

Ten Candidates, Two Runs

# TEN CANDIDATES, each with TWO RUNS. The search methods were measured in the Supervised
# Learning course and are not repeated here; the only thing that changes is HOW MANY RUNS
# the number of each candidate is read from. The choice is made on the validation set,
# the report on the test set.
CANDIDATES = [(4, 0.15, 30), (4, 0.30, 30), (8, 0.15, 30), (8, 0.30, 30), (8, 0.60, 30),
              (8, 0.30, 20), (8, 0.30, 50), (16, 0.15, 30), (16, 0.30, 30), (16, 0.60, 30)]
VAL_SCORES, TEST_SCORES = [], []
print(f"{'cand':>4} {'hidden':>6} {'rate':>5} {'batch':>6} {'val r1':>7} {'val r2':>7} "
      f"{'|diff|':>7} {'test r1':>7} {'test r2':>7}")
for no, (h, step, batch) in enumerate(CANDIDATES, 1):
    P = [train_model(SEED + k, h, step, batch, 40) for k in (1, 2)]
    d = [accuracy(p, X_valid, y_valid, h) for p in P]
    s = [accuracy(p, X_test, y_test, h) for p in P]
    VAL_SCORES.append(d)
    TEST_SCORES.append(s)
    print(f"{no:>4} {h:>6} {step:>5.2f} {batch:>6} {d[0]:>7.4f} {d[1]:>7.4f} "
          f"{abs(d[0] - d[1]):>7.4f} {s[0]:>7.4f} {s[1]:>7.4f}")

cand hidden  rate  batch  val r1  val r2  |diff| test r1 test r2
   1      4  0.15     30  0.8135  0.8175  0.0040  0.8095  0.8056
   2      4  0.30     30  0.8016  0.8254  0.0238  0.8095  0.8016
   3      8  0.15     30  0.8135  0.8175  0.0040  0.8016  0.8056
   4      8  0.30     30  0.8095  0.8175  0.0079  0.8095  0.7937
   5      8  0.60     30  0.8016  0.8056  0.0040  0.7857  0.8056
   6      8  0.30     20  0.7976  0.8095  0.0119  0.7976  0.7976
   7      8  0.30     50  0.8095  0.8175  0.0079  0.8016  0.8056
   8     16  0.15     30  0.8294  0.8254  0.0040  0.8135  0.8056
   9     16  0.30     30  0.8294  0.8175  0.0119  0.8095  0.8016
  10     16  0.60     30  0.8016  0.8016  0.0000  0.7976  0.7976

The first validation column gives a familiar result: the two highest numbers are 0.8294 and both belong to the sixteen-unit candidates, so capacity seems to have paid off. The second column breaks this: candidate number two — the smallest network on the list — gives 0.8016 in the first run, then goes up to 0.8254 in the second and draws level with candidate number eight.

What the Search Chooses

# The candidate the search chooses depends on which run it looks from.
SPREAD = max(d[0] for d in VAL_SCORES) - min(d[0] for d in VAL_SCORES)
RUN_GAP = [abs(d[0] - d[1]) for d in VAL_SCORES]
print(f"spread between candidates (run 1, validation) {SPREAD:.4f}")
print(f"difference between the two runs of the same candidate: at most {max(RUN_GAP):.4f}, "
      f"median {sorted(RUN_GAP)[len(RUN_GAP) // 2]:.4f}")
PAIRS = [(i, j) for i in range(len(CANDIDATES)) for j in range(i + 1, len(CANDIDATES))]
tied = sum(1 for i, j in PAIRS
           if abs(VAL_SCORES[i][0] - VAL_SCORES[j][0]) <= max(RUN_GAP[i], RUN_GAP[j]))
flipped = sum(1 for i, j in PAIRS
              if (VAL_SCORES[i][0] - VAL_SCORES[j][0]) * (VAL_SCORES[i][1] - VAL_SCORES[j][1]) < 0)
print(f"of {len(PAIRS)} candidate pairs, {tied} cannot be told apart by the run difference; "
      f"in {flipped} pairs the order FLIPS between the two runs")
for k in (0, 1):
    best = max(range(len(CANDIDATES)), key=lambda i: (VAL_SCORES[i][k], -i))
    print(f"chosen by run {k + 1}: candidate {best + 1} (hidden {CANDIDATES[best][0]}, "
          f"rate {CANDIDATES[best][1]:.2f}, batch {CANDIDATES[best][2]}): validation "
          f"{VAL_SCORES[best][k]:.4f}, test {TEST_SCORES[best][k]:.4f}")
avg = sum(s[0] for s in TEST_SCORES) / len(TEST_SCORES)
print(f"test mean of the ten candidates {avg:.4f}; baseline {BASELINE:.4f}")

spread between candidates (run 1, validation) 0.0317
difference between the two runs of the same candidate: at most 0.0238, median 0.0079
of 45 candidate pairs, 23 cannot be told apart by the run difference; in 7 pairs the order FLIPS between the two runs
chosen by run 1: candidate 8 (hidden 16, rate 0.15, batch 30): validation 0.8294, test 0.8135
chosen by run 2: candidate 2 (hidden 4, rate 0.30, batch 30): validation 0.8254, test 0.8016
test mean of the ten candidates 0.8036; baseline 0.7579

The number of the lesson is in the first two lines. The spread between candidates is 0.0317, the largest difference between two runs of the same candidate is 0.0238; the two are of the same order of magnitude. Twenty-three of the forty-five candidate pairs, that is, more than half, cannot be told apart by the run difference, and in seven pairs the order flips between the two runs.

The search that looks from the first run chooses candidate number eight: sixteen units. The search that looks from the second run chooses candidate number two: four units. The same list, the same budget and two different answers to the question of capacity; the test numbers of the chosen ones are 0.8135 and 0.8016.

The test mean of the ten candidates is 0.8036: the candidate chosen from the first run is 0.0099 above it, the one chosen from the second run 0.0020 below. Depending on which run it looks from, the search earns two and a half subscribers or loses half a subscriber. The tuning budget produced the inflation of the number in the Supervised Learning course; here there is a problem that comes before it — if the order of the search is built from noise, there is no gain to inflate. The remedy is expensive: measuring every candidate with several runs and writing its number as the mean of the runs; the budget goes down from ten candidates to five.

Summary

• The spread between candidates on the validation set is 0.0317 and the largest difference between two runs of a single candidate is 0.0238; the two are of the same order of magnitude.
• Twenty-three of the forty-five candidate pairs cannot be told apart by the run difference, and in seven the order flips.
• The search that looks from the first run chooses the sixteen-unit candidate, the one that looks from the second run the four-unit candidate: the same list gives two different answers to the question of capacity, with test numbers of 0.8135 and 0.8016.
• The test mean of the ten candidates is 0.8036 and the baseline is 0.7579; depending on which run it looks from, the search earns two and a half subscribers or loses half a subscriber.

Course Wrap-Up

The computation of twenty-four lessons is in a single table, and every number was read from its own lesson. The rows do not carry a common unit: the course used three separate fictional datasets and the last column writes which one it was read from. The criterion is accuracy in twenty-three rows and larger is better; the Architectures 10 row is unlabeled, is read by reconstruction error and there smaller is better. The second run is retraining the same setup with only the starting values changed.

The rule of the course is this: the number of a network can come from its training and not from its capacity; a result that does not write that the same capacity gave two different numbers in two runs counts as unmeasured. The third column comes down to a single number: of the twenty-one lessons in which the difference made by capacity or architecture was written in its own lesson, in fourteen the difference of the second run came out larger than that difference — in the convolutional architecture 0.0750 against the 0.0575 added by depth, in multi-head attention 0.0500 against the 0.0250 added by the number of heads, in the perceptron four subscribers against the one subscriber added by capacity. In the remaining seven the difference made by capacity was equal to the run difference or larger than it.

The second claim of the course is read from the same table and is not hidden. In fourteen rows of the subscriber table the comparison lines come from the Supervised Learning course: decision tree 0.7857, linear model 0.8016. In the nine lessons of Neural Network Fundamentals the number of the network did not exceed 0.8016 in any run; the highest of the eighteen runs is 0.7976 and in three lessons the number of the network stopped exactly at the number of the tree. In the five lessons of Training Practice the number of the network stayed between 0.8016 and 0.8135, that is, at or above the line of the linear model; but they ran with a feature set of eight columns and the ten initializations of the first lesson spread between 0.7857–0.8095, with both lines inside this range. The deep network could not pass the decision tree and the linear model on this fictional tabular data in a way that is separable from run-to-run variability. The gain came not from capacity but from training practice, and in at least one more lesson a deeper network could not pass a shallower one.

In twenty-four lessons the architecture was built and the second run was written next to every number. But the inputs were always ready-made numbers: a column, a grid cell, a reading sequence, an event code. How a text or an image is turned into numbers, against which baseline that conversion is measured and what the measures of the field itself are was never asked. The next course, Natural Language Processing and Computer Vision, opens with this question.