Experiment Tracking

In the previous three lessons the setup of the training was opened up piece by piece, and every piece produced a decision: by which route the gradient is computed, in which order the examples are given, how many go together, what the step is. Every decision moved the result and none of them was the model itself. How many of these decisions have to be written down for a training to give the same number a second time was not asked.

The question is not new and part of it is closed. The Data Collection and Preparation course wrote the preparation notebook as thirty-nine decisions; the Data Analysis with Python course counted the working environment as nine items; the Visualization, Reporting and Analysis at Scale course counted the report itself as eleven items and found the total to be fifty-nine. In the pipeline lesson of the Data Preparation and Feature Engineering course one more number was placed next to this list: the preparation chain learned 131 numbers from the training set, and all of them had to be read from the right set. These computations are not repeated; they are taken as input. What this lesson adds is the training itself.

• TP18. The dataset, the network, the baseline and the backpropagation are inherited from Lesson 01. The items of the training loop are gathered in a single dictionary and the run is read from this dictionary; only the scaling item is applied outside the loop, while the data is being prepared.
• TP19. To release an item does not mean to delete it but to move it to another value that is equally defensible. A batch of twenty-five instead of thirty, a step of 0.25 instead of 0.30, fifty epochs instead of forty: none of them is wrong.
• TP20. Items are released one by one while the others stay pinned. The criterion is the accuracy on the test set, and the test set does not change in any setup.
• TP21. The “summation order” item is a trace of concurrency: when the gradients of a batch are summed in parallel, which term is added first can change from run to run. Here this is modeled by summing the examples of the batch in reverse order; since floating-point addition is not associative, the result separates in the last digits.
• TP22. Two quantities are counted separately: the shift of the parameters and the shift of the result. While the first is not zero the second can be zero; the threshold is how far from the decision boundary one stands.
• TP23. No experiment tracking tool, notebook application or repository name is mentioned. Tracking here is the item list itself: if the list can be carried, the run can be carried.

The Dataset, the Network and the Item Dictionary

# COMMON — FICTIONAL. The subscriber table of the Introduction to Machine Learning through
# Unsupervised Learning courses is generated with the same seed (20260218);
# the training set is reduced from 756 to 300 rows. The items of the training loop are
# gathered in a single dictionary and the run is read from this dictionary.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]
FIELDS = ["area_south", "area_east", "area_west", "area_central",
          "household", "satisfaction", "period", "mean"]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def mix(items, seed):
    r, s = rng(seed), list(items)
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return s


DATA = []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() < 0.046:
        continue
    subscriber_id = 10001 + i
    r, v = rng(SEED + 7000 + subscriber_id), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            b[2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + subscriber_id)
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (household >= 5) - 0.20 * (rating >= 4) + b[3]
         + 0.9 * (subscriber_id - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": household, "satisfaction": rating, "period": len(v),
         "mean": round(mean, 2), "suspicious": int(z > 1.35)}
    for q in REGION[1:]:
        x["area_" + q[0]] = int(b[0] == q[0])
    DATA.append(x)

K = mix(DATA, SEED + 90000)
TRAIN, TEST = K[:300], K[1008:]


def table(rows, stats=None):
    X = [[float(s[a]) for a in FIELDS] for s in rows]
    if stats is None:                     # scaling is learned from the training set only
        stats = []
        for j in range(4, 8):
            m = sum(v[j] for v in X) / len(X)
            stats.append((j, m, math.sqrt(sum((v[j] - m) ** 2 for v in X) / len(X))))
    for v in X:
        for j, m, sd in stats:
            v[j] = (v[j] - m) / sd
    return X, [s["suspicious"] for s in rows], stats


X_train, y_train, STATS = table(TRAIN)
X_test, y_test, _ = table(TEST, STATS)
BASELINE = sum(v == 0 for v in y_test) / len(y_test)

G, H = len(FIELDS), 8


def init_weights(seed):                  # gain-scaled initialization
    r = rng(seed)
    return [(r() * 2 - 1) * math.sqrt(2.0 / G) for _ in range(H * G)] + [0.0] * H \
        + [(r() * 2 - 1) * math.sqrt(2.0 / H) for _ in range(H)] + [0.0]


def gradient(p, X, y, idx):              # the hand-written backpropagation of Lesson 01
    g = [0.0] * len(p)
    for k in idx:
        x, hh = X[k], [0.0] * H
        for j in range(H):
            s = p[H * G + j] + sum(p[j * G + i] * x[i] for i in range(G))
            hh[j] = s if s > 0 else 0.0
        s = p[-1] + sum(p[H * G + H + j] * hh[j] for j in range(H))
        d = (1.0 / (1.0 + math.exp(-s)) if s > -30 else 0.0) - y[k]
        g[-1] += d
        for j in range(H):
            g[H * G + H + j] += d * hh[j]
            if hh[j] > 0:
                g[H * G + j] += d * p[H * G + H + j]
                for i in range(G):
                    g[j * G + i] += d * p[H * G + H + j] * x[i]
    return [v / len(idx) for v in g]


DEFAULTS = {"seed": SEED + 1, "order": SEED + 700, "batch": 30, "step": 0.30,
            "epochs": 40, "split": "slice", "summing": "forward", "scale": "train"}


def train_model(X, y, **item):
    a = dict(DEFAULTS, **item)
    p = init_weights(a["seed"])
    for t in range(a["epochs"]):
        perm = mix(range(len(X)), a["order"] + t)
        n = len(perm) // a["batch"]
        for b in range(n):
            idx = perm[b * a["batch"]:(b + 1) * a["batch"]] if a["split"] == "slice" \
                else perm[b::n]
            if a["summing"] == "backward":
                idx = idx[::-1]
            p = [u - a["step"] * v for u, v in zip(p, gradient(p, X, y, idx))]
    return p


def accuracy(p, X, y):
    n = 0
    for k in range(len(X)):
        x = X[k]
        o = p[-1] + sum(p[H * G + H + j] * max(0.0, p[H * G + j]
                        + sum(p[j * G + i] * x[i] for i in range(G)))
                        for j in range(H))
        n += (o >= 0.0) == (y[k] == 1)
    return n / len(X)


print(f"fictional: train {len(X_train)} rows (reduced from 756), test {len(X_test)}, "
      f"parameters {len(init_weights(SEED))}; baseline {BASELINE:.4f}")

fictional: train 300 rows (reduced from 756), test 252, parameters 81; baseline 0.7579

How Many Items

# The items to pin. The first three sources are INHERITED FROM EARLIER COURSES and are not
# repeated here; this lesson adds only those SPECIFIC TO TRAINING.
INHERITED = [("preparation notebook (Data Collection and Preparation)", 39),
             ("working environment (Data Analysis with Python)", 9),
             ("report (Visualization, Reporting and Analysis at Scale)", 11)]
TRAINING = ["initialization seed", "example order seed", "batch size",
            "batch split style", "learning rate", "number of epochs",
            "summation order (concurrency)", "set the scale is read from"]
for name, n in INHERITED:
    print(f"  {name:<55} {n:>3} items")
print(f"  {'training itself (this lesson)':<55} {len(TRAINING):>3} items")
print(f"  {'total':<55} {sum(n for _, n in INHERITED) + len(TRAINING):>3} items")
print("\ntraining-specific items and their default values:")
print("  " + ", ".join(f"{a}={DEFAULTS[k]}" for a, k in
                       zip(TRAINING, ("seed", "order", "batch", "split", "step",
                                      "epochs", "summing", "scale"))))

  preparation notebook (Data Collection and Preparation)   39 items
  working environment (Data Analysis with Python)           9 items
  report (Visualization, Reporting and Analysis at Scale)  11 items
  training itself (this lesson)                             8 items
  total                                                    67 items

training-specific items and their default values:
  initialization seed=20260219, example order seed=20260918, batch size=30, batch split style=slice, learning rate=0.3, number of epochs=40, summation order (concurrency)=forward, set the scale is read from=train

The total is sixty-seven. Fifty-nine were counted before this lesson and are not repeated here; eight belong to the training itself. None of the eight is the architecture of the model — the number of layers, the number of units and the activation do not enter this list, because they are the definition of the reported model. What is on the list are decisions about how the reported number was produced, and most of them fit in a single line.

Each Item One by One

# Each item is released ALONE: it is moved to another equally defensible value and the
# others stay pinned. When the scale item is released, the scale statistics are read
# from the training+test set instead of the training set.
X_train2, y_train2, _ = table(TRAIN, table(TRAIN + TEST)[2])
REFERENCE, P0 = accuracy(train_model(X_train, y_train), X_test, y_test), train_model(X_train, y_train)
ALTERNATIVES = [({"seed": SEED + 2}, "20260220"), ({"order": SEED + 800}, "20261018"),
                ({"batch": 25}, "25"), ({"split": "strided"}, "strided"),
                ({"step": 0.25}, "0.25"), ({"epochs": 50}, "50"),
                ({"summing": "backward"}, "backward"), (None, "train+test")]
print(f"baseline {BASELINE:.4f}, with all items pinned {REFERENCE:.4f}\n")
print(f"{'released item':<30} {'new value':<14} {'test':>7} {'shift':>8} "
      f"{'subscribers':>11} {'parameters':>10}")
SHIFT = []
for name, (d, new) in zip(TRAINING, ALTERNATIVES):
    p = train_model(X_train, y_train, **d) if d else train_model(X_train2, y_train2)
    v = accuracy(p, X_test, y_test)
    SHIFT.append(abs(v - REFERENCE))
    print(f"{name:<30} {new:<14} {v:>7.4f} {v - REFERENCE:>+8.4f} "
          f"{round(abs(v - REFERENCE) * len(X_test)):>11} "
          f"{max(abs(a - b) for a, b in zip(p, P0)):>10.2e}")
print(f"\n{sum(1 for u in SHIFT if u > 1e-9)} of {len(SHIFT)} items move the result alone; "
      f"largest shift {max(SHIFT):.4f} "
      f"({round(max(SHIFT) * len(X_test))} subscribers)")
for d, name in (({"epochs": 50, "summing": "backward"}, "epochs 50 + summing backward"),
                ({"epochs": 50, "batch": 25}, "epochs 50 + batch 25")):
    v = accuracy(train_model(X_train, y_train, **d), X_test, y_test)
    print(f"  {name:<28} {v:.4f}  shift {v - REFERENCE:+.4f} "
          f"({round(abs(v - REFERENCE) * len(X_test))} subscribers)")

baseline 0.7579, with all items pinned 0.8095

released item                  new value         test    shift subscribers parameters
initialization seed            20260220        0.7937  -0.0159           4   1.46e+00
example order seed             20261018        0.8016  -0.0079           2   7.45e-02
batch size                     25              0.8056  -0.0040           1   2.76e-01
batch split style              strided         0.8135  +0.0040           1   4.80e-01
learning rate                  0.25            0.8056  -0.0040           1   1.98e-01
number of epochs               50              0.8095  +0.0000           0   2.10e-01
summation order (concurrency)  backward        0.8095  +0.0000           0   6.66e-16
set the scale is read from     train+test      0.8056  -0.0040           1   3.18e-01

6 of 8 items move the result alone; largest shift 0.0159 (4 subscribers)
  epochs 50 + summing backward 0.8095  shift +0.0000 (0 subscribers)
  epochs 50 + batch 25         0.8016  shift -0.0079 (2 subscribers)

Six of the eight items move the result alone and the largest shift is 0.0159, four subscribers. That item is the initialization seed. It is not a setting of the model and not a preparation decision; it is the place where the weights start. The third number, the signature of this course, is also the largest row in the reproducibility list. The one subscriber each that the learning rate and the batch size move is a quarter of what the seed moves.

The last column separates two items from each other. When the summation order is released the parameters move by 6.66e-16, that is, by the last digit of floating-point addition; not one decision of the test set changes. When the number of epochs goes from forty to fifty the parameters move by 0.2098 — fourteen digits above the summation order — and again not one decision of the test set changes. The two zeros are not the same zero: in the first the network is the same network, in the second the network is another network and only none of the decisions has crossed the boundary.

The price of this separation is in the last two rows. The number of epochs alone moves zero, the batch size alone moves one subscriber; released together they move two subscribers. An item that does not move the result alone is not an item that does not need to be pinned. The item for the set the scale is read from confirms this as well: when the test set is mixed into the scale statistics the number moves by only one subscriber, yet the same leakage had produced a 0.0476 inflation with the decision tree in the Data Preparation and Feature Engineering course. That a leakage looks small is not proof that it is absent.

Summary

• Reproducing a training requires pinning 67 items: on top of the 39 items of the Data Collection and Preparation course, the 9 of the Data Analysis with Python course and the 11 of the Visualization, Reporting and Analysis at Scale course, 8 training-specific items are added.
• Six of the eight items move the result when released alone; with all items pinned the test number is 0.8095 and the baseline is 0.7579.
• The largest shift is 0.0159 (four subscribers) and its source is the initialization seed — not a setting of the model, the third number of this course.
• The shift of the parameters and the shift of the result are separate things: the summation order moves the parameters by 6.66e-16 and the number of epochs by 0.2098, and neither changes a single decision.
• The number of epochs, which moves zero alone, moves two subscribers when released together with the batch size; and reading the scale from the test set moves one subscriber here, while in the Data Preparation and Feature Engineering course the same leakage had inflated the number by 0.0476.

Next Step

In this lesson the items were all held pinned and released one by one. In a real training the opposite is done: a few items are released on purpose, a candidate list is built and the best is chosen. The search methods were measured in the Supervised Learning course and will not be repeated; but in that course the number of every candidate was read from a single run. What was measured in this lesson makes one ask: if the difference between two runs of the same candidate is four subscribers and the difference between two candidates is one subscriber, what is the search choosing. The next lesson takes up the hyperparameter search with this question and closes the course.