Experiment Tracking

In the previous lesson three runs were made and three different numbers came out. The only reason we could see the three side by side was that all three had been run in the same block. In a real study these runs are spread over days, done by different people, and all that is left is a single number: an accuracy written on a slide, relayed in a message, or remembered by one person.

This lesson goes after that number. Suppose we have a run space and someone passes along this information: “a run gave 0.8078”. There is a single question: how much work it takes to find that run’s weights and the items it used. The measure is the number of re-run runs.

• RP12. The run space is eighteen runs: six pinning levels (from zero to five items) times three trials. The space is a choice, and every number sensitive to it is written together with the size of the space.
• RP13. Experiment tracking is storing a run’s parameters, metrics and output together with the run itself. It is not a product but a record-keeping discipline.
• RP14. The baseline is the setup with no record: no run has been written down, and all that is at hand is the reported number.
• RP15. The procedure for a search with no record is this: the space is re-run from start to finish in order, and it stops the moment the reported number is matched. The number of repeats depends on this order, and the order is a choice.
• RP16. The cost of the record is counted as stored fields: the number of fields per record times the number of runs. Raw bytes are not written.
• RP17. Training practice, run variability, and the tuning budget were measured in the Deep Learning course; they are not repeated here. This lesson only counts how much the record shortens the search.
• RP18. The label distribution is {0: 1752, 1: 448} as set up in the previous lesson, the base class ratio is 0.7964, and the accuracy numbers in this lesson carry no performance claim above that ratio. The resolution is 0.0017.

The Run Space and the Ledger

The first block rebuilds, verbatim, the fictional source, pipeline and model of the previous lesson.

# FICTIONAL. Meter readings, the pipeline and the model are defined within the lesson; no real
# learning library or experiment tracking product is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


TRAIN = PERIODS[1:9]
TEST = PERIODS[9:]
THRESHOLD = 40


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


def features(records, window=None, digits=4):
    """Independent of the weights , computed once."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {
            "subscriber_mean": mean(sub[k["subscriber"]], i),
            "region_mean": mean(reg[k["region"]], i),
            "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def train(records, feats, grid=None, threshold=THRESHOLD, periods=None):
    grid = GRID if grid is None else grid
    periods = TRAIN if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    best, best_w = -1.0, None
    for w in grid:
        d = sum(1 for k in subset
                if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k))
        o = d / len(subset)
        if o > best:
            best, best_w = o, w
    return {"weights": best_w, "train_accuracy": round(best, 4),
            "candidates": len(grid), "train_rows": len(subset)}


def evaluate(records, model, feats, threshold=THRESHOLD, periods=None):
    periods = TEST if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    d = sum(1 for k in subset
            if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0)
            == label(k))
    return round(d / len(subset), 4)


ITEMS = ("data_version", "feature_code", "split", "grid", "rounding")


def run(records, pinned, trial=0):
    """Every item that is not pinned changes with the trial."""
    rec = records if "data_version" in pinned else records[: len(records) - 20 * trial]
    window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial)
    digits = 4 if "rounding" in pinned else max(4 - trial, 0)
    grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial]
    periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):]
    feats = features(rec, window, digits)
    m = train(rec, feats, grid)
    return {"weights": m["weights"], "train": m["train_accuracy"],
            "test": evaluate(rec, m, feats, periods=periods)}

An experiment record is not a line written after the run is over; it is produced inside the run, because only the run itself knows which items it used. The make_record below does this: it executes a run in the space and writes the number of rows that run read, the number of candidates it tried, the feature window, the number of test periods, the weights it found, and the accuracy it achieved into a single record.

S = rows()
SPACE = [(ITEMS[:n], trial) for n in range(6) for trial in range(3)]


def make_record(no):
    """Executes run number no in the space and records the items it used."""
    pinned, trial = SPACE[no - 1]
    r = run(S, pinned, trial)
    return {"run": no, "pinned": len(pinned), "trial": trial,
            "rows": len(S) if "data_version" in pinned else len(S) - 20 * trial,
            "candidates": len(GRID) if "grid" in pinned else len(GRID) - 6 * trial,
            "window": None if "feature_code" in pinned else (None if trial == 0 else 12 - trial),
            "test_periods": len(TEST) if "split" in pinned else len(PERIODS) - 9 - (trial % 2),
            "weights": r["weights"], "test": r["test"]}


LEDGER = [make_record(no) for no in range(1, len(SPACE) + 1)]
print("runs:", len(LEDGER), " fields per record:", len(LEDGER[0]),
      " stored fields:", len(LEDGER) * len(LEDGER[0]))
print()
print(f"{'run':<7}{'pinned':>8}{'trial':>7}{'rows':>7}{'candidates':>12}{'window':>8}{'periods':>9}{'test':>9}")
for k in LEDGER[:3] + LEDGER[-3:]:
    print(f"{k['run']:<7}{k['pinned']:>8}{k['trial']:>7}{k['rows']:>7}{k['candidates']:>12}"
          f"{str(k['window']):>8}{k['test_periods']:>9}{k['test']:>9.4f}")
counts = {}
for k in LEDGER:
    counts[k["test"]] = counts.get(k["test"], 0) + 1
print()
print("different test numbers:", len(counts), " number -> how many runs:", dict(sorted(counts.items())))

runs: 18  fields per record: 9  stored fields: 162

run      pinned  trial   rows  candidates  window  periods     test
1             0      0   2200          27    None        3   0.8067
2             0      1   2180          21      11        2   0.7955
3             0      2   2160          15      10        3   0.8078
16            5      0   2200          27    None        3   0.8067
17            5      1   2200          27    None        3   0.8067
18            5      2   2200          27    None        3   0.8067

different test numbers: 4  number -> how many runs: {0.795: 2, 0.7955: 1, 0.8067: 14, 0.8078: 1}

The first three lines of the ledger are the previous lesson’s unpinned baseline; in the last three lines all five items are pinned, and all three trials use the same rows, the same candidates, the same window, and the same test periods. The last line gives the main finding: the eighteen runs produced only four different numbers, and of these 0.8067 came out in fourteen runs.

Searching for the Run That Produced a Number

Now the place where the record pays off can be measured. In the setup with no record, all that is at hand is the reported number, and the only procedure is to re-run the space from the start. In the setup with a record, the ledger is read. In between there is a third case, and in practice it is the most common one: only the metric was recorded, that is, the run number and the accuracy were written but the parameters were not.

TARGETS = (0.8078, 0.7955, 0.7950, 0.8067, 0.8100)


def search_without_record(target):
    """No record: the space is re-run from the start, and it stops when the number matches."""
    repeats = 0
    for pinned, trial in SPACE:
        repeats += 1
        if run(S, pinned, trial)["test"] == target:
            return repeats
    return repeats


METRICS_ONLY = [(k["run"], k["test"]) for k in LEDGER]
print("metrics-only record -- fields:", len(METRICS_ONLY) * 2, " full record -- fields:",
      len(LEDGER) * len(LEDGER[0]))
print()
print(f"{'target':<9}{'no record':>10}{'metrics only':>14}{'full record':>13}{'same-number runs':>18}")
for h in TARGETS:
    a = search_without_record(h)
    candidates = [k["run"] for k in LEDGER if k["test"] == h]
    o = 0 if not any(s == h for _, s in METRICS_ONLY) else a
    print(f"{h:<9.4f}{a:>10}{o:>14}{0:>13}{len(candidates):>18}")
print()
print("runs giving 0.8067:", [k["run"] for k in LEDGER if k["test"] == 0.8067])

metrics-only record -- fields: 36  full record -- fields: 162

target    no record  metrics only  full record  same-number runs
0.8078            3             3            0                 1
0.7955            2             2            0                 1
0.7950            5             5            0                 2
0.8067            1             1            0                14
0.8100           18             0            0                 0

runs giving 0.8067: [1, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]

The first column is the cost of the setup with no record, and depending on the target it varies between 1 and 18. The last row is the most expensive: the reported 0.8100 was never produced in this space, so the search repeats all eighteen runs and nothing is left in hand. This is the most common outcome of a search with no record, because the reported number often comes from the state the space was in that day, and the space has changed.

The third column is 0 for every target. A search with a full record needs no re-run; the ledger is read, and that is all.

A Number Does Not Identify a Run

The real finding of the table sits in the last column, and it matters more than the repeat counts. At the target 0.8067, the search with no record “succeeds” in one run: the first run in the space gives this number and the search stops. But fourteen runs give the same number. The first run the search finds does not have to be the run that produced the reported number; it is only one of fourteen candidates.

The consequence is this: re-running can reproduce a number, but it cannot diagnose a run. If the reported number is 0.8067, whether that run read 2200 rows or 2160 rows, whether it tried 27 candidates or 15, and whether it split the test into three periods or two cannot be deduced from the number. The ledger answers all of these questions in a single line and tells the fourteen candidates apart. What the record gives is not savings on repeats but diagnosis; savings on repeats are a by-product.

The Cost of the Record

The cost of the ledger has two items. The first is stored fields: each run writes nine fields, and eighteen runs hold 162 fields. This grows in direct proportion to the number of runs; if hundreds of runs are made in a tuning sweep, the ledger grows by that same proportion and needs a retention policy. The second is the added step: make_record runs inside the run, which means a write step enters the training pipeline, and that step has to read the items the run used. A record that cannot read an item cannot write it.

The second column shows how this cost can be wasted. A ledger that records only metrics holds 36 fields, a little more than a fifth of the full ledger, and it shortens only one of five searches: at the never-produced target 0.8100 it brings eighteen repeats down to zero. For the remaining four targets the number of repeats is exactly the same as in the setup with no record, because when the parameters are not written the run still has to be re-run. That a ledger is being kept is not an assurance; unless which fields it holds is measured, the ledger is nothing but a cost.

Summary

• Experiment tracking means writing a run’s parameters, metrics and output inside the run itself; since only the run knows its items, the record cannot be completed afterward.
• The eighteen-run space produces only four different numbers, and 0.8067 comes out in fourteen runs.
• In the setup with no record, finding the run that produced a reported number costs 1, 2, 3, 5, or 18 repeated runs; for a number that was never produced, the entire space is run in vain. With a full record, the number of repeats is 0 for all five targets.
• Re-running reproduces a number but cannot diagnose the run: at the target 0.8067 the search stops after one repeat, and the run it finds is only one of fourteen candidates.
• The cost of the record is nine fields per run, 162 fields in total, and a write step added to the training pipeline.
• A 36-field ledger that records only metrics shortens only one of five searches; what is measured is not the existence of a ledger but which fields it holds.

Next Step

The ledger wrote down the number of rows the run used, but it did not write down which rows they were. All eighteen runs read their data from the same place, and the name of that place never changed; whether its content changed was never asked. When a run records that it read 2200 rows, the run a week later also records that it read 2200 rows, and the two numbers look the same. The next lesson measures the content under that name: in a setup with no versioning, how many runs point to the same name and give different results, how many with versioning, and how much the versioning itself costs.