--- title: 'Experiment Tracking' source: 'https://academia.sh/en/courses/mlops-fundamentals/experiment-tracking' course: 'MLOps Fundamentals' language: en updated: '2026-10-21T05:03:01+00:00' license: 'CC BY-SA 4.0' objectives: - 'Builds an experiment record that carries the parameters, metrics and outputs for every run in a run space.' - 'Compares the number of repeated runs needed to find the run that produced a result in setups with and without a record.' - 'Computes the cost of the record in stored fields and shows which search an incomplete record does not shorten.' --- # Experiment Tracking Measuring how recording parameters, metrics and outputs makes a run findable again: in a space of eighteen runs with no record, finding the weights that produced a reported number takes 3 re-runs for a target of 0.8078, 2 for 0.7955, 5 for 0.7950, 1 for 0.8067, and 18 for a target that was never produced, while with a full record all five need 0 repeats; the number alone does not identify the run, because fourteen of the eighteen runs give 0.8067 and the candidate found by re-running is only one of fourteen; the cost of the record is nine fields per run and 162 fields in total, while a 36-field ledger that records only metrics shortens only one of five searches. In the previous lesson three runs were made and three different numbers came out. The only reason we could see the three side by side was that all three had been run in the same block. In a real study these runs are spread over days, done by different people, and all that is left is a single number: an accuracy written on a slide, relayed in a message, or remembered by one person. This lesson goes after that number. Suppose we have a run space and someone passes along this information: "a run gave 0.8078". There is a single question: how much work it takes to find that run's weights and the items it used. The measure is **the number of re-run runs**. - **RP12.** The run space is **eighteen runs**: six pinning levels (from zero to five items) times three trials. The space is a choice, and every number sensitive to it is written together with the size of the space. - **RP13.** **Experiment tracking** is storing a run's parameters, metrics and output together with the run itself. It is not a product but **a record-keeping discipline**. - **RP14.** The baseline is **the setup with no record**: no run has been written down, and all that is at hand is the reported number. - **RP15.** The procedure for a search with no record is this: the space is re-run **from start to finish in order**, and it stops the moment the reported number is matched. The number of repeats depends on this order, and the order is **a choice**. - **RP16.** The cost of the record is counted as **stored fields**: the number of fields per record times the number of runs. Raw bytes are not written. - **RP17.** Training practice, run variability, and the tuning budget were measured in the Deep Learning course; they are **not repeated** here. This lesson only counts **how much the record shortens the search**. - **RP18.** The label distribution is `{0: 1752, 1: 448}` as set up in the previous lesson, the **base class ratio is 0.7964**, and the accuracy numbers in this lesson carry no performance claim above that ratio. The resolution is **0.0017**. ## The Run Space and the Ledger The first block rebuilds, verbatim, the fictional source, pipeline and model of the previous lesson. ```python # FICTIONAL. Meter readings, the pipeline and the model are defined within the lesson; no real # learning library or experiment tracking product is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] TEST = PERIODS[9:] THRESHOLD = 40 def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) ITEMS = ("data_version", "feature_code", "split", "grid", "rounding") def run(records, pinned, trial=0): """Every item that is not pinned changes with the trial.""" rec = records if "data_version" in pinned else records[: len(records) - 20 * trial] window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial) digits = 4 if "rounding" in pinned else max(4 - trial, 0) grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial] periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):] feats = features(rec, window, digits) m = train(rec, feats, grid) return {"weights": m["weights"], "train": m["train_accuracy"], "test": evaluate(rec, m, feats, periods=periods)} ``` An experiment record is not a line written after the run is over; it is produced **inside the run**, because only the run itself knows which items it used. The `make_record` below does this: it executes a run in the space and writes the number of rows that run read, the number of candidates it tried, the feature window, the number of test periods, the weights it found, and the accuracy it achieved into a single record. ```python S = rows() SPACE = [(ITEMS[:n], trial) for n in range(6) for trial in range(3)] def make_record(no): """Executes run number no in the space and records the items it used.""" pinned, trial = SPACE[no - 1] r = run(S, pinned, trial) return {"run": no, "pinned": len(pinned), "trial": trial, "rows": len(S) if "data_version" in pinned else len(S) - 20 * trial, "candidates": len(GRID) if "grid" in pinned else len(GRID) - 6 * trial, "window": None if "feature_code" in pinned else (None if trial == 0 else 12 - trial), "test_periods": len(TEST) if "split" in pinned else len(PERIODS) - 9 - (trial % 2), "weights": r["weights"], "test": r["test"]} LEDGER = [make_record(no) for no in range(1, len(SPACE) + 1)] print("runs:", len(LEDGER), " fields per record:", len(LEDGER[0]), " stored fields:", len(LEDGER) * len(LEDGER[0])) print() print(f"{'run':<7}{'pinned':>8}{'trial':>7}{'rows':>7}{'candidates':>12}{'window':>8}{'periods':>9}{'test':>9}") for k in LEDGER[:3] + LEDGER[-3:]: print(f"{k['run']:<7}{k['pinned']:>8}{k['trial']:>7}{k['rows']:>7}{k['candidates']:>12}" f"{str(k['window']):>8}{k['test_periods']:>9}{k['test']:>9.4f}") counts = {} for k in LEDGER: counts[k["test"]] = counts.get(k["test"], 0) + 1 print() print("different test numbers:", len(counts), " number -> how many runs:", dict(sorted(counts.items()))) ``` ``` runs: 18 fields per record: 9 stored fields: 162 run pinned trial rows candidates window periods test 1 0 0 2200 27 None 3 0.8067 2 0 1 2180 21 11 2 0.7955 3 0 2 2160 15 10 3 0.8078 16 5 0 2200 27 None 3 0.8067 17 5 1 2200 27 None 3 0.8067 18 5 2 2200 27 None 3 0.8067 different test numbers: 4 number -> how many runs: {0.795: 2, 0.7955: 1, 0.8067: 14, 0.8078: 1} ``` The first three lines of the ledger are the previous lesson's unpinned baseline; in the last three lines all five items are pinned, and all three trials use the same rows, the same candidates, the same window, and the same test periods. The last line gives the main finding: the eighteen runs produced only **four different numbers**, and of these **0.8067 came out in fourteen runs**. ## Searching for the Run That Produced a Number Now the place where the record pays off can be measured. In the setup with no record, all that is at hand is the reported number, and the only procedure is to re-run the space from the start. In the setup with a record, the ledger is read. In between there is a third case, and in practice it is the most common one: **only the metric was recorded**, that is, the run number and the accuracy were written but the parameters were not. ```python TARGETS = (0.8078, 0.7955, 0.7950, 0.8067, 0.8100) def search_without_record(target): """No record: the space is re-run from the start, and it stops when the number matches.""" repeats = 0 for pinned, trial in SPACE: repeats += 1 if run(S, pinned, trial)["test"] == target: return repeats return repeats METRICS_ONLY = [(k["run"], k["test"]) for k in LEDGER] print("metrics-only record -- fields:", len(METRICS_ONLY) * 2, " full record -- fields:", len(LEDGER) * len(LEDGER[0])) print() print(f"{'target':<9}{'no record':>10}{'metrics only':>14}{'full record':>13}{'same-number runs':>18}") for h in TARGETS: a = search_without_record(h) candidates = [k["run"] for k in LEDGER if k["test"] == h] o = 0 if not any(s == h for _, s in METRICS_ONLY) else a print(f"{h:<9.4f}{a:>10}{o:>14}{0:>13}{len(candidates):>18}") print() print("runs giving 0.8067:", [k["run"] for k in LEDGER if k["test"] == 0.8067]) ``` ``` metrics-only record -- fields: 36 full record -- fields: 162 target no record metrics only full record same-number runs 0.8078 3 3 0 1 0.7955 2 2 0 1 0.7950 5 5 0 2 0.8067 1 1 0 14 0.8100 18 0 0 0 runs giving 0.8067: [1, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] ``` The first column is the cost of the setup with no record, and depending on the target it varies between 1 and 18. The last row is the most expensive: the reported 0.8100 was never produced in this space, so the search repeats **all eighteen runs** and nothing is left in hand. This is the most common outcome of a search with no record, because the reported number often comes from the state the space was in that day, and the space has changed. The third column is **0** for every target. A search with a full record needs no re-run; the ledger is read, and that is all. ## A Number Does Not Identify a Run The real finding of the table sits in the last column, and it matters more than the repeat counts. At the target 0.8067, the search with no record "succeeds" in **one** run: the first run in the space gives this number and the search stops. But **fourteen runs** give the same number. The first run the search finds does not have to be the run that produced the reported number; it is only one of fourteen candidates. The consequence is this: re-running can reproduce **a number**, but it cannot diagnose **a run**. If the reported number is 0.8067, whether that run read 2200 rows or 2160 rows, whether it tried 27 candidates or 15, and whether it split the test into three periods or two cannot be deduced from the number. The ledger answers all of these questions in a single line and tells the fourteen candidates apart. What the record gives is **not savings on repeats but diagnosis**; savings on repeats are a by-product. ## The Cost of the Record The cost of the ledger has two items. The first is **stored fields**: each run writes nine fields, and eighteen runs hold **162 fields**. This grows in direct proportion to the number of runs; if hundreds of runs are made in a tuning sweep, the ledger grows by that same proportion and needs a retention policy. The second is **the added step**: `make_record` runs inside the run, which means a write step enters the training pipeline, and that step has to read the items the run used. A record that cannot read an item cannot write it. The second column shows how this cost can be wasted. A ledger that records only metrics holds 36 fields, a little more than a fifth of the full ledger, and it shortens **only one** of five searches: at the never-produced target 0.8100 it brings eighteen repeats down to zero. For the remaining four targets the number of repeats is **exactly the same** as in the setup with no record, because when the parameters are not written the run still has to be re-run. That a ledger is being kept is not an assurance; unless **which fields it holds** is measured, the ledger is nothing but a cost. ## Summary - Experiment tracking means writing a run's parameters, metrics and output inside the run itself; since only the run knows its items, the record cannot be completed afterward. - The eighteen-run space produces only **four different numbers**, and **0.8067 comes out in fourteen runs**. - In the setup with no record, finding the run that produced a reported number costs **1**, **2**, **3**, **5**, or **18** repeated runs; for a number that was never produced, the entire space is run in vain. With a full record, the number of repeats is **0** for all five targets. - Re-running reproduces a number but cannot diagnose the run: at the target 0.8067 the search stops after one repeat, and the run it finds is only one of **fourteen candidates**. - The cost of the record is **nine fields** per run, **162 fields** in total, and a write step added to the training pipeline. - A **36-field** ledger that records only metrics shortens only one of five searches; what is measured is not the existence of a ledger but **which fields it holds**. ## Next Step The ledger wrote down the number of rows the run used, but it did not write down **which** rows they were. All eighteen runs read their data from the same place, and the name of that place never changed; whether its content changed was never asked. When a run records that it read 2200 rows, the run a week later also records that it read 2200 rows, and the two numbers look the same. The next lesson measures the content under that name: in a setup with no versioning, how many runs point to the same name and give different results, how many with versioning, and how much the versioning itself costs.