--- title: 'Model Registry' source: 'https://academia.sh/en/courses/mlops-fundamentals/model-registry' course: 'MLOps Fundamentals' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the lifecycle stages of a model and writes what tying each stage transition to a run id provides.' - 'Shows that the model itself carries no identity by counting the runs that produce the same weight triple.' - "Compares the number of records looked at to answer a question in setups with and without a registry and counts the registry's own cost." --- # Model Registry Counting the lifecycle stages of a model and which run each stage transition is tied to: in a record of sixteen runs only two different weight triples come out, the triple in production is carried exactly by twelve runs, and those twelve runs report their own result with two different numbers such as 0.7950 and 0.8067. Without a model registry, finding which run the production model came from means looking at all sixteen records and the search never goes below twelve candidates, while the previous production run to roll back to cannot be found at all; with a registry that ties four stages and five transitions to a run id, both questions are answered by reading a single record, and the cost of the registry is twenty-one records. Five items were pinned and the three runs now give the same number. That is the answer to the question of reproducibility, but it opens one more question: **which run** did the model in hand come from? The only thing a trained model carries is its weights; the weights do not carry within them the data version, the feature window, or the test split that produced them. This lesson measures that gap. Whether a model is being tried, being tested, or in service, and **with which run** it came to that state, lives not in the model's file but in a separate record. A **model registry** is exactly this record: it ties the stages and the transitions between stages to a run id. What is to be measured is how many records that tie makes you read when answering a question. - **RP57.** The source, the first three stages of the pipeline, the fictional model, and the registry are **fictional**; no real learning library or registry product is called. The seed is **20260218**. - **RP58.** A **run** is a single training job executed with particular settings, and it carries an id. A **model** is the weight triple that run produced, and it carries no id. - **RP59.** A **stage** is the state of the model in its lifecycle. Four are counted here: `candidate`, `staging`, `production`, `archived`. A **transition** is the record that moves a model from one stage to another, and it is **tied to a run id**. - **RP60.** The baseline is the **setup without a registry**: the runs are recorded somewhere, but no stage is tied to a run. The only way to get from a model at a stage to a run is **matching the weights**. - **RP61.** The label is imbalanced: `{0: 1752, 1: 448}`, the **base class ratio is 0.7964**, and in the test set it is **0.8067**. Accuracy read alone misleads; in this lesson two runs will carry the same model and report different numbers. - **RP62.** The run record itself and data versioning were handled in the earlier lessons of this topic; the only thing added here is **tying the stage transition to the run**. The lineage procedure was measured in the Introduction to Data Engineering course and is **not repeated**. - **RP63.** The registry is built on a record of **sixteen runs**. The runs are the combination of four feature windows, two grid sizes, and two test splits. - **RP64.** Resolution: the full test set is **600 rows**, the step is **1/600 = 0.0017**. In this lesson the difference between the two numbers reported by the same weights is **0.0117**, that is, larger than six times the step. ## The Model Has No Identity A trained model is the output of a procedure, and the output does not contain the procedure that produced it. Here the model consists of three numbers; in a real setup it would be far larger, but the situation does not change. Inside the model it is not written which data it was trained with, which feature window was chosen, or over which periods the test was done. The only thing that carries this information is **the run record**, and unless the tie between the model and the run is built separately, it does not exist. When the tie is not built, the only thing that can be done is **matching**: comparing the weights in hand with the weights in the run records and seeing which one fits. This method has a silent assumption: that every run produces a different model. The blocks below will test that assumption. The first block builds the source, the first three stages of the pipeline that face the model, and the fictional model. ```python # FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based # model are defined within the lesson; no real learning library or model registry # product is called. The registry is also built within the lesson as a list of records. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """window=None -> full history; window=n -> last n periods.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) ``` ## Twelve Runs That Produce the Same Weights The second block produces a record of sixteen runs. The runs really are different from one another: four different feature windows, two different grid sizes, and two different test splits have been tried. Then the record is grouped by the weight triple produced and the test accuracy reported. ```python import collections S = rows() dist = collections.Counter(label(k) for k in S) print("rows:", len(S), " label distribution:", dict(sorted(dist.items())), " base class ratio:", round(dist[0] / len(S), 4), " in test:", round(sum(1 for k in S if k["period"] in TEST and label(k) == 0) / sum(1 for k in S if k["period"] in TEST), 4)) # Run record: four feature windows x two grid sizes x two test splits. FEATS = {p: features(S, p) for p in (None, 1, 2, 3)} RUNS = [] for p in (None, 1, 2, 3): for cand in (27, 21): for per in (TEST, PERIODS[10:]): m = train(S, FEATS[p], GRID[:cand]) RUNS.append({"id": f"K{len(RUNS) + 1:02d}", "window": p, "candidates": cand, "periods": len(per), "weights": m["weights"], "test": evaluate(S, m, FEATS[p], periods=per)}) groups = {} for k in RUNS: groups.setdefault((k["weights"], k["test"]), []).append(k["id"]) print("\nruns:", len(RUNS), " different weight triples:", len(set(k["weights"] for k in RUNS)), " different (weights , test):", len(groups)) print(f"{'weights':<16}{'test':>8}{'runs':>7} ids") for (w, s), ks in groups.items(): print(f"{str(w):<16}{s:>8.4f}{len(ks):>7} {','.join(ks)}") ``` ``` rows: 2200 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 in test: 0.8067 runs: 16 different weight triples: 2 different (weights , test): 4 weights test runs ids (0.6, 0.4, 0) 0.8067 6 K01,K03,K09,K11,K13,K15 (0.6, 0.4, 0) 0.7950 6 K02,K04,K10,K12,K14,K16 (0.6, 0.0, 0) 0.8067 2 K05,K07 (0.6, 0.0, 0) 0.7950 2 K06,K08 ``` Sixteen separate runs produce only **two different weight triples**. The assumption of matching collapses here: someone holding `(0.6, 0.4, 0)` finds **twelve candidates** even if they read the whole run record, and the search goes no lower. This is not a flaw but the natural result of searching: every training job that wanders over a finite grid can arrive at the same candidate, and most of the time it does. The second number the table carries is heavier. Of the twelve runs carrying the same `(0.6, 0.4, 0)` triple, six report their result as **0.8067** and six as **0.7950**. Both are correct; what makes the difference is not the model itself but **over how many periods** the run did its test. So when the question "what was its accuracy" is asked about the model in production, the answer that can be given by going from the weights is a range 0.0117 wide, and that is larger than six times the resolution. Label imbalance makes this even more unfortunate: since the base class ratio in the test is 0.8067, both numbers sit within the band of the majority class, and taking the difference between them for a performance difference is misleading. ## The Run That Stages and Transitions Are Tied To The model registry does not touch the model's file; instead it keeps **a separate record**, and this record ties a stage to a run id. The four stages and the sequence of transitions between them are the contents of the registry; each transition row says which run was moved to which stage. The third block builds the registry and answers two questions in two setups. ```python # Model registry: four stages and the transitions between stages. Every transition is tied to ONE RUN. STAGES = ("candidate", "staging", "production", "archived") TRANSITIONS = [("K03", "candidate", "staging"), ("K03", "staging", "production"), ("K11", "candidate", "staging"), ("K11", "staging", "production"), ("K03", "production", "archived")] CURRENT = {} for run_id, _, target in TRANSITIONS: CURRENT[target] = run_id PRODUCTION_WEIGHTS = next(k for k in RUNS if k["id"] == CURRENT["production"])["weights"] matching = [k for k in RUNS if k["weights"] == PRODUCTION_WEIGHTS] scores = sorted(set(k["test"] for k in matching)) print("weights in production:", PRODUCTION_WEIGHTS, " transitions in the registry:", len(TRANSITIONS)) print(f"\n{'question':<38}{'looked at without registry':>28}{'answer without registry':>25}" f"{'looked at with registry':>25}{'answer':>8}") print(f"{'which run did this weight come from':<38}{len(RUNS):>28}" f"{str(len(matching)) + ' candidates':>25}{1:>25}{CURRENT['production']:>8}") print(f"{'to roll back: previous production':<38}{len(RUNS):>28}{'none':>25}" f"{1:>25}{CURRENT['archived']:>8}") print("\nruns carrying the same weights:", len(matching), " different tests they report:", len(scores), "->", " .. ".join(f"{x:.4f}" for x in scores), " gap:", round(scores[-1] - scores[0], 4)) print("cost of the registry: run records", len(RUNS), "+ transition records", len(TRANSITIONS), "=", len(RUNS) + len(TRANSITIONS), "records ;", "stages:", len(STAGES)) ``` ``` weights in production: (0.6, 0.4, 0) transitions in the registry: 5 question looked at without registry answer without registry looked at with registry answer which run did this weight come from 16 12 candidates 1 K11 to roll back: previous production 16 none 1 K03 runs carrying the same weights: 12 different tests they report: 2 -> 0.7950 .. 0.8067 gap: 0.0117 cost of the registry: run records 16 + transition records 5 = 21 records ; stages: 4 ``` The first row shows what the stage transition gains. In the setup without a registry all sixteen records are looked at and twelve candidates remain; the search does as much work as the number of records looked at and **never narrows**. In the setup with a registry, because the `production` stage is tied to a run id, a single record is read and the answer is `K11`. What is gained is not a saving of fifteen records; what is gained is **a unique answer**, and in the setup without a registry that answer cannot be obtained whatever the number of records. The second row is sharper. When a rollback is needed, the question asked becomes "which run was the previous production run", and in the setup without a registry the answer is not merely hard, it **does not exist**: the only thing in hand is the weights now in place, and the weights of the previous model have been written over. In the registry, on the other hand, it is recorded that `K03` passed from the `production` stage to the `archived` stage; the run to return to is read in a single record. The value of the stages lies less in their saying where the model is than in their saying **where it came from and where to go back to**. The cost of the registry is in the last row and it is modest: sixteen run records and five transition records, **21 records** in total. This cost does not touch the model itself, does not enlarge the weights, and does not slow inference; it only makes it mandatory that one row be written at every stage transition. What is lost when it is not written is exactly what this lesson measured: whom the number in production belongs to. ## Summary - A trained model carries only its weights; it carries within it neither the data version, nor the feature window, nor the test split that produced it, and therefore it has no identity. - Sixteen separate runs produce only two different weight triples; since **12 runs** carry the triple in production exactly, a search done by matching weights never reaches a unique answer. - The runs carrying the same triple report their result as **0.7950** and **0.8067**; the 0.0117 in between is larger than six times the resolution, and label imbalance (base ratio 0.7964, 0.8067 in the test) makes it easy to take this difference for a performance difference. - With a registry that ties four stages and five transitions to a run id, the question "which run did the production model come from" is answered in **a single record**; in the setup without a registry 16 records are read and 12 candidates remain. - The question of rollback has no answer in the setup without a registry; in the registry the archive transition gives the previous production run in a single record, and the registry's total cost is 21 records. ## Next Step In six lessons the model was made **reproducible**. The run itself was recorded and versioned, two different contents given the same name were told apart, the feature went from being computed twice in two places to being computed in a single place, it was counted which item's pinning made the three runs the same, and every stage transition was tied to a run id. Now it is written whom a number standing in production belongs to and how it can be obtained again. The only thing not written is **how this model will get there**. In none of the six lessons was training itself built as an end-to-end pipeline, no gate stopped a new data version or a falling metric from going to release, and the model was never packaged as an interface a caller could use. The next topic closes these three gaps in order: it counts the steps of the training pipeline and its end-to-end re-runnability, measures which breakage the code, data, and model gates stop, and writes how many callers break when the interface contract is not written.