Model Registry

Five items were pinned and the three runs now give the same number. That is the answer to the question of reproducibility, but it opens one more question: which run did the model in hand come from? The only thing a trained model carries is its weights; the weights do not carry within them the data version, the feature window, or the test split that produced them.

This lesson measures that gap. Whether a model is being tried, being tested, or in service, and with which run it came to that state, lives not in the model’s file but in a separate record. A model registry is exactly this record: it ties the stages and the transitions between stages to a run id. What is to be measured is how many records that tie makes you read when answering a question.

• RP57. The source, the first three stages of the pipeline, the fictional model, and the registry are fictional; no real learning library or registry product is called. The seed is 20260218.
• RP58. A run is a single training job executed with particular settings, and it carries an id. A model is the weight triple that run produced, and it carries no id.
• RP59. A stage is the state of the model in its lifecycle. Four are counted here: candidate, staging, production, archived. A transition is the record that moves a model from one stage to another, and it is tied to a run id.
• RP60. The baseline is the setup without a registry: the runs are recorded somewhere, but no stage is tied to a run. The only way to get from a model at a stage to a run is matching the weights.
• RP61. The label is imbalanced: {0: 1752, 1: 448}, the base class ratio is 0.7964, and in the test set it is 0.8067. Accuracy read alone misleads; in this lesson two runs will carry the same model and report different numbers.
• RP62. The run record itself and data versioning were handled in the earlier lessons of this topic; the only thing added here is tying the stage transition to the run. The lineage procedure was measured in the Introduction to Data Engineering course and is not repeated.
• RP63. The registry is built on a record of sixteen runs. The runs are the combination of four feature windows, two grid sizes, and two test splits.
• RP64. Resolution: the full test set is 600 rows, the step is 1/600 = 0.0017. In this lesson the difference between the two numbers reported by the same weights is 0.0117, that is, larger than six times the step.

The Model Has No Identity

A trained model is the output of a procedure, and the output does not contain the procedure that produced it. Here the model consists of three numbers; in a real setup it would be far larger, but the situation does not change. Inside the model it is not written which data it was trained with, which feature window was chosen, or over which periods the test was done. The only thing that carries this information is the run record, and unless the tie between the model and the run is built separately, it does not exist.

When the tie is not built, the only thing that can be done is matching: comparing the weights in hand with the weights in the run records and seeing which one fits. This method has a silent assumption: that every run produces a different model. The blocks below will test that assumption. The first block builds the source, the first three stages of the pipeline that face the model, and the fictional model.

# FICTIONAL. Meter readings, the first three stages of the pipeline, and the threshold-based
# model are defined within the lesson; no real learning library or model registry
# product is called. The registry is also built within the lesson as a list of records.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    """Consumption from the period difference. Without a previous period the row cannot be computed."""
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


TRAIN = PERIODS[1:9]        # 2026-02 .. 2026-09
TEST = PERIODS[9:]          # 2026-10 .. 2026-12
THRESHOLD = 40              # consumption above this many m3 counts as "high"


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


def features(records, window=None, digits=4):
    """window=None -> full history; window=n -> last n periods."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {
            "subscriber_mean": mean(sub[k["subscriber"]], i),
            "region_mean": mean(reg[k["region"]], i),
            "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def train(records, feats, grid=None, threshold=THRESHOLD, periods=None):
    grid = GRID if grid is None else grid
    periods = TRAIN if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    best, best_w = -1.0, None
    for w in grid:
        d = sum(1 for k in subset
                if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k))
        o = d / len(subset)
        if o > best:
            best, best_w = o, w
    return {"weights": best_w, "train_accuracy": round(best, 4),
            "candidates": len(grid), "train_rows": len(subset)}


def evaluate(records, model, feats, threshold=THRESHOLD, periods=None):
    periods = TEST if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    d = sum(1 for k in subset
            if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0)
            == label(k))
    return round(d / len(subset), 4)

Twelve Runs That Produce the Same Weights

The second block produces a record of sixteen runs. The runs really are different from one another: four different feature windows, two different grid sizes, and two different test splits have been tried. Then the record is grouped by the weight triple produced and the test accuracy reported.

import collections

S = rows()
dist = collections.Counter(label(k) for k in S)
print("rows:", len(S), " label distribution:", dict(sorted(dist.items())),
      " base class ratio:", round(dist[0] / len(S), 4),
      " in test:", round(sum(1 for k in S if k["period"] in TEST and label(k) == 0)
                         / sum(1 for k in S if k["period"] in TEST), 4))

# Run record: four feature windows x two grid sizes x two test splits.
FEATS = {p: features(S, p) for p in (None, 1, 2, 3)}
RUNS = []
for p in (None, 1, 2, 3):
    for cand in (27, 21):
        for per in (TEST, PERIODS[10:]):
            m = train(S, FEATS[p], GRID[:cand])
            RUNS.append({"id": f"K{len(RUNS) + 1:02d}", "window": p, "candidates": cand,
                         "periods": len(per), "weights": m["weights"],
                         "test": evaluate(S, m, FEATS[p], periods=per)})

groups = {}
for k in RUNS:
    groups.setdefault((k["weights"], k["test"]), []).append(k["id"])
print("\nruns:", len(RUNS), " different weight triples:",
      len(set(k["weights"] for k in RUNS)), " different (weights , test):", len(groups))
print(f"{'weights':<16}{'test':>8}{'runs':>7}   ids")
for (w, s), ks in groups.items():
    print(f"{str(w):<16}{s:>8.4f}{len(ks):>7}   {','.join(ks)}")

rows: 2200  label distribution: {0: 1752, 1: 448}  base class ratio: 0.7964  in test: 0.8067

runs: 16  different weight triples: 2  different (weights , test): 4
weights             test   runs   ids
(0.6, 0.4, 0)     0.8067      6   K01,K03,K09,K11,K13,K15
(0.6, 0.4, 0)     0.7950      6   K02,K04,K10,K12,K14,K16
(0.6, 0.0, 0)     0.8067      2   K05,K07
(0.6, 0.0, 0)     0.7950      2   K06,K08

Sixteen separate runs produce only two different weight triples. The assumption of matching collapses here: someone holding (0.6, 0.4, 0) finds twelve candidates even if they read the whole run record, and the search goes no lower. This is not a flaw but the natural result of searching: every training job that wanders over a finite grid can arrive at the same candidate, and most of the time it does.

The second number the table carries is heavier. Of the twelve runs carrying the same (0.6, 0.4, 0) triple, six report their result as 0.8067 and six as 0.7950. Both are correct; what makes the difference is not the model itself but over how many periods the run did its test. So when the question “what was its accuracy” is asked about the model in production, the answer that can be given by going from the weights is a range 0.0117 wide, and that is larger than six times the resolution. Label imbalance makes this even more unfortunate: since the base class ratio in the test is 0.8067, both numbers sit within the band of the majority class, and taking the difference between them for a performance difference is misleading.

The Run That Stages and Transitions Are Tied To

The model registry does not touch the model’s file; instead it keeps a separate record, and this record ties a stage to a run id. The four stages and the sequence of transitions between them are the contents of the registry; each transition row says which run was moved to which stage. The third block builds the registry and answers two questions in two setups.

# Model registry: four stages and the transitions between stages. Every transition is tied to ONE RUN.
STAGES = ("candidate", "staging", "production", "archived")
TRANSITIONS = [("K03", "candidate", "staging"), ("K03", "staging", "production"),
               ("K11", "candidate", "staging"), ("K11", "staging", "production"),
               ("K03", "production", "archived")]
CURRENT = {}
for run_id, _, target in TRANSITIONS:
    CURRENT[target] = run_id
PRODUCTION_WEIGHTS = next(k for k in RUNS if k["id"] == CURRENT["production"])["weights"]

matching = [k for k in RUNS if k["weights"] == PRODUCTION_WEIGHTS]
scores = sorted(set(k["test"] for k in matching))
print("weights in production:", PRODUCTION_WEIGHTS, " transitions in the registry:", len(TRANSITIONS))
print(f"\n{'question':<38}{'looked at without registry':>28}{'answer without registry':>25}"
      f"{'looked at with registry':>25}{'answer':>8}")
print(f"{'which run did this weight come from':<38}{len(RUNS):>28}"
      f"{str(len(matching)) + ' candidates':>25}{1:>25}{CURRENT['production']:>8}")
print(f"{'to roll back: previous production':<38}{len(RUNS):>28}{'none':>25}"
      f"{1:>25}{CURRENT['archived']:>8}")
print("\nruns carrying the same weights:", len(matching), " different tests they report:",
      len(scores), "->", " .. ".join(f"{x:.4f}" for x in scores),
      " gap:", round(scores[-1] - scores[0], 4))
print("cost of the registry: run records", len(RUNS), "+ transition records", len(TRANSITIONS),
      "=", len(RUNS) + len(TRANSITIONS), "records ;", "stages:", len(STAGES))

weights in production: (0.6, 0.4, 0)  transitions in the registry: 5

question                                looked at without registry  answer without registry  looked at with registry  answer
which run did this weight come from                             16            12 candidates                        1     K11
to roll back: previous production                               16                     none                        1     K03

runs carrying the same weights: 12  different tests they report: 2 -> 0.7950 .. 0.8067  gap: 0.0117
cost of the registry: run records 16 + transition records 5 = 21 records ; stages: 4

The first row shows what the stage transition gains. In the setup without a registry all sixteen records are looked at and twelve candidates remain; the search does as much work as the number of records looked at and never narrows. In the setup with a registry, because the production stage is tied to a run id, a single record is read and the answer is K11. What is gained is not a saving of fifteen records; what is gained is a unique answer, and in the setup without a registry that answer cannot be obtained whatever the number of records.

The second row is sharper. When a rollback is needed, the question asked becomes “which run was the previous production run”, and in the setup without a registry the answer is not merely hard, it does not exist: the only thing in hand is the weights now in place, and the weights of the previous model have been written over. In the registry, on the other hand, it is recorded that K03 passed from the production stage to the archived stage; the run to return to is read in a single record. The value of the stages lies less in their saying where the model is than in their saying where it came from and where to go back to.

The cost of the registry is in the last row and it is modest: sixteen run records and five transition records, 21 records in total. This cost does not touch the model itself, does not enlarge the weights, and does not slow inference; it only makes it mandatory that one row be written at every stage transition. What is lost when it is not written is exactly what this lesson measured: whom the number in production belongs to.

Summary

• A trained model carries only its weights; it carries within it neither the data version, nor the feature window, nor the test split that produced it, and therefore it has no identity.
• Sixteen separate runs produce only two different weight triples; since 12 runs carry the triple in production exactly, a search done by matching weights never reaches a unique answer.
• The runs carrying the same triple report their result as 0.7950 and 0.8067; the 0.0117 in between is larger than six times the resolution, and label imbalance (base ratio 0.7964, 0.8067 in the test) makes it easy to take this difference for a performance difference.
• With a registry that ties four stages and five transitions to a run id, the question “which run did the production model come from” is answered in a single record; in the setup without a registry 16 records are read and 12 candidates remain.
• The question of rollback has no answer in the setup without a registry; in the registry the archive transition gives the previous production run in a single record, and the registry’s total cost is 21 records.

Next Step

In six lessons the model was made reproducible. The run itself was recorded and versioned, two different contents given the same name were told apart, the feature went from being computed twice in two places to being computed in a single place, it was counted which item’s pinning made the three runs the same, and every stage transition was tied to a run id. Now it is written whom a number standing in production belongs to and how it can be obtained again.

The only thing not written is how this model will get there. In none of the six lessons was training itself built as an end-to-end pipeline, no gate stopped a new data version or a falling metric from going to release, and the model was never packaged as an interface a caller could use. The next topic closes these three gaps in order: it counts the steps of the training pipeline and its end-to-end re-runnability, measures which breakage the code, data, and model gates stop, and writes how many callers break when the interface contract is not written.