What Is MLOps

Over forty-nine lessons, data was moved, stored, transformed and partitioned. The number of rows each stage of the pipeline drops was counted, how many stages later a corruption was caught was measured, and how rows are laid out and what is gained when they are split among four workers was written down. All of this work had a purpose that was never once opened up: the model this data feeds.

This course asks two questions: whether the same model can be reproduced from the same data, and how a trained model reaches production. The first topic covers the first. This lesson sets up the course’s measurement scheme, the label and the fictional model, and then measures a single thing: how many different numbers come out when the same job is run three times with no item pinned.

• RP1. The source, the pipeline and the model are fictional. No real learning library, experiment tracking setup, feature store, model registry, or serving framework is called, and none is named. The seed is 20260218.
• RP2. The course’s rule: a model’s number is meaningful together with the path that produced it. A number that cannot be reproduced is not a measurement but a memory.
• RP3. In every lesson the baseline is a setup that is unversioned, unpinned and single-path.
• RP4. The label: a period’s consumption is “high” (1) if it exceeds 40 m3 and 0 if it does not. The threshold is a choice, and every result sensitive to it is written together with the threshold.
• RP5. The label distribution is imbalanced: {0: 1752, 1: 448}, that is, a base class ratio of 0.7964. An accuracy number is not counted as measured until it is written how far above this it sits. The baseline procedure was measured in the Supervised Learning course; here it is inherited, not repeated.
• RP6. The fictional model is a threshold with three weights; training is a search on a 27-candidate grid.
• RP7. The training periods are eight (2026-02 through 2026-09, 1600 rows) and the test periods are three (2026-10 through 2026-12, 600 rows).
• RP8. Resolution: since the test set is 600 rows, the smallest step of the accuracy is 0.0017.
• RP9. The training pipeline is handled on a single machine in this course. Partitioning, shuffling, and skew were measured in the Distributed Data Processing course and are not repeated here.
• RP10. The units counted are accuracy, candidate, training row, pinned item and identical run; time is not written in raw milliseconds.
• RP11. Five pinning items: data_version, feature_code, split, grid, rounding. MLOps is the name of a practice, not a product name.

Applying Software Practice to a Model

A program’s output is determined by its source, and all the tools of software practice were built on top of this assumption: version control stores the source, regression tests pin the behavior, continuous integration sends every change down the same path. These procedures were measured in the Continuous Integration and Delivery course and in the Data Pipelines and Orchestration course; they are not repeated.

In a model the assumption fails. What determines the number a model gives is not only the source code. There are at least five items: the data the model sees, how that data is split into training and test, which code computes the feature, which candidates the search space consists of, and how many digits the numbers are rounded to. Of these five items, only one lives inside the source file; the remaining four sit either in a directory, in a command-line option, or nowhere. MLOps is the name of the practice of tying these items to the discipline source code already has: it brings not new mathematics but record keeping.

The Fictional Model and the Label

The block below rebuilds, verbatim, the fictional measurement network source from the Introduction to Data Engineering course and the first three stages of the pipeline, and adds this course’s label, feature generation, and fictional model on top.

# FICTIONAL. Meter readings, the pipeline and the model are defined within the lesson; no real
# learning library, experiment tracking setup, or model registry is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


TRAIN = PERIODS[1:9]       # 2026-02 .. 2026-09
TEST = PERIODS[9:]         # 2026-10 .. 2026-12
THRESHOLD = 40             # consumption above this many m3 counts as "high"


def rows():
    v, _ = s1_screen(generate_source())
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


# ---- Feature generation. Independent of the weights , computed once.
def features(records, window=None, digits=4):
    """window=None -> full history (training path)."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    feats = {}
    for k in records:
        i = PERIODS.index(k["period"])
        feats[(k["subscriber"], k["period"])] = {
            "subscriber_mean": mean(sub[k["subscriber"]], i),
            "region_mean": mean(reg[k["region"]], i),
            "commercial": 1 if k["type"] == "commercial" else 0}
    return feats


# ---- FICTIONAL model: a three-weight threshold. Training = searching the weights on a grid.
GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def train(records, feats, grid=None, threshold=THRESHOLD, periods=None):
    grid = GRID if grid is None else grid
    periods = TRAIN if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    best, best_w = -1.0, None
    for w in grid:
        d = sum(1 for k in subset
                if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k))
        o = d / len(subset)
        if o > best:
            best, best_w = o, w
    return {"weights": best_w, "train_accuracy": round(best, 4),
            "candidates": len(grid), "train_rows": len(subset)}


def evaluate(records, model, feats, threshold=THRESHOLD, periods=None):
    periods = TEST if periods is None else periods
    subset = [k for k in records if k["period"] in periods]
    d = sum(1 for k in subset
            if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0)
            == label(k))
    return round(d / len(subset), 4)


# ---- Reproducibility: the pinned items
ITEMS = ("data_version", "feature_code", "split", "grid", "rounding")


def run(records, pinned, trial=0):
    """Every item that is not pinned changes with the trial."""
    rec = records if "data_version" in pinned else records[: len(records) - 20 * trial]
    window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial)
    digits = 4 if "rounding" in pinned else max(4 - trial, 0)
    grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial]
    periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):]
    feats = features(rec, window, digits)
    m = train(rec, feats, grid)
    return {"weights": m["weights"], "train": m["train_accuracy"],
            "test": evaluate(rec, m, feats, periods=periods)}

The model looks at a subscriber’s past consumption average, the average of the subscriber’s region, and whether the subscriber is commercial; it produces a score with three weights, and if the score exceeds the threshold it says “high”. Training means trying the 27 candidates and choosing the one that gets the most right on the training set. Since the features are independent of the weights, features computes them once.

S = rows()
distribution = {0: 0, 1: 0}
for k in S:
    distribution[label(k)] += 1
print("rows out of the pipeline:", len(S), " threshold:", THRESHOLD)
print("label distribution:", distribution)
print("base class ratio:", round(distribution[0] / len(S), 4))
print()
feats = features(S)
model = train(S, feats)
print("candidates:", model["candidates"], " train rows:", model["train_rows"])
print("weights:", model["weights"])
print("train accuracy:", model["train_accuracy"])
print("test accuracy:", evaluate(S, model, feats))

rows out of the pipeline: 2200  threshold: 40
label distribution: {0: 1752, 1: 448}
base class ratio: 0.7964

candidates: 27  train rows: 1600
weights: (0.6, 0.4, 0)
train accuracy: 0.7931
test accuracy: 0.8067

Two numbers sit side by side here, and both appear throughout the rest of the course. The first is the label distribution: of the 2200 rows, 1752 are “not high” and 448 are “high”, so the distribution is imbalanced. The second is the base class ratio: a rule that learns nothing and says “not high” for every row gets an accuracy of 0.7964. The model got 0.8067; on paper it looks 0.0103 above the base.

A Number Is Not Read Without Its Base Class Ratio

That comparison is wrong, because 0.7964 is the base ratio of all the data, while the model was measured only on the test set. The correct one is made with the test set’s own base ratio.

test_rows = [k for k in S if k["period"] in TEST]
high = sum(1 for k in test_rows if label(k) == 1)
predicted = {0: 0, 1: 0}
hits = 0
for k in test_rows:
    t = 1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > THRESHOLD else 0
    predicted[t] += 1
    if t == 1 and label(k) == 1:
        hits += 1
print("test rows:", len(test_rows), " labeled high:", high)
print("test base class ratio:", round((len(test_rows) - high) / len(test_rows), 4))
print("model test accuracy:", evaluate(S, model, feats))
print("model prediction distribution:", predicted)
print(f"recall: {hits / high:.4f}")
print(f"above the base: {evaluate(S, model, feats) - (len(test_rows) - high) / len(test_rows):.4f}")

test rows: 600  labeled high: 116
test base class ratio: 0.8067
model test accuracy: 0.8067
model prediction distribution: {0: 600, 1: 0}
recall: 0.0000
above the base: 0.0000

The test set’s base ratio is 0.8067, and the model’s accuracy is also 0.8067. The difference is 0.0000. The reason shows in the last two lines: the model does not say “high” to any of the 600 test rows, catches none of the 116 rows labeled high, and its recall stays at 0.0000.

This result is written on purpose. The fictional model does not rise above the baseline, and that frees what this course measures from confusion: from here on, every wobble in the accuracy comes not from the model’s performance but from the path that produced the number. The procedure for building a model, choosing a threshold, and coping with imbalance was measured in the Machine Learning and Data Science curriculum and is not repeated. The rule is this: when 0.8067 is written alone it is a performance claim and it is wrong; when it is written next to the base ratio it says the model measures nothing.

The Same Job, Three Runs, Three Numbers

Now the real question can be asked. The run above was done once and gave one number; if the same job is done again a week later, will the same number come out? Nothing is pinned: whatever is in the store that day is read, and the feature code may have changed, the grid may have shortened, the split may have shifted. run simulates this setup for three trials.

print("pinned items: 0 -- none of the five is pinned")
print(f"{'trial':<5}{'weights':>16}{'train':>9}{'test':>9}")
for trial in range(3):
    r = run(S, (), trial)
    print(f"{trial:<5}{str(r['weights']):>16}{r['train']:>9.4f}{r['test']:>9.4f}")
print()
print(f"{'trial':<5}{'rows':>7}{'candidates':>12}{'test periods':>14}{'test rows':>11}{'base ratio':>12}")
for trial in range(3):
    rec = S[: len(S) - 20 * trial]
    periods = PERIODS[9 + (trial % 2):]
    test3 = [k for k in rec if k["period"] in periods]
    high3 = sum(1 for k in test3 if label(k) == 1)
    print(f"{trial:<5}{len(rec):>7}{len(GRID) - 6 * trial:>12}{len(periods):>14}"
          f"{len(test3):>11}{(len(test3) - high3) / len(test3):>12.4f}")

pinned items: 0 -- none of the five is pinned
trial         weights    train     test
0       (0.6, 0.4, 0)   0.7931   0.8067
1       (0.6, 0.4, 0)   0.7932   0.7955
2       (0.6, 0.4, 0)   0.7926   0.8078

trial   rows  candidates  test periods  test rows  base ratio
0       2200          27             3        600      0.8067
1       2180          21             2        396      0.7955
2       2160          15             3        588      0.8078

The upper table is the course’s baseline: 0.8067, 0.7955, 0.8078. Between the largest and the smallest there is a 0.0123 difference; since the resolution is 0.0017, this is larger than seven steps, so it is not noise. The weights of the three runs, though, are exactly the same: (0.6, 0.4, 0).

The lower table says where the difference comes from. In each trial the number of rows read dropped, the number of candidates fell from 27 to 21 and to 15, and the test periods changed: three periods in the first trial, two in the second. The last column gives the definitive answer. The test base ratios of the three trials are 0.8067, 0.7955, and 0.8078 respectively; the same as the three numbers in the upper table. Since the model says “not high” to every row in all three trials, what is measured is not the model’s performance but which rows fall into the test set in that trial.

From this the course’s rule can be read. The three numbers are not three answers to the same question; they are the answers to three different questions, and which question they answer is written nowhere. If one of these numbers went into a report a week later, the person reading it could not know how much data was read, how many candidates were tried, or which periods were set aside for testing. A number that cannot be reproduced is not a measurement but a memory.

The Course’s Three Columns

For the rest of the course, every lesson puts three numbers side by side. The first is the unpinned baseline: the three runs above, three different numbers. The second is the pinned setup: the same job, with certain items pinned, counting how many of the runs come out identical. The third is the cost: the pinned item, the stored record, the added step. The third column cannot be left empty, because a pinning whose contribution cannot be shown gives an unmeasured assurance.

Summary

• MLOps is the name of the practice of tying the five items that determine a model’s number to the discipline source code already has: data, split, feature code, search space and rounding. Of the five, only one lives inside the source file.
• The fictional model is a threshold with three weights, and training is a search on a 27-candidate grid. The label is produced with a 40 m3 threshold, the distribution comes out {0: 1752, 1: 448}, and the base class ratio is 0.7964.
• The model’s test accuracy is 0.8067, and the test set’s base ratio is also 0.8067. The model says “high” to none of the 600 rows, its recall is 0.0000, and it sits 0.0000 above the base.
• On the unpinned baseline, three runs give 0.8067, 0.7955, and 0.8078; the 0.0123 difference between them is seven times the 0.0017 resolution. The weights of the three runs are exactly the same, and all three numbers equal the base ratio of their own test set.
• The course’s rule is this: a model’s number is meaningful together with the path that produced it, and a number that cannot be reproduced is not a measurement but a memory.

Next Step

Three runs were made and three numbers came out, but none was written down anywhere. The only reason we can see the numbers side by side is that we ran all three in the same block; in a real study these three runs are spread over three separate days, and all that remains between them is a remembered number. The next lesson sets up that record and measures a single question: how many runs have to be repeated to find the weights that produced a result, how many with the record, and how much the record itself costs.