--- title: 'What Is MLOps' source: 'https://academia.sh/en/courses/mlops-fundamentals/what-is-mlops' course: 'MLOps Fundamentals' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - "Defines MLOps practice as tying the items that determine a model's number to software practice." - 'Computes the label distribution and the base class ratio on a fictional model.' - 'Measures the three different test accuracies that three runs give on a baseline with no item pinned and shows the source of the difference.' --- # What Is MLOps Setting up the application of software engineering practice to a model-producing pipeline and measuring the unpinned baseline: when the 2200 rows coming out of the pipeline are labeled with a 40 m3 threshold the distribution is 1752 to 448, the base class ratio is 0.7964, and the model searched on a 27-candidate grid gives 0.7931 in training and 0.8067 on the test set, but since the test set's own base ratio is also 0.8067 the model calls none of the 600 test rows high and its recall stays at 0.0000; when nothing is pinned and the same job is run three times, the weights stay the same in all three runs while the test accuracy comes out 0.8067, 0.7955, and 0.8078, and since all three numbers equal the base ratio of their own test set, the difference comes not from the model but from the set changing. Over forty-nine lessons, data was moved, stored, transformed and partitioned. The number of rows each stage of the pipeline drops was counted, how many stages later a corruption was caught was measured, and how rows are laid out and what is gained when they are split among four workers was written down. All of this work had a purpose that was never once opened up: the **model** this data feeds. This course asks two questions: whether **the same model can be reproduced** from the same data, and **how a trained model reaches production**. The first topic covers the first. This lesson sets up the course's measurement scheme, the label and the fictional model, and then measures a single thing: how many different numbers come out when the same job is run three times with no item pinned. - **RP1.** The source, the pipeline and the model are **fictional**. No real learning library, experiment tracking setup, feature store, model registry, or serving framework is called, and none is named. The seed is **20260218**. - **RP2.** The course's rule: a model's number is meaningful **together with the path that produced it**. **A number that cannot be reproduced is not a measurement but a memory.** - **RP3.** In every lesson the baseline is a **setup that is unversioned, unpinned and single-path**. - **RP4.** The label: a period's consumption is "high" (1) if it exceeds **40 m3** and 0 if it does not. The threshold is **a choice**, and every result sensitive to it is written together with the threshold. - **RP5.** The label distribution is **imbalanced**: `{0: 1752, 1: 448}`, that is, a **base class ratio of 0.7964**. An accuracy number is **not counted as measured** until it is written how far above this it sits. The baseline procedure was measured in the Supervised Learning course; here it is **inherited**, not repeated. - **RP6.** The fictional model is **a threshold with three weights**; training is **a search on a 27-candidate grid**. - **RP7.** The training periods are eight (`2026-02` through `2026-09`, **1600 rows**) and the test periods are three (`2026-10` through `2026-12`, **600 rows**). - **RP8.** Resolution: since the test set is 600 rows, the smallest step of the accuracy is **0.0017**. - **RP9.** **The training pipeline is handled on a single machine in this course.** Partitioning, shuffling, and skew were measured in the Distributed Data Processing course and are not repeated here. - **RP10.** The units counted are accuracy, candidate, training row, pinned item and identical run; **time is not written in raw milliseconds**. - **RP11.** Five pinning items: `data_version`, `feature_code`, `split`, `grid`, `rounding`. MLOps is **the name of a practice**, not a product name. ## Applying Software Practice to a Model A program's output is determined by its source, and all the tools of software practice were built on top of this assumption: version control stores the source, regression tests pin the behavior, continuous integration sends every change down the same path. These procedures were measured in the Continuous Integration and Delivery course and in the Data Pipelines and Orchestration course; they are not repeated. In a model the assumption fails. What determines the number a model gives is not only the source code. There are at least five items: the **data** the model sees, **how** that data is split into training and test, **which code** computes the feature, **which candidates** the search space consists of, and **how many digits** the numbers are rounded to. Of these five items, only one lives inside the source file; the remaining four sit either in a directory, in a command-line option, or nowhere. MLOps is the name of the practice of tying these items to the discipline source code already has: it brings not new mathematics but **record keeping**. ## The Fictional Model and the Label The block below rebuilds, verbatim, the fictional measurement network source from the Introduction to Data Engineering course and the first three stages of the pipeline, and adds this course's label, feature generation, and fictional model on top. ```python # FICTIONAL. Meter readings, the pipeline and the model are defined within the lesson; no real # learning library, experiment tracking setup, or model registry is called. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped TRAIN = PERIODS[1:9] # 2026-02 .. 2026-09 TEST = PERIODS[9:] # 2026-10 .. 2026-12 THRESHOLD = 40 # consumption above this many m3 counts as "high" def rows(): v, _ = s1_screen(generate_source()) v, _ = s2_dedupe(v) v, _ = s3_consumption(v) return v def label(k): return 1 if k["m3"] > THRESHOLD else 0 # ---- Feature generation. Independent of the weights , computed once. def features(records, window=None, digits=4): """window=None -> full history (training path).""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 feats = {} for k in records: i = PERIODS.index(k["period"]) feats[(k["subscriber"], k["period"])] = { "subscriber_mean": mean(sub[k["subscriber"]], i), "region_mean": mean(reg[k["region"]], i), "commercial": 1 if k["type"] == "commercial" else 0} return feats # ---- FICTIONAL model: a three-weight threshold. Training = searching the weights on a grid. GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats, grid=None, threshold=THRESHOLD, periods=None): grid = GRID if grid is None else grid periods = TRAIN if periods is None else periods subset = [k for k in records if k["period"] in periods] best, best_w = -1.0, None for w in grid: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > threshold else 0) == label(k)) o = d / len(subset) if o > best: best, best_w = o, w return {"weights": best_w, "train_accuracy": round(best, 4), "candidates": len(grid), "train_rows": len(subset)} def evaluate(records, model, feats, threshold=THRESHOLD, periods=None): periods = TEST if periods is None else periods subset = [k for k in records if k["period"] in periods] d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > threshold else 0) == label(k)) return round(d / len(subset), 4) # ---- Reproducibility: the pinned items ITEMS = ("data_version", "feature_code", "split", "grid", "rounding") def run(records, pinned, trial=0): """Every item that is not pinned changes with the trial.""" rec = records if "data_version" in pinned else records[: len(records) - 20 * trial] window = None if "feature_code" in pinned else (None if trial == 0 else 12 - trial) digits = 4 if "rounding" in pinned else max(4 - trial, 0) grid = GRID if "grid" in pinned else GRID[: len(GRID) - 6 * trial] periods = TEST if "split" in pinned else PERIODS[9 + (trial % 2):] feats = features(rec, window, digits) m = train(rec, feats, grid) return {"weights": m["weights"], "train": m["train_accuracy"], "test": evaluate(rec, m, feats, periods=periods)} ``` The model looks at a subscriber's past consumption average, the average of the subscriber's region, and whether the subscriber is commercial; it produces a score with three weights, and if the score exceeds the threshold it says "high". Training means trying the 27 candidates and choosing the one that gets the most right on the training set. Since the features are independent of the weights, `features` computes them **once**. ```python S = rows() distribution = {0: 0, 1: 0} for k in S: distribution[label(k)] += 1 print("rows out of the pipeline:", len(S), " threshold:", THRESHOLD) print("label distribution:", distribution) print("base class ratio:", round(distribution[0] / len(S), 4)) print() feats = features(S) model = train(S, feats) print("candidates:", model["candidates"], " train rows:", model["train_rows"]) print("weights:", model["weights"]) print("train accuracy:", model["train_accuracy"]) print("test accuracy:", evaluate(S, model, feats)) ``` ``` rows out of the pipeline: 2200 threshold: 40 label distribution: {0: 1752, 1: 448} base class ratio: 0.7964 candidates: 27 train rows: 1600 weights: (0.6, 0.4, 0) train accuracy: 0.7931 test accuracy: 0.8067 ``` Two numbers sit side by side here, and both appear throughout the rest of the course. The first is **the label distribution**: of the 2200 rows, 1752 are "not high" and 448 are "high", so the distribution is imbalanced. The second is **the base class ratio**: a rule that learns nothing and says "not high" for every row gets an accuracy of **0.7964**. The model got 0.8067; on paper it looks 0.0103 above the base. ## A Number Is Not Read Without Its Base Class Ratio That comparison is wrong, because 0.7964 is the base ratio of **all** the data, while the model was measured only on the test set. The correct one is made with the test set's own base ratio. ```python test_rows = [k for k in S if k["period"] in TEST] high = sum(1 for k in test_rows if label(k) == 1) predicted = {0: 0, 1: 0} hits = 0 for k in test_rows: t = 1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > THRESHOLD else 0 predicted[t] += 1 if t == 1 and label(k) == 1: hits += 1 print("test rows:", len(test_rows), " labeled high:", high) print("test base class ratio:", round((len(test_rows) - high) / len(test_rows), 4)) print("model test accuracy:", evaluate(S, model, feats)) print("model prediction distribution:", predicted) print(f"recall: {hits / high:.4f}") print(f"above the base: {evaluate(S, model, feats) - (len(test_rows) - high) / len(test_rows):.4f}") ``` ``` test rows: 600 labeled high: 116 test base class ratio: 0.8067 model test accuracy: 0.8067 model prediction distribution: {0: 600, 1: 0} recall: 0.0000 above the base: 0.0000 ``` The test set's base ratio is **0.8067**, and the model's accuracy is also **0.8067**. The difference is **0.0000**. The reason shows in the last two lines: the model does not say "high" to **any** of the 600 test rows, catches none of the 116 rows labeled high, and its **recall stays at 0.0000**. This result is written on purpose. The fictional model **does not rise above the baseline**, and that frees what this course measures from confusion: from here on, every wobble in the accuracy comes **not from the model's performance but from the path that produced the number**. The procedure for building a model, choosing a threshold, and coping with imbalance was measured in the Machine Learning and Data Science curriculum and is not repeated. The rule is this: **when 0.8067 is written alone it is a performance claim and it is wrong**; when it is written next to the base ratio it says the model measures nothing. ## The Same Job, Three Runs, Three Numbers Now the real question can be asked. The run above was done once and gave one number; if the same job is done again a week later, will the same number come out? Nothing is pinned: whatever is in the store that day is read, and the feature code may have changed, the grid may have shortened, the split may have shifted. `run` simulates this setup for three trials. ```python print("pinned items: 0 -- none of the five is pinned") print(f"{'trial':<5}{'weights':>16}{'train':>9}{'test':>9}") for trial in range(3): r = run(S, (), trial) print(f"{trial:<5}{str(r['weights']):>16}{r['train']:>9.4f}{r['test']:>9.4f}") print() print(f"{'trial':<5}{'rows':>7}{'candidates':>12}{'test periods':>14}{'test rows':>11}{'base ratio':>12}") for trial in range(3): rec = S[: len(S) - 20 * trial] periods = PERIODS[9 + (trial % 2):] test3 = [k for k in rec if k["period"] in periods] high3 = sum(1 for k in test3 if label(k) == 1) print(f"{trial:<5}{len(rec):>7}{len(GRID) - 6 * trial:>12}{len(periods):>14}" f"{len(test3):>11}{(len(test3) - high3) / len(test3):>12.4f}") ``` ``` pinned items: 0 -- none of the five is pinned trial weights train test 0 (0.6, 0.4, 0) 0.7931 0.8067 1 (0.6, 0.4, 0) 0.7932 0.7955 2 (0.6, 0.4, 0) 0.7926 0.8078 trial rows candidates test periods test rows base ratio 0 2200 27 3 600 0.8067 1 2180 21 2 396 0.7955 2 2160 15 3 588 0.8078 ``` The upper table is the course's baseline: **0.8067**, **0.7955**, **0.8078**. Between the largest and the smallest there is a **0.0123** difference; since the resolution is 0.0017, this is larger than seven steps, so it is not noise. The **weights of the three runs, though, are exactly the same**: `(0.6, 0.4, 0)`. The lower table says where the difference comes from. In each trial the number of rows read dropped, the number of candidates fell from 27 to 21 and to 15, and **the test periods changed**: three periods in the first trial, two in the second. The last column gives the definitive answer. The test base ratios of the three trials are **0.8067**, **0.7955**, and **0.8078** respectively; the **same** as the three numbers in the upper table. Since the model says "not high" to every row in all three trials, what is measured is not the model's performance but which rows fall into the test set in that trial. From this the course's rule can be read. The three numbers are not three answers to the same question; they are the answers to **three different questions**, and which question they answer is written nowhere. If one of these numbers went into a report a week later, the person reading it could not know how much data was read, how many candidates were tried, or which periods were set aside for testing. **A number that cannot be reproduced is not a measurement but a memory.** ## The Course's Three Columns For the rest of the course, every lesson puts three numbers side by side. The first is **the unpinned baseline**: the three runs above, three different numbers. The second is **the pinned setup**: the same job, with certain items pinned, counting how many of the runs come out identical. The third is **the cost**: the pinned item, the stored record, the added step. The third column cannot be left empty, because **a pinning whose contribution cannot be shown gives an unmeasured assurance**. ## Summary - MLOps is the name of the practice of tying the five items that determine a model's number to the discipline source code already has: data, split, feature code, search space and rounding. Of the five, only one lives inside the source file. - The fictional model is a threshold with three weights, and training is a search on a 27-candidate grid. The label is produced with a 40 m3 threshold, the distribution comes out `{0: 1752, 1: 448}`, and the **base class ratio** is **0.7964**. - The model's test accuracy is **0.8067**, and the test set's base ratio is also **0.8067**. The model says "high" to none of the 600 rows, its **recall is 0.0000**, and it sits **0.0000** above the base. - On the unpinned baseline, three runs give **0.8067**, **0.7955**, and **0.8078**; the **0.0123** difference between them is seven times the 0.0017 resolution. The weights of the three runs are exactly the same, and all three numbers equal the base ratio of their own test set. - The course's rule is this: a model's number is meaningful together with the path that produced it, and **a number that cannot be reproduced is not a measurement but a memory**. ## Next Step Three runs were made and three numbers came out, but none was written down anywhere. The only reason we can see the numbers side by side is that we ran all three in the same block; in a real study these three runs are spread over three separate days, and all that remains between them is a remembered number. The next lesson sets up that record and measures a single question: how many runs have to be repeated to find the weights that produced a result, how many with the record, and how much the record itself costs.