Skip to content
academia.sh
tr
Log in

Course Advanced

MLOps Fundamentals

Published Updated

By the end of this course

Start course

01

Reproducibility

Counting, item by item, whether the same model can be reproduced from the same data: an unversioned and unpinned baseline in which three runs give three different test accuracies, experiment tracking and what recording parameters, metrics and outputs does to make a run findable again along with the cost of the record itself, data and model versioning separating two different contents given the same name, the consistency a feature store keeps between training and inference and the same weights giving an accuracy of 0.8067 on the training path and 0.7800 on the serving path, the skew closing completely when the feature window is widened to two, environment pinning in which three of five items make three runs identical while the contributions of the grid and rounding items cannot be measured, and which run the stage transitions in a model registry are tied to.

  1. 01 What Is MLOps Setting up the application of software engineering practice to a model-producing pipeline and measuring the unpinned baseline: when the 2200 rows coming out of the pipeline are labeled with a 40 m3 threshold the distribution is 1752 to 448, the base class ratio is 0.7964, and the model searched on a 27-candidate grid gives 0.7931 in training and 0.8067 on the test set, but since the test set's own base ratio is also 0.8067 the model calls none of the 600 test rows high and its recall stays at 0.0000; when nothing is pinned and the same job is run three times, the weights stay the same in all three runs while the test accuracy comes out 0.8067, 0.7955, and 0.8078, and since all three numbers equal the base ratio of their own test set, the difference comes not from the model but from the set changing.
  2. 02 Experiment Tracking Measuring how recording parameters, metrics and outputs makes a run findable again: in a space of eighteen runs with no record, finding the weights that produced a reported number takes 3 re-runs for a target of 0.8078, 2 for 0.7955, 5 for 0.7950, 1 for 0.8067, and 18 for a target that was never produced, while with a full record all five need 0 repeats; the number alone does not identify the run, because fourteen of the eighteen runs give 0.8067 and the candidate found by re-running is only one of fourteen; the cost of the record is nine fields per run and 162 fields in total, while a 36-field ledger that records only metrics shortens only one of five searches.
  3. 03 Data and Model Versioning Measuring how well two different contents given the same name are told apart: when four deliveries come under the same `measurement-2026` name with 2200, 2180, 2160, and again 2200 rows, in an unversioned setup a single name leaves all four runs ambiguous and three different test numbers, 0.8067, 0.8064, and 0.8078, stand behind the same name, while naming by content signature produces three names and brings ambiguous runs to zero; on the model side, because the weights of the four runs are exactly the same, a model identity that looks only at the weights collects the four runs into a single version and three versions separate only when the data signature is added; as the cost, the date-named scheme stores 8740 rows while the content-addressed scheme stores 6540 rows and 8740 rows are read for the signature at every delivery; the output signature does not see a change the pipeline swallowed, and when sixty records arrive twice the source signature changes while the output signature and the accuracy stay exactly the same.
  4. 04 Feature Stores Measuring whether a feature is computed the same way between training and inference: with the same weights the test accuracy comes out 0.8067 through the training path and 0.7800 through the serving path, and the model never changes. The whole 0.0267 in between comes from which history the feature is computed with; it flips the prediction in 22 of the 600 test rows, breaking 19 and fixing three, and closes entirely when the feature window is raised to two. A feature store reduces the two computations to one and zeroes the skew; in return it brings the cost of a store of 6600 values, a fourth stage in the model branch, and rewriting everything stored when the feature code changes.
  5. 05 Environment Pinning Counting, one by one, the items that bring a run to the same result a second time: in an unpinned setup three runs find the same weights yet give three different test accuracies such as 0.8067, 0.7955, and 0.8078, because the rows read, the feature window, the rounding digits, the grid searched, and the test split all move in every run. When three of the five items are pinned the three runs come out exactly the same, the result does not change even if the grid and the rounding are not pinned, and when tested one at a time only two items remain whose contribution can be measured. The cost of pinning is fifteen item records, eighteen extra candidate evaluations, and sixty extra row reads.
  6. 06 Model Registry Counting the lifecycle stages of a model and which run each stage transition is tied to: in a record of sixteen runs only two different weight triples come out, the triple in production is carried exactly by twelve runs, and those twelve runs report their own result with two different numbers such as 0.7950 and 0.8067. Without a model registry, finding which run the production model came from means looking at all sixteen records and the search never goes below twelve candidates, while the previous production run to roll back to cannot be found at all; with a registry that ties four stages and five transitions to a run id, both questions are answered by reading a single record, and the cost of the registry is twenty-one records.

02

Training and Deployment

Counting, step by step, the path by which a trained model reaches production: connecting data preparation, training and evaluation steps into a single pipeline and the rows each step drops, code, data and model gates firing separately in continuous integration and which gate stops which corruption, pinning the runtime and the interface contract in packaging and the caller breaking when no contract is written, the latency class, the cost per prediction and the freshness trade-off of batch, online and streaming inference, request batching and caching lowering the cost per prediction while adding to latency, and what shrinking a model to run at the edge takes from accuracy and on which task it takes it first.

  1. 01 Training Pipelines Counting the training pipeline as the first step of a trained model's path to production: the rows dropped by each of six steps are measured separately, and while the consumption step loses 200 rows its own counter reads 0, the feature step leaves 200 rows history-less without dropping a single row, the split separates 1600 and 600 without dropping a single row; with a label distribution of 1752 to 448 and a base class ratio of 0.7964, the model says 0 to 600 of the 600 rows in the test set and its accuracy of 0.8067 is exactly the base class ratio; when the 40 late-arriving rows are backfilled, the only setup that raises the training rows from 1520 to 1600 is the pipeline that re-runs end to end, and all seven setups report the same 0.8067.
  2. 02 Continuous Integration for Machine Learning Counting which corruption each of the code, data and model gates placed between the run and the release stops: of the six corruption classes the data gate stops four and the model gate stops three, the code gate stops none but catches a feature code change at the first step without looking at production data at all; the unit change stops only at the model gate, by dropping the training accuracy from 0.7931 to 0.5919; the late-arriving row stops at no gate because its deviation in training rows is exactly 0.0500 and the tolerance is 0.0500 with a strict comparison; two of the four runs the data gate stopped would have produced exactly the same model as the reference, and in eight of eight runs the test accuracy of 0.8067 cannot rise above the base class ratio.
  3. 03 Model Packaging Turning a trained model into a single unit with its weights, threshold and interface declaration, and counting the callers that break when no contract is written: when a field name changes in the 1800 calls of three callers the dropped calls rise from 600 to 1200, when the threshold field falls out of the package the number of dropped calls does not change but all 1200 of the 1200 calls that pass change their answer; a contract that declares only the input fields catches two of the three versions while a contract that also declares the output and the parameter catches all three before release, and for that it reads 3 sample records instead of 1800 calls; the label distribution is 1752 to 448, the base class ratio is 0.7964, and the test accuracy of the model, 0.8067, is exactly the base ratio in the test set.
  4. 04 Serving Patterns Running the same package in three placements, batch, online and streaming, and counting the latency class, the freshness and the cost per prediction of the three separately: all three patterns give the same answer to the same 600 requests and the accuracy is 0.8067 in all three, so an observation that looks only at the output cannot tell which pattern ran. The difference is in three columns: the work done at request time is a lookup in the batch and streaming patterns and a scan of 352 to 550 rows in the online pattern; the age of the answer is one period in the batch pattern and zero in the others; the rows read per prediction are 11.00, 463.87 and 3.67. The staleness sweep gives 0 changed predictions up to six periods, and the reason is not the pattern but that the decision score stays at least 7.6468 away from the threshold. The cost is a denominator decision: the batch and online patterns are equal at 4.74 predictions used, and which task rests on which pattern is read from that point.
  5. 05 Inference Scaling Counting how request batching, caching and the choice of accelerator class are traded off between the cost per prediction and the latency of the waiting request: as the batch size goes from one to two, four, eight, sixteen, thirty-two, sixty-four, one hundred twenty-eight and two hundred over the same 600 requests, the cost per prediction falls from 463.87 to 22.00 while the worst wait rises from 0 to 199 arrivals and the average wait to 99.50; the answers are identical in all nine rows and the accuracy stays at 0.8067, so the batch size is not a quality setting. The last group of each period stays at eight requests at sizes sixteen, thirty-two and sixty-four and waits for a group that will never fill; this wait does not show in the worst wait column at all. The crossover point of the two accelerator classes is the ratio of the setup difference to the unit difference and is 80 requests, but in the table the switch shows at one hundred twenty-eight because what is measured is not the requested size but the realized group size. On a stream of 600 requests the cache, with a hit rate of 0.6783, lowers the rows read per request from 464.86 to 148.70 and together with batching gives 26.38; the two gains cannot be multiplied.
  6. 06 Edge and On-Device Inference Counting what moving the model to the source of the request takes from accuracy and from which task it takes first: the number written to the device is five, what is dropped is the history of 341 to 539 rows that produces the region feature, and when that history is dropped the two-class task stays at 0.8067 while the three-tier task falls from 0.4833 to 0.4733. The quantization sweep gives the same order: the two-class task does not move a single prediction down to level 4, the three-tier task moves two predictions at level 16 and five at level 8, and at level 2 the two collapse to 0.3833 and 0.2250. The independence gained by running at the edge is measured in correct answers: when the network outage rate rises to 0.20, the correct answers of the central setup fall from 484 to 391 and from 290 to 234, while the edge stays fixed at 484 and 284; the outage rate at which the two setups are equal is 0.0000 for the two-class task and 0.0207 for the three-tier task. The lesson closes the course and gathers the numbers of the twelve lessons into a single table.

Final exam

MLOps Fundamentals — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close