Skip to content
academia.sh
tr
Log in

Unit 01 of 02

Reproducibility

Counting, item by item, whether the same model can be reproduced from the same data: an unversioned and unpinned baseline in which three runs give three different test accuracies, experiment tracking and what recording parameters, metrics and outputs does to make a run findable again along with the cost of the record itself, data and model versioning separating two different contents given the same name, the consistency a feature store keeps between training and inference and the same weights giving an accuracy of 0.8067 on the training path and 0.7800 on the serving path, the skew closing completely when the feature window is widened to two, environment pinning in which three of five items make three runs identical while the contributions of the grid and rounding items cannot be measured, and which run the stage transitions in a model registry are tied to.

Published Updated

Lessons

  1. 01 What Is MLOps Setting up the application of software engineering practice to a model-producing pipeline and measuring the unpinned baseline: when the 2200 rows coming out of the pipeline are labeled with a 40 m3 threshold the distribution is 1752 to 448, the base class ratio is 0.7964, and the model searched on a 27-candidate grid gives 0.7931 in training and 0.8067 on the test set, but since the test set's own base ratio is also 0.8067 the model calls none of the 600 test rows high and its recall stays at 0.0000; when nothing is pinned and the same job is run three times, the weights stay the same in all three runs while the test accuracy comes out 0.8067, 0.7955, and 0.8078, and since all three numbers equal the base ratio of their own test set, the difference comes not from the model but from the set changing. 14 min read
  2. 02 Experiment Tracking Measuring how recording parameters, metrics and outputs makes a run findable again: in a space of eighteen runs with no record, finding the weights that produced a reported number takes 3 re-runs for a target of 0.8078, 2 for 0.7955, 5 for 0.7950, 1 for 0.8067, and 18 for a target that was never produced, while with a full record all five need 0 repeats; the number alone does not identify the run, because fourteen of the eighteen runs give 0.8067 and the candidate found by re-running is only one of fourteen; the cost of the record is nine fields per run and 162 fields in total, while a 36-field ledger that records only metrics shortens only one of five searches. 13 min read
  3. 03 Data and Model Versioning Measuring how well two different contents given the same name are told apart: when four deliveries come under the same `measurement-2026` name with 2200, 2180, 2160, and again 2200 rows, in an unversioned setup a single name leaves all four runs ambiguous and three different test numbers, 0.8067, 0.8064, and 0.8078, stand behind the same name, while naming by content signature produces three names and brings ambiguous runs to zero; on the model side, because the weights of the four runs are exactly the same, a model identity that looks only at the weights collects the four runs into a single version and three versions separate only when the data signature is added; as the cost, the date-named scheme stores 8740 rows while the content-addressed scheme stores 6540 rows and 8740 rows are read for the signature at every delivery; the output signature does not see a change the pipeline swallowed, and when sixty records arrive twice the source signature changes while the output signature and the accuracy stay exactly the same. 9 min read
  4. 04 Feature Stores Measuring whether a feature is computed the same way between training and inference: with the same weights the test accuracy comes out 0.8067 through the training path and 0.7800 through the serving path, and the model never changes. The whole 0.0267 in between comes from which history the feature is computed with; it flips the prediction in 22 of the 600 test rows, breaking 19 and fixing three, and closes entirely when the feature window is raised to two. A feature store reduces the two computations to one and zeroes the skew; in return it brings the cost of a store of 6600 values, a fourth stage in the model branch, and rewriting everything stored when the feature code changes. 12 min read
  5. 05 Environment Pinning Counting, one by one, the items that bring a run to the same result a second time: in an unpinned setup three runs find the same weights yet give three different test accuracies such as 0.8067, 0.7955, and 0.8078, because the rows read, the feature window, the rounding digits, the grid searched, and the test split all move in every run. When three of the five items are pinned the three runs come out exactly the same, the result does not change even if the grid and the rounding are not pinned, and when tested one at a time only two items remain whose contribution can be measured. The cost of pinning is fifteen item records, eighteen extra candidate evaluations, and sixty extra row reads. 14 min read
  6. 06 Model Registry Counting the lifecycle stages of a model and which run each stage transition is tied to: in a record of sixteen runs only two different weight triples come out, the triple in production is carried exactly by twelve runs, and those twelve runs report their own result with two different numbers such as 0.7950 and 0.8067. Without a model registry, finding which run the production model came from means looking at all sixteen records and the search never goes below twelve candidates, while the previous production run to roll back to cannot be found at all; with a registry that ties four stages and five transitions to a run id, both questions are answered by reading a single record, and the cost of the registry is twenty-one records. 14 min read

Start typing to search.

↑↓ ↵ Esc navigate · open · close