Unit 02 of 02
Release and Governance
Counting the release, change and retirement of a model in an auditable way: in shadow deployment no request goes to the new model yet the measurement can still be taken, and in the canary the share sent to the new model moves the result by a measurable amount; the smallest difference an online experiment can distinguish with two hundred rows per period, and a comparison below that difference remaining inconclusive; a retraining trigger giving the same 0.9800 as the old model in all three windows when the rule has not changed, and the training cost being paid for nothing; decision records making findable which model and which feature produced a prediction; compliance obligations being named by their classes without writing a regulation name, and how many records the engineering counterpart of each class requires; tracking training and inference cost separately; by which measure the retirement of a model is decided, and for how long the retrospective auditability of the removed model is preserved.
Published Updated
Lessons
- 01 Shadow and Canary Deployment Staging the release of a new model and counting how much the share sent to the new model moves both the result and the measure: in the shadow placement 0 requests go to the new model yet the candidate is measured on all two hundred rows and comes out at 0.9650, when the canary share is 0.10 the candidate is measured on only twenty rows, the resolution goes from 0.0050 to 0.0500 and the same candidate looks like 1.0000. The training search cannot separate two candidates at 0.9800 and 0.9741, while in production there is a 0.0150 difference between them, and as the share shrinks the trace of this difference falls below the measure. 13 min read
- 02 Online Experiments Counting how small a difference the comparison of two models in production can distinguish at this set size: with two hundred rows per period the resolution is 0.0050, with six periods it is 0.0008, yet single-period measurements of the same candidate move within a band 0.0400 wide and change sign. A sweep of one hundred thirty-five candidates gives the limit: the smallest difference of the 133 candidates that carry the same sign in all six of six periods is 0.0275, and the difference of the single candidate whose sign moves is 0.0050. No candidate was measured in the interval between them, and the minimum detectable effect lies between these two numbers; every comparison below it is inconclusive. 13 min read
- 03 Retraining Policy Counting the triggers that decide when a model is retrained and what that decision really gains: in production with a data drift all three retraining windows give 0.9800 in the period 2026-12, and this is exactly the same as the number of the old model that was never retrained. When the calendar, indicator and performance triggers are run in two degradation scenarios, the accuracy gained is 0.0000 in all six of six combinations, and in return 3,078,000 rows are scored. That what determines the gain is the window and not the trigger is shown by the retraining triggered at the same moment under a concept drift gaining 0.0000 with window 6 and 0.1100 with window 1. 7 min read
- 04 Model Auditability Counting, with and without a decision record, whether the reason for a single produced prediction can still be reached months later: with the decision record a prediction's model, decision threshold, feature values and score are read on one row, and 1.00 row falls per prediction. Without the record 84,740 rows are read for the same two hundred predictions, which comes to 423.70 per prediction, and even so the model of 189 predictions cannot be determined, because in the canary period the two models say the same thing for 189 of the predictions. The cost of the record is 12,000 stored fields for six periods, and this is 0.8333 times the source readings. 9 min read
- 05 Compliance Requirements Counting how many additional fields and how many stored records each of the four obligation classes owed to the outside asks for: the baseline record keeps three fields per decision, and 36 of the 1200 decisions differ when recomputed from today's source, because the 80 late-arriving rows changed the features of 239 decisions. The record-keeping obligation adds four fields per decision and makes recomputation 1200 of 1200; the explanation obligation adds zero fields, because all five explanation values are derived from those four fields; the right to contest adds three fields only on the 108 negative decisions within one point of the threshold, and those 108 decisions cover 26 of the model's 26 wrong predictions; the retention policy adds no fields at all, it deletes records. With a retention of three periods 600 decision records are dropped, and the number of decisions deleted while the contest window is still open is 600. 11 min read
- 06 Cost Management Tracking training and inference separately and counting what a single total hides: the unit of work is one score evaluation, the inference of six monitoring periods costs 1200 score evaluations and stays constant whatever the frequency, while one retraining costs 162,000. When the retraining frequency goes from once in six periods to every period, the training item rises from 162,000 to 972,000, the training share rises from 0.9926 to 0.9988, and the 2026-12 accuracy stays at 0.9800 in all five setups; because the accuracy gained is 0.0000, the cost per gain is undefined. The cost per prediction read from the total is 811.00, read from inference it is 1.00: making inference completely free removes 0.0012 of the total, moving the frequency from one to six removes 0.8323 of it, and 0.0000 accuracy is paid in return. The crossover point lies between 27,000 and 162,000 requests per period. 10 min read
- 07 Model Retirement Counting by which measure the retirement decision is made and what removing the model leaves behind retrospectively: the indicator criterion fires at 2026-10 in both scenarios, and in one the retirement is unjustified because accuracy went from 0.9700 to 0.9800; the accuracy criterion never fires on the harmless drift and shows the 2026-09 degradation at 2026-11 on the harmful drift, two periods late; the challenger comparison criterion measures an advantage of 0.0000 on data drift and 0.0900 on concept drift and is the only criterion that separates the two scenarios. After removal without a decision record, six of six periods are unexplainable; when the model version record is deleted while the decision record exists, 1200 predictions are read but the ones whose justification is computed are 0. The lesson is the closing of the course and of the curriculum. 13 min read