Skip to content
academia.sh
tr
Log in

Course Advanced

Model Monitoring and Governance

Published Updated

By the end of this course

Start course

01

Monitoring

Counting by which measure and after how many periods it is seen whether a model in production has degraded: the input summary going from 25.9220 to 27.0472 as the input distribution grows every period and not firing at any alert band, the prediction share in the same drift going from 0.4550 to 0.5450 and firing in three periods at the 0.05 band while accuracy goes from 0.9700 to 0.9800 and never falls, the input summary not changing at all when the rule changes and accuracy collapsing from 0.9700 to 0.8150 while the prediction share stays almost constant, the accuracy of a period being known only two periods later because of the delayed label and the two-period gap between the moment of degradation and the moment it is seen, the proxy metrics used when there is no label firing under a harmless drift and staying silent under a harmful drift, the number of firings growing as the alert band narrows and how many of them correspond to real degradation, the share covered by label collection in production and the bias of the collected label.

  1. 01 Data Drift Measuring the change of the input distribution in production and building the three conditions of a monitorable model: when the decision threshold is searched as a setting separate from the label threshold, when the training criterion is switched from accuracy to balanced accuracy, and when the label is tied to a rule the features see, recall rises from 0.0000 to 1.0000, because the same 135-candidate search stays at 0.5203 balanced accuracy with the old label, which is coin-flip level. In the corrected setup the input drifts period by period, the input summary rises from 25.9220 to 27.0472, the prediction share goes from 0.4550 to 0.5450 and the alert fires in three periods in the 0.05 band, yet accuracy never falls below the 0.9700 baseline; all three firings are false positives, and what the drift actually took away, compared with the no-drift run, is 0.0100 in 2026-12, that is two resolution steps.
  2. 02 Concept Drift Measuring a change of the rule between input and label while the input does not change at all: when the threshold for counting as high drops from 30 to 27, not a single value changes on the input side, yet the label of 132 of the 1200 monitoring rows changes and the accuracy of the model falls from 0.9700 to 0.8150, that is, thirty-one resolution steps. When the same three indicators are read side by side under the two kinds of drift, what the proxy metrics see and what they miss turns into a number: the input summary fires in no band in either scenario, the prediction share fires in three periods in the 0.05 band in both and so cannot tell the harmless from the harmful, and in the 0.10 band it fires in two periods under the harmless drift and in none under the harmful one, while accuracy stays silent under the harmless drift and fires in four periods under the harmful one, remaining the only measure that tells them apart.
  3. 03 Performance Monitoring Measuring when performance can be known in a production setup where the true label does not arrive together with the prediction: because the accuracy of a period can only be computed two periods later, the drop in accuracy that concept drift caused in 2026-09 can be seen for the first time in 2026-11, and 400 predictions are produced unmeasured in this gap. In 2026-09 the actual accuracy is 0.8150 while the newest measure at hand is the 0.9700 of 2026-07, so the indicator is not wrong, it is old. The delay sweep gives a linear cost: at a delay of 0 periods the degradation is seen in the same period and the unmeasured predictions are 0, at 1 period 200, at 2 periods 400, at 3 periods 600, and at 4 periods the degradation is never seen inside the monitoring window; when no label is ever collected, the number of measurable periods is 0 and the degradation is never seen.
  4. 04 Proxy Metrics Counting the measures that remain at hand in the two periods before the label arrives: when the input summary, the prediction share and the input completeness are put side by side in three scenarios, the three proxies of the concept-drift setup come out digit for digit the same as the three proxies of the sound setup, because the rule has changed and the input has not. The prediction share deviates by 0.0957 even in the sound setup and exceeds the 0.05 band, and it rises to 0.1596 in the data-drift setup, yet in that setup accuracy goes from 0.9700 to 0.9800; the input summary stays at 0.0382 in the only scenario where the input really drifts and does not fire, because the drift touches 102 of 200 subscribers and the mean dilutes it; the input completeness deviates by 0.0000 in all three scenarios and shows that it works by dropping two periods at once to 0.8000 when the reading of a single period never arrives. The only measure that separates the scenarios is accuracy, and it arrives two periods later.
  5. 05 Alerts and Thresholds Counting what a narrower alert band gains and what it costs: at the 0.05 band, across three measures and three scenarios, 13 periods fire and 6 of them are false positives; at 0.10 there are 6 firings and 2 false positives; at 0.20 there is no false positive at all but all four real degradations are missed, because a relative band of 0.20 on the 0.9700 baseline asks accuracy to fall below 0.7760 and the lowest value is 0.8150. The prediction share fires in exactly three periods at the 0.05 band both in the sound setup and in the concept-drift setup; the same firing counts as a pure false positive in one place and as a catch in the other, so it counts as unmeasured. Only accuracy never fires in the two harmless scenarios and fires in four periods in the harmful scenario, and the detection lag is two periods.
  6. 06 Feedback Collection Counting the share covered by a label collected from production and the bias it carries: when five collection procedures are tried on the same 1200 rows, collecting labels for the positive predictions only gives a coverage of 0.4858 and shifts the measured accuracy +0.0894 above the real accuracy in the concept-drift production, because that procedure measures precision, not accuracy; period by period the measured accuracy rises from 0.9362 to 1.0000 while the real accuracy falls to 0.8500, and an alert built on the drop never fires. Negative predictions only deviate in the opposite direction by -0.0845, and the rows near the decision threshold deviate by -0.2909 with a coverage of 0.1850. The procedure that collects every fourth row deviates by +0.0033 and +0.0100 with half the coverage; coverage does not correct bias, the bias comes from the selection rule.

02

Release and Governance

Counting the release, change and retirement of a model in an auditable way: in shadow deployment no request goes to the new model yet the measurement can still be taken, and in the canary the share sent to the new model moves the result by a measurable amount; the smallest difference an online experiment can distinguish with two hundred rows per period, and a comparison below that difference remaining inconclusive; a retraining trigger giving the same 0.9800 as the old model in all three windows when the rule has not changed, and the training cost being paid for nothing; decision records making findable which model and which feature produced a prediction; compliance obligations being named by their classes without writing a regulation name, and how many records the engineering counterpart of each class requires; tracking training and inference cost separately; by which measure the retirement of a model is decided, and for how long the retrospective auditability of the removed model is preserved.

  1. 01 Shadow and Canary Deployment Staging the release of a new model and counting how much the share sent to the new model moves both the result and the measure: in the shadow placement 0 requests go to the new model yet the candidate is measured on all two hundred rows and comes out at 0.9650, when the canary share is 0.10 the candidate is measured on only twenty rows, the resolution goes from 0.0050 to 0.0500 and the same candidate looks like 1.0000. The training search cannot separate two candidates at 0.9800 and 0.9741, while in production there is a 0.0150 difference between them, and as the share shrinks the trace of this difference falls below the measure.
  2. 02 Online Experiments Counting how small a difference the comparison of two models in production can distinguish at this set size: with two hundred rows per period the resolution is 0.0050, with six periods it is 0.0008, yet single-period measurements of the same candidate move within a band 0.0400 wide and change sign. A sweep of one hundred thirty-five candidates gives the limit: the smallest difference of the 133 candidates that carry the same sign in all six of six periods is 0.0275, and the difference of the single candidate whose sign moves is 0.0050. No candidate was measured in the interval between them, and the minimum detectable effect lies between these two numbers; every comparison below it is inconclusive.
  3. 03 Retraining Policy Counting the triggers that decide when a model is retrained and what that decision really gains: in production with a data drift all three retraining windows give 0.9800 in the period 2026-12, and this is exactly the same as the number of the old model that was never retrained. When the calendar, indicator and performance triggers are run in two degradation scenarios, the accuracy gained is 0.0000 in all six of six combinations, and in return 3,078,000 rows are scored. That what determines the gain is the window and not the trigger is shown by the retraining triggered at the same moment under a concept drift gaining 0.0000 with window 6 and 0.1100 with window 1.
  4. 04 Model Auditability Counting, with and without a decision record, whether the reason for a single produced prediction can still be reached months later: with the decision record a prediction's model, decision threshold, feature values and score are read on one row, and 1.00 row falls per prediction. Without the record 84,740 rows are read for the same two hundred predictions, which comes to 423.70 per prediction, and even so the model of 189 predictions cannot be determined, because in the canary period the two models say the same thing for 189 of the predictions. The cost of the record is 12,000 stored fields for six periods, and this is 0.8333 times the source readings.
  5. 05 Compliance Requirements Counting how many additional fields and how many stored records each of the four obligation classes owed to the outside asks for: the baseline record keeps three fields per decision, and 36 of the 1200 decisions differ when recomputed from today's source, because the 80 late-arriving rows changed the features of 239 decisions. The record-keeping obligation adds four fields per decision and makes recomputation 1200 of 1200; the explanation obligation adds zero fields, because all five explanation values are derived from those four fields; the right to contest adds three fields only on the 108 negative decisions within one point of the threshold, and those 108 decisions cover 26 of the model's 26 wrong predictions; the retention policy adds no fields at all, it deletes records. With a retention of three periods 600 decision records are dropped, and the number of decisions deleted while the contest window is still open is 600.
  6. 06 Cost Management Tracking training and inference separately and counting what a single total hides: the unit of work is one score evaluation, the inference of six monitoring periods costs 1200 score evaluations and stays constant whatever the frequency, while one retraining costs 162,000. When the retraining frequency goes from once in six periods to every period, the training item rises from 162,000 to 972,000, the training share rises from 0.9926 to 0.9988, and the 2026-12 accuracy stays at 0.9800 in all five setups; because the accuracy gained is 0.0000, the cost per gain is undefined. The cost per prediction read from the total is 811.00, read from inference it is 1.00: making inference completely free removes 0.0012 of the total, moving the frequency from one to six removes 0.8323 of it, and 0.0000 accuracy is paid in return. The crossover point lies between 27,000 and 162,000 requests per period.
  7. 07 Model Retirement Counting by which measure the retirement decision is made and what removing the model leaves behind retrospectively: the indicator criterion fires at 2026-10 in both scenarios, and in one the retirement is unjustified because accuracy went from 0.9700 to 0.9800; the accuracy criterion never fires on the harmless drift and shows the 2026-09 degradation at 2026-11 on the harmful drift, two periods late; the challenger comparison criterion measures an advantage of 0.0000 on data drift and 0.0900 on concept drift and is the only criterion that separates the two scenarios. After removal without a decision record, six of six periods are unexplainable; when the model version record is deleted while the decision record exists, 1200 predictions are read but the ones whose justification is computed are 0. The lesson is the closing of the course and of the curriculum.

Final exam

Model Monitoring and Governance — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close