--- title: 'Cost Management' source: 'https://academia.sh/en/courses/model-monitoring/cost-management' course: 'Model Monitoring and Governance' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures training and inference cost separately with a single unit of work.' - 'Shows which item grows as the retraining frequency rises and which one stays constant.' - 'Separates, with numbers, which decision the cost per prediction read from the single total hides.' - 'Computes the crossover point at which inference passes training, in requests per period.' --- # Cost Management Tracking training and inference separately and counting what a single total hides: the unit of work is one score evaluation, the inference of six monitoring periods costs 1200 score evaluations and stays constant whatever the frequency, while one retraining costs 162,000. When the retraining frequency goes from once in six periods to every period, the training item rises from 162,000 to 972,000, the training share rises from 0.9926 to 0.9988, and the 2026-12 accuracy stays at 0.9800 in all five setups; because the accuracy gained is 0.0000, the cost per gain is undefined. The cost per prediction read from the total is 811.00, read from inference it is 1.00: making inference completely free removes 0.0012 of the total, moving the frequency from one to six removes 0.8323 of it, and 0.0000 accuracy is paid in return. The crossover point lies between 27,000 and 162,000 requests per period. The previous lesson took the debt owed to the outside and wrote it as records: four obligation classes, seven fields per decision, 8402 stored values. Everything counted was written to **storage**. The running of the model itself, the work repeated every period, entered no line. That work is not one piece. A production model does two separate jobs: it is **retrained** period by period and it produces **inference** every period. The two are usually written to one budget item and read as a single total. This lesson splits that total in two, sweeps the retraining frequency and counts which decision the single number makes invisible. - **RG47.** The source, the pipeline, the feature path, the model and the cost counter are **fictional**. No cloud service, accelerator, product or price is called; no currency is written. Seed 20260218. - **RG48.** The unit of work is **one score evaluation**: multiplying a row's features by the weights and summing. Training and inference do only this, so both enter the same table **with the same unit**. The counter is inside the `score` function; cost is not assumed, it is **measured**. - **RG49.** No time is written. Cost is counted in **score evaluations**, delay in **periods**. - **RG50.** The baseline is **the setup without retraining**: the first model is trained once and never changes over the six monitoring periods. - **RG51.** In production the input drifts: after `2026-09` commercial consumption grows by 0.06 per period. The model is `(0.6, 0.0, 4)`, the decision threshold 18, the candidates 135, the training rows 1600. Monitoring is six periods, the **base class ratio 0.5342**, the step of period accuracy 0.0050. - **RG52.** The inference volume is 200 requests per period and is swept as a **free variable** in the last section. The cost per prediction was measured in the Building Applications with Model APIs course of the AI Engineering curriculum and in the scaling lesson of the MLOps Fundamentals course; their procedures are **not repeated**, and the measure here adds a single thing on top of them: **training and inference are counted separately.** ## Counting the Two Items Separately The first block sets up the source, the feature path and the model, places a counter in the `score` function and sweeps the retraining frequency. ```python # FICTIONAL. The source , the consumption stage of the pipeline , the feature path , the # threshold model and the cost counter are generated inside the lesson; no cloud service , # accelerator , product or price is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) previous = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if previous is not None: out.append(dict(k, m3=int(k["index"]) - int(previous["index"]))) return out def features(records): """Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def average(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), 4) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": average(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": average(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) MONITORED = PERIODS[6:] RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 COUNTER = {"score": 0} # the single unit of work: one score evaluation def score(o, w): COUNTER["score"] += 1 return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def train(records, feats, periods): """The training criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched.""" subset = [k for k in records if k["period"] in periods] best = (-1.0, None, None) for w in GRID: for decision in DECISION_GRID: tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, decision), label(o) tp, fp, fn, tn = (tp + (p and g), fp + (p and not g), fn + (not p and g), tn + (not p and not g)) total = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0) if total / 2 > best[0]: best = (total / 2, w, decision) return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4), "rows": len(subset)} def drift(records, start="2026-09", slope=0.06): """FICTIONAL data drift: commercial consumption grows by `slope` every period.""" b = PERIODS.index(start) return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4)) if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k) for k in records] def accuracy(records, m, feats, period): subset = [k for k in records if k["period"] == period] d = sum(1 for k in subset if predict(feats[(k["subscriber"], k["period"])], m["weights"], m["decision_threshold"]) == label(feats[(k["subscriber"], k["period"])])) return round(d / len(subset), 4) CLEAN = consumption(generate_source()) FEATS_C = features(CLEAN) monitored_c = [k for k in CLEAN if k["period"] in MONITORED] p1 = sum(label(FEATS_C[(k["subscriber"], k["period"])]) for k in monitored_c) / len(monitored_c) DRIFTED = drift(CLEAN) # the input drifts in production FEATS = features(DRIFTED) COUNTER["score"] = 0 FIRST = train(DRIFTED, FEATS, PERIODS[1:9]) FIRST_COST = COUNTER["score"] print(f"base class ratio {round(max(p1, 1 - p1), 4)} , monitoring periods {len(MONITORED)} ," f" requests per period 200") print(f"first training: weights {FIRST['weights']} , decision threshold {FIRST['decision_threshold']} ," f" rows {FIRST['rows']} , candidates {len(GRID) * len(DECISION_GRID)}") print(f"measured training cost (score evaluations): {FIRST_COST}") COUNTER["score"] = 0 last = accuracy(DRIFTED, FIRST, FEATS, "2026-12") print(f"inference cost of one period (score evaluations): {COUNTER['score']} ," f" 2026-12 accuracy {last}") WINDOW = 6 print("\nfrequency retrainings training cost inference cost total training share" " 2026-12 accuracy") RESULT = [] for f in (0, 6, 3, 2, 1): COUNTER["score"] = 0 m = FIRST if f: for i, d in enumerate(MONITORED): if (i + 1) % f == 0: j = PERIODS.index(d) m = train(DRIFTED, FEATS, [x for x in PERIODS if j - WINDOW < PERIODS.index(x) <= j]) training = COUNTER["score"] COUNTER["score"] = 0 acc = accuracy(DRIFTED, m, FEATS, "2026-12") inference = len(MONITORED) * 200 total = training + inference RESULT.append((f, training, inference, total, acc)) print(f"{f if f else '-':>9} {len(MONITORED) // f if f else 0:>11} {training:>13}" f" {inference:>14} {total:>6} {round(training / total, 4):>14} {acc:>16}") ``` ``` base class ratio 0.5342 , monitoring periods 6 , requests per period 200 first training: weights (0.6, 0.0, 4) , decision threshold 18 , rows 1600 , candidates 135 measured training cost (score evaluations): 216000 inference cost of one period (score evaluations): 200 , 2026-12 accuracy 0.98 frequency retrainings training cost inference cost total training share 2026-12 accuracy - 0 0 1200 1200 0.0 0.98 6 1 162000 1200 163200 0.9926 0.98 3 2 324000 1200 325200 0.9963 0.98 2 3 486000 1200 487200 0.9975 0.98 1 6 972000 1200 973200 0.9988 0.98 ``` The scale difference between the two items comes directly from the counter. One period of inference is **200 score evaluations** and comes to 1200 over six periods; the first training scans 1600 rows with 135 candidates and costs **216,000**. A single retraining with a six-period window is **162,000**, that is **135 times** the inference of the whole monitoring horizon. Down the frequency column the inference item **never changes**; only the training item does, and it rises from 162,000 to **972,000** when going from once in six periods to every period. The last column writes what this increase buys. The `2026-12` accuracy is **0.9800** in all five setups — **exactly the same** as that of the model that is never retrained. This is the cost-side counterpart of the second claim of the course: the input drifted, the alert fired, six retrainings were run and **the accuracy did not move by a single step**. Since the period step is 0.0050, the 0.0000 is truly zero, not a small gain too small to measure. As long as the rule does not change, a correct model stays correct on a drifted input too, and retraining brings only the **cost**. ## What the Single Total Hides The second block reads the same results per prediction and sweeps the inference volume. ```python # This block continues the RESULT , MONITORED , FIRST_COST and WINDOW definitions of the first block. baseline_accuracy = RESULT[0][4] n_predictions = len(MONITORED) * 200 print("frequency per prediction from total per prediction from inference" " accuracy gained cost per gain") for f, training, inference, total, acc in RESULT: gain = round(acc - baseline_accuracy, 4) per_gain = "undefined" if gain <= 0 else str(round(training / gain)) print(f"{f if f else '-':>9} {round(total / n_predictions, 2):>25}" f" {round(inference / n_predictions, 2):>29} {gain:>15} {per_gain:>13}") print(f"\nshare removed from the total by making inference completely free:" f" {round(RESULT[-1][2] / RESULT[-1][3], 4)}") print(f"share removed from the total by moving from frequency 1 to frequency 6:" f" {round((RESULT[-1][3] - RESULT[1][3]) / RESULT[-1][3], 4)}" f" , accuracy paid {round(RESULT[1][4] - RESULT[-1][4], 4)}") print("\nrequests per period inference (6 periods) training frequency 6 training frequency 1" " which inference exceeded") for requests in (200, 2000, 20000, 50000, 200000): c = len(MONITORED) * requests passed = [str(f) for f, e, _, _, _ in RESULT[1:] if c > e] print(f"{requests:>19} {c:>21} {RESULT[1][1]:>20} {RESULT[-1][1]:>20}" f" {(' , '.join(passed) if passed else 'none'):>24}") print(f"\ncrossover point (requests per period): frequency 6 -> {RESULT[1][1] // len(MONITORED)}" f" , frequency 1 -> {RESULT[-1][1] // len(MONITORED)}") # Monitoring's own item: three indicators are computed every period. indicators = len(MONITORED) * 200 # prediction share , 0 if the inference predictions are reused print(f"monitoring item (score evaluations): at most {indicators} , when inference predictions" f" are reused 0 ; one retraining {RESULT[1][1]}") ``` ``` frequency per prediction from total per prediction from inference accuracy gained cost per gain - 1.0 1.0 0.0 undefined 6 136.0 1.0 0.0 undefined 3 271.0 1.0 0.0 undefined 2 406.0 1.0 0.0 undefined 1 811.0 1.0 0.0 undefined share removed from the total by making inference completely free: 0.0012 share removed from the total by moving from frequency 1 to frequency 6: 0.8323 , accuracy paid 0.0 requests per period inference (6 periods) training frequency 6 training frequency 1 which inference exceeded 200 1200 162000 972000 none 2000 12000 162000 972000 none 20000 120000 162000 972000 none 50000 300000 162000 972000 6 200000 1200000 162000 972000 6 , 3 , 2 , 1 crossover point (requests per period): frequency 6 -> 27000 , frequency 1 -> 162000 monitoring item (score evaluations): at most 1200 , when inference predictions are reused 0 ; one retraining 162000 ``` The first two columns answer the question of the lesson. When the total is read as one number and divided by the number of predictions, the cost per prediction is **811.00**; the same computation from the inference item alone gives **1.00**. The entire 810-fold difference is training and **belongs to no prediction**. A reader who sees only this number concludes that the prediction is expensive and turns to making inference cheaper: batching requests, a cache, shrinking the model. **The ceiling of that road is 0.0012** — even if inference were completely free, only 1.2 per thousand of the total would be removed. The decision that stays invisible lies the other way. Moving the frequency from every period to once in six periods removes **0.8323** of the total and costs **0.0000 accuracy**. The single total hides this decision, because in the total nothing separates training from inference; the training share lies between 0.9926 and 0.9988 and **the frequency at which one stands cannot be read from the total**. When the two items are counted separately the decision is a single line. The cost per gain column says the same thing once more: **undefined** in all five setups, because the divisor is zero. **The justification of a cost item can only be written together with a gain; spending without a shown gain counts as unmeasured.** The last sweep shows that the decision depends on volume. At 200 requests per period inference exceeds no training item; at 50,000 requests it exceeds only the sparsest frequency; at 200,000 requests it exceeds all four at once. The **crossover point** is **27,000** requests per period for frequency 6 and **162,000** for frequency 1. This means a conclusion such as "training always dominates" cannot be written: the dominant item depends on which side of the volume one stands, and **the single total does not show that side either**. The item of monitoring itself is the smallest row of this table. Computing three indicators over six periods is at most 1200 score evaluations, and it falls to zero when the predictions produced in inference are reused. But the work monitoring **triggers** is not small: the retraining started by a single alert that fires is 162,000. **The cost of monitoring is not in the indicator itself but in the response given to the indicator.** ## Summary - The unit of work is a **score evaluation** and the counter is inside the `score` function; training and inference enter the same table with the same unit. One period of inference is 200, the first training 216,000, a retraining with a six-period window **162,000** score evaluations. - In the frequency sweep the inference item **stays constant at 1200**, the training item rises from 162,000 to **972,000** and the training share rises from 0.9926 to 0.9988. - The `2026-12` accuracy is **0.9800** in all five setups. Since the period step is 0.0050, the accuracy gained is truly **0.0000** and the cost per gain is **undefined**. - The cost per prediction read from the total is **811.00**, read from inference **1.00**. Making inference free removes 0.0012 of the total, thinning out the frequency removes **0.8323** of it; the second costs 0.0000 accuracy and is **invisible in the single total**. - The crossover point lies between **27,000** and **162,000** requests per period; which item dominates depends on volume. The monitoring item is at most 1200; the retraining it triggers is 162,000. ## Next Step The two items were separated and a counterpart was written for both: retraining brings 0.0000 accuracy, while inference produces the whole value of the model for 200 requests per period. The only column missing from this table is the column of **the end**. Retraining can be thinned out, inference can be made cheaper, records can be shortened; but when a model will **end** is written nowhere. The next lesson builds the measure of the decision to retire a model and counts what removing the model leaves behind retrospectively: how many periods of decisions become **unexplainable** once it is removed.