Cost Management

The previous lesson took the debt owed to the outside and wrote it as records: four obligation classes, seven fields per decision, 8402 stored values. Everything counted was written to storage. The running of the model itself, the work repeated every period, entered no line.

That work is not one piece. A production model does two separate jobs: it is retrained period by period and it produces inference every period. The two are usually written to one budget item and read as a single total. This lesson splits that total in two, sweeps the retraining frequency and counts which decision the single number makes invisible.

• RG47. The source, the pipeline, the feature path, the model and the cost counter are fictional. No cloud service, accelerator, product or price is called; no currency is written. Seed 20260218.
• RG48. The unit of work is one score evaluation: multiplying a row’s features by the weights and summing. Training and inference do only this, so both enter the same table with the same unit. The counter is inside the score function; cost is not assumed, it is measured.
• RG49. No time is written. Cost is counted in score evaluations, delay in periods.
• RG50. The baseline is the setup without retraining: the first model is trained once and never changes over the six monitoring periods.
• RG51. In production the input drifts: after 2026-09 commercial consumption grows by 0.06 per period. The model is (0.6, 0.0, 4), the decision threshold 18, the candidates 135, the training rows 1600. Monitoring is six periods, the base class ratio 0.5342, the step of period accuracy 0.0050.
• RG52. The inference volume is 200 requests per period and is swept as a free variable in the last section.

The cost per prediction was measured in the Building Applications with Model APIs course of the AI Engineering curriculum and in the scaling lesson of the MLOps Fundamentals course; their procedures are not repeated, and the measure here adds a single thing on top of them: training and inference are counted separately.

Counting the Two Items Separately

The first block sets up the source, the feature path and the model, places a counter in the score function and sweeps the retraining frequency.

# FICTIONAL. The source , the consumption stage of the pipeline , the feature path , the
# threshold model and the cost counter are generated inside the lesson; no cloud service ,
# accelerator , product or price is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out = []
    for k in records:
        i = PERIODS.index(k["period"])
        previous = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if previous is not None:
            out.append(dict(k, m3=int(k["index"]) - int(previous["index"])))
    return out


def features(records):
    """Independent of the weights , computed once."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def average(series, i):
        chosen = [v for d, v in series if d < i]
        return round(sum(chosen) / len(chosen), 4) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": average(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": average(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)
MONITORED = PERIODS[6:]
RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6
COUNTER = {"score": 0}          # the single unit of work: one score evaluation


def score(o, w):
    COUNTER["score"] += 1
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def predict(o, w, decision):
    return 1 if score(o, w) > decision else 0


def label(o, threshold=RULE_THRESHOLD):
    return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0


def train(records, feats, periods):
    """The training criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched."""
    subset = [k for k in records if k["period"] in periods]
    best = (-1.0, None, None)
    for w in GRID:
        for decision in DECISION_GRID:
            tp = tn = fp = fn = 0
            for k in subset:
                o = feats[(k["subscriber"], k["period"])]
                p, g = predict(o, w, decision), label(o)
                tp, fp, fn, tn = (tp + (p and g), fp + (p and not g),
                                  fn + (not p and g), tn + (not p and not g))
            total = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0)
            if total / 2 > best[0]:
                best = (total / 2, w, decision)
    return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4),
            "rows": len(subset)}


def drift(records, start="2026-09", slope=0.06):
    """FICTIONAL data drift: commercial consumption grows by `slope` every period."""
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


def accuracy(records, m, feats, period):
    subset = [k for k in records if k["period"] == period]
    d = sum(1 for k in subset
            if predict(feats[(k["subscriber"], k["period"])], m["weights"], m["decision_threshold"])
            == label(feats[(k["subscriber"], k["period"])]))
    return round(d / len(subset), 4)


CLEAN = consumption(generate_source())
FEATS_C = features(CLEAN)
monitored_c = [k for k in CLEAN if k["period"] in MONITORED]
p1 = sum(label(FEATS_C[(k["subscriber"], k["period"])]) for k in monitored_c) / len(monitored_c)
DRIFTED = drift(CLEAN)                    # the input drifts in production
FEATS = features(DRIFTED)

COUNTER["score"] = 0
FIRST = train(DRIFTED, FEATS, PERIODS[1:9])
FIRST_COST = COUNTER["score"]
print(f"base class ratio {round(max(p1, 1 - p1), 4)} , monitoring periods {len(MONITORED)} ,"
      f" requests per period 200")
print(f"first training: weights {FIRST['weights']} , decision threshold {FIRST['decision_threshold']} ,"
      f" rows {FIRST['rows']} , candidates {len(GRID) * len(DECISION_GRID)}")
print(f"measured training cost (score evaluations): {FIRST_COST}")

COUNTER["score"] = 0
last = accuracy(DRIFTED, FIRST, FEATS, "2026-12")
print(f"inference cost of one period (score evaluations): {COUNTER['score']} ,"
      f" 2026-12 accuracy {last}")

WINDOW = 6
print("\nfrequency  retrainings  training cost  inference cost   total  training share"
      "  2026-12 accuracy")
RESULT = []
for f in (0, 6, 3, 2, 1):
    COUNTER["score"] = 0
    m = FIRST
    if f:
        for i, d in enumerate(MONITORED):
            if (i + 1) % f == 0:
                j = PERIODS.index(d)
                m = train(DRIFTED, FEATS, [x for x in PERIODS
                                           if j - WINDOW < PERIODS.index(x) <= j])
    training = COUNTER["score"]
    COUNTER["score"] = 0
    acc = accuracy(DRIFTED, m, FEATS, "2026-12")
    inference = len(MONITORED) * 200
    total = training + inference
    RESULT.append((f, training, inference, total, acc))
    print(f"{f if f else '-':>9}  {len(MONITORED) // f if f else 0:>11}  {training:>13}"
          f"  {inference:>14}  {total:>6}  {round(training / total, 4):>14}  {acc:>16}")

base class ratio 0.5342 , monitoring periods 6 , requests per period 200
first training: weights (0.6, 0.0, 4) , decision threshold 18 , rows 1600 , candidates 135
measured training cost (score evaluations): 216000
inference cost of one period (score evaluations): 200 , 2026-12 accuracy 0.98

frequency  retrainings  training cost  inference cost   total  training share  2026-12 accuracy
        -            0              0            1200    1200             0.0              0.98
        6            1         162000            1200  163200          0.9926              0.98
        3            2         324000            1200  325200          0.9963              0.98
        2            3         486000            1200  487200          0.9975              0.98
        1            6         972000            1200  973200          0.9988              0.98

The scale difference between the two items comes directly from the counter. One period of inference is 200 score evaluations and comes to 1200 over six periods; the first training scans 1600 rows with 135 candidates and costs 216,000. A single retraining with a six-period window is 162,000, that is 135 times the inference of the whole monitoring horizon. Down the frequency column the inference item never changes; only the training item does, and it rises from 162,000 to 972,000 when going from once in six periods to every period.

The last column writes what this increase buys. The 2026-12 accuracy is 0.9800 in all five setups — exactly the same as that of the model that is never retrained. This is the cost-side counterpart of the second claim of the course: the input drifted, the alert fired, six retrainings were run and the accuracy did not move by a single step. Since the period step is 0.0050, the 0.0000 is truly zero, not a small gain too small to measure. As long as the rule does not change, a correct model stays correct on a drifted input too, and retraining brings only the cost.

What the Single Total Hides

The second block reads the same results per prediction and sweeps the inference volume.

# This block continues the RESULT , MONITORED , FIRST_COST and WINDOW definitions of the first block.
baseline_accuracy = RESULT[0][4]
n_predictions = len(MONITORED) * 200
print("frequency  per prediction from total  per prediction from inference"
      "  accuracy gained  cost per gain")
for f, training, inference, total, acc in RESULT:
    gain = round(acc - baseline_accuracy, 4)
    per_gain = "undefined" if gain <= 0 else str(round(training / gain))
    print(f"{f if f else '-':>9}  {round(total / n_predictions, 2):>25}"
          f"  {round(inference / n_predictions, 2):>29}  {gain:>15}  {per_gain:>13}")

print(f"\nshare removed from the total by making inference completely free:"
      f" {round(RESULT[-1][2] / RESULT[-1][3], 4)}")
print(f"share removed from the total by moving from frequency 1 to frequency 6:"
      f" {round((RESULT[-1][3] - RESULT[1][3]) / RESULT[-1][3], 4)}"
      f" , accuracy paid {round(RESULT[1][4] - RESULT[-1][4], 4)}")

print("\nrequests per period  inference (6 periods)  training frequency 6  training frequency 1"
      "  which inference exceeded")
for requests in (200, 2000, 20000, 50000, 200000):
    c = len(MONITORED) * requests
    passed = [str(f) for f, e, _, _, _ in RESULT[1:] if c > e]
    print(f"{requests:>19}  {c:>21}  {RESULT[1][1]:>20}  {RESULT[-1][1]:>20}"
          f"  {(' , '.join(passed) if passed else 'none'):>24}")
print(f"\ncrossover point (requests per period): frequency 6 -> {RESULT[1][1] // len(MONITORED)}"
      f" , frequency 1 -> {RESULT[-1][1] // len(MONITORED)}")

# Monitoring's own item: three indicators are computed every period.
indicators = len(MONITORED) * 200          # prediction share , 0 if the inference predictions are reused
print(f"monitoring item (score evaluations): at most {indicators} , when inference predictions"
      f" are reused 0 ; one retraining {RESULT[1][1]}")

frequency  per prediction from total  per prediction from inference  accuracy gained  cost per gain
        -                        1.0                            1.0              0.0      undefined
        6                      136.0                            1.0              0.0      undefined
        3                      271.0                            1.0              0.0      undefined
        2                      406.0                            1.0              0.0      undefined
        1                      811.0                            1.0              0.0      undefined

share removed from the total by making inference completely free: 0.0012
share removed from the total by moving from frequency 1 to frequency 6: 0.8323 , accuracy paid 0.0

requests per period  inference (6 periods)  training frequency 6  training frequency 1  which inference exceeded
                200                   1200                162000                972000                      none
               2000                  12000                162000                972000                      none
              20000                 120000                162000                972000                      none
              50000                 300000                162000                972000                         6
             200000                1200000                162000                972000             6 , 3 , 2 , 1

crossover point (requests per period): frequency 6 -> 27000 , frequency 1 -> 162000
monitoring item (score evaluations): at most 1200 , when inference predictions are reused 0 ; one retraining 162000

The first two columns answer the question of the lesson. When the total is read as one number and divided by the number of predictions, the cost per prediction is 811.00; the same computation from the inference item alone gives 1.00. The entire 810-fold difference is training and belongs to no prediction. A reader who sees only this number concludes that the prediction is expensive and turns to making inference cheaper: batching requests, a cache, shrinking the model. The ceiling of that road is 0.0012 — even if inference were completely free, only 1.2 per thousand of the total would be removed.

The decision that stays invisible lies the other way. Moving the frequency from every period to once in six periods removes 0.8323 of the total and costs 0.0000 accuracy. The single total hides this decision, because in the total nothing separates training from inference; the training share lies between 0.9926 and 0.9988 and the frequency at which one stands cannot be read from the total. When the two items are counted separately the decision is a single line. The cost per gain column says the same thing once more: undefined in all five setups, because the divisor is zero. The justification of a cost item can only be written together with a gain; spending without a shown gain counts as unmeasured.

The last sweep shows that the decision depends on volume. At 200 requests per period inference exceeds no training item; at 50,000 requests it exceeds only the sparsest frequency; at 200,000 requests it exceeds all four at once. The crossover point is 27,000 requests per period for frequency 6 and 162,000 for frequency 1. This means a conclusion such as “training always dominates” cannot be written: the dominant item depends on which side of the volume one stands, and the single total does not show that side either.

The item of monitoring itself is the smallest row of this table. Computing three indicators over six periods is at most 1200 score evaluations, and it falls to zero when the predictions produced in inference are reused. But the work monitoring triggers is not small: the retraining started by a single alert that fires is 162,000. The cost of monitoring is not in the indicator itself but in the response given to the indicator.

Summary

• The unit of work is a score evaluation and the counter is inside the score function; training and inference enter the same table with the same unit. One period of inference is 200, the first training 216,000, a retraining with a six-period window 162,000 score evaluations.
• In the frequency sweep the inference item stays constant at 1200, the training item rises from 162,000 to 972,000 and the training share rises from 0.9926 to 0.9988.
• The 2026-12 accuracy is 0.9800 in all five setups. Since the period step is 0.0050, the accuracy gained is truly 0.0000 and the cost per gain is undefined.
• The cost per prediction read from the total is 811.00, read from inference 1.00. Making inference free removes 0.0012 of the total, thinning out the frequency removes 0.8323 of it; the second costs 0.0000 accuracy and is invisible in the single total.
• The crossover point lies between 27,000 and 162,000 requests per period; which item dominates depends on volume. The monitoring item is at most 1200; the retraining it triggers is 162,000.

Next Step

The two items were separated and a counterpart was written for both: retraining brings 0.0000 accuracy, while inference produces the whole value of the model for 200 requests per period. The only column missing from this table is the column of the end. Retraining can be thinned out, inference can be made cheaper, records can be shortened; but when a model will end is written nowhere. The next lesson builds the measure of the decision to retire a model and counts what removing the model leaves behind retrospectively: how many periods of decisions become unexplainable once it is removed.