--- title: 'Serving Patterns' source: 'https://academia.sh/en/courses/mlops-fundamentals/serving-patterns' course: 'MLOps Fundamentals' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Runs batch, online and streaming inference on the same workload and shows that the three give the same answer.' - 'Counts the latency class, the age of the answer and the rows read per prediction of each pattern separately.' - 'Shows that the cost per prediction depends on the number of predictions used and computes the point where two patterns cross.' - 'Justifies that a cost being unmeasurable does not mean the cost is absent by the distance of the decision score from the threshold.' --- # Serving Patterns Running the same package in three placements, batch, online and streaming, and counting the latency class, the freshness and the cost per prediction of the three separately: all three patterns give the same answer to the same 600 requests and the accuracy is 0.8067 in all three, so an observation that looks only at the output cannot tell which pattern ran. The difference is in three columns: the work done at request time is a lookup in the batch and streaming patterns and a scan of 352 to 550 rows in the online pattern; the age of the answer is one period in the batch pattern and zero in the others; the rows read per prediction are 11.00, 463.87 and 3.67. The staleness sweep gives 0 changed predictions up to six periods, and the reason is not the pattern but that the decision score stays at least 7.6468 away from the threshold. The cost is a denominator decision: the batch and online patterns are equal at 4.74 predictions used, and which task rests on which pattern is read from that point. The previous lesson packaged the model: the runtime was pinned, the interface contract was written, and the number of calls that dropped when no contract was written was counted. What the package accepts and what it returns is now written down. The only thing not written is **when** this package runs. The same package can run once at the start of a period and write two hundred predictions to a table, can run once whenever a request arrives, or can refresh itself whenever a new reading lands. The weights are the same in all three. These three placements are called a **serving pattern**. The difference between them is not in the model but in the moment and the scope at which **inference** is done. This lesson runs the three side by side on the same workload and, instead of a single accuracy number, writes three numbers separately: the work done at the moment the request arrives, the age of the newest reading the answer rests on, and the **cost per prediction**. - **TD41.** The source, the pipeline and the model are **fictional** and are generated inside the lesson. No serving framework, inference service, accelerator or cloud service is called; all three patterns are written here. The seed is 20260218. - **TD42.** The label is imbalanced: the distribution is `{0: 1752, 1: 448}` and the **base class ratio is 0.7964**. No accuracy is read without writing how far above this ratio it stands. The test set is 600 rows, so the smallest step of accuracy is 0.0017. - **TD43.** In this course the training pipeline is handled on a **single machine**; partitioning and shuffling were measured in the Distributed Data Processing course and are not repeated. - **TD44.** The cost is counted as **rows read**; no raw milliseconds are written. The latency is written as a **class**: `lookup` is the reading of a single stored record at request time; `scan` is the reading of the history of the subscriber and of its region at request time. - **TD45.** Freshness is counted in **periods**: the age of the answer is the number of periods that have to pass before the newest reading is reflected in the prediction. - **TD46.** The serving horizon is the test periods, `2026-10`, `2026-11`, `2026-12`, and in each period one request arrives for each of the two hundred subscribers. 600 requests in total. - **TD47.** The batch pattern runs once per period. This is a **choice** and is swept below. ## Three Patterns Side by Side The first block sets up the source, the three stages of the pipeline up to consumption, the feature path and the fictional model; then it runs the three patterns with the same weights. The features are computed **once** and are independent of the weights. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages of the # pipeline, the feature path and the threshold model with three weights are generated inside # the lesson; no learning library, serving framework or inference service is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is not None: out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records, window=None, digits=4): """The training path sees the full history. Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))] return round(sum(chosen) / len(chosen), digits) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def train(records, feats): subset = [k for k in records if k["period"] in TRAIN] best, best_w = -1.0, None for w in GRID: d = sum(1 for k in subset if (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k)) if d / len(subset) > best: best, best_w = d / len(subset), w return {"weights": best_w, "train_rows": len(subset), "candidates": len(GRID)} ALL = s3_consumption(generate_source()) FEATS = features(ALL) MODEL = train(ALL, FEATS) W = MODEL["weights"] SERVING, REQUESTS = TEST, sorted(SUBSCRIBERS) def batch_run(records, period, w): """The whole history is scanned once , all predictions are produced and stored.""" feats = features(records) return ({k["subscriber"]: (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) for k in records if k["period"] == period}, len(records)) def online_request(idx, sub, period, w): """At request time the history of the subscriber and of its region is scanned.""" by_sub, by_reg = idx i, reg = PERIODS.index(period), by_sub[sub][0]["region"] sub_vals = [k["m3"] for k in by_sub[sub] if PERIODS.index(k["period"]) < i] reg_vals = [k["m3"] for k in by_reg[reg] if PERIODS.index(k["period"]) < i] o = {"subscriber_mean": round(sum(sub_vals) / len(sub_vals), 4) if sub_vals else 0.0, "region_mean": round(sum(reg_vals) / len(reg_vals), 4) if reg_vals else 0.0, "commercial": 1 if by_sub[sub][0]["type"] == "commercial" else 0} return (1 if score(o, w) > THRESHOLD else 0), len(by_sub[sub]) + len(by_reg[reg]) def stream_run(records, w): """As each reading arrives the state is updated incrementally and the prediction is refreshed.""" sub_state, reg_state, pred, read, refreshes = {}, {}, {}, 0, 0 for k in sorted(records, key=lambda k: (PERIODS.index(k["period"]), k["subscriber"])): read += 1 sub, reg = k["subscriber"], k["region"] o = {"subscriber_mean": round(sub_state[sub][0] / sub_state[sub][1], 4) if sub in sub_state else 0.0, "region_mean": round(reg_state[reg][0] / reg_state[reg][1], 4) if reg in reg_state else 0.0, "commercial": 1 if k["type"] == "commercial" else 0} pred[(sub, k["period"])] = 1 if score(o, w) > THRESHOLD else 0 sub_state.setdefault(sub, [0, 0]); reg_state.setdefault(reg, [0, 0]) sub_state[sub][0] += k["m3"]; sub_state[sub][1] += 1 reg_state[reg][0] += k["m3"]; reg_state[reg][1] += 1 refreshes += 1 return pred, read, refreshes, len(sub_state) + len(reg_state) by_sub, by_reg = {}, {} for k in ALL: by_sub.setdefault(k["subscriber"], []).append(k) by_reg.setdefault(k["region"], []).append(k) IDX = (by_sub, by_reg) TRUTH = {(k["subscriber"], k["period"]): label(k) for k in ALL if k["period"] in SERVING} N = len(REQUESTS) * len(SERVING) def accuracy(t): return round(sum(1 for c in TRUTH if t[c] == TRUTH[c]) / N, 4) BATCH, read_b = {}, 0 for d in SERVING: t, o = batch_run(ALL, d, W) read_b += o for a in REQUESTS: BATCH[(a, d)] = t[a] ONLINE, read_o, min_read, max_read = {}, 0, 10 ** 9, 0 for d in SERVING: for a in REQUESTS: p, o = online_request(IDX, a, d, W) ONLINE[(a, d)], read_o = p, read_o + o min_read, max_read = min(min_read, o), max(max_read, o) STREAM, read_s, refreshes, state = stream_run(ALL, W) print(f"training rows {MODEL['train_rows']} , weights {W} , candidates {MODEL['candidates']}") print(f"serving horizon {len(SERVING)} periods x {len(REQUESTS)} subscribers = {N} requests") print("pattern request time answer age rows read per prediction accuracy") for name, age, rd, acc in (("batch", 1, read_b, accuracy(BATCH)), ("online", 0, read_o, accuracy(ONLINE)), ("stream", 0, read_s, accuracy(STREAM))): print(f"{name:<9} {'scan' if name == 'online' else 'lookup':>12} {age:>10}" f" {rd:>12} {rd / N:>14.2f} {acc:>8}") print(f"rows read per online request: min {min_read} , max {max_read}") print(f"stream: refreshes {refreshes} , used {N} , state entries held {state}") ``` ``` training rows 1600 , weights (0.6, 0.4, 0) , candidates 27 serving horizon 3 periods x 200 subscribers = 600 requests pattern request time answer age rows read per prediction accuracy batch lookup 1 6600 11.00 0.8067 online scan 0 278322 463.87 0.8067 stream lookup 0 2200 3.67 0.8067 rows read per online request: min 352 , max 550 stream: refreshes 2200 , used 600 , state entries held 205 ``` The last column is **0.8067** in all three rows. The same weights, the same feature path, the same 600 requests; all three patterns give the same answers. The label is imbalanced and the base class ratio is 0.7964; the base ratio of the test set itself is also 0.8067, so this setup stands **0.0000** above the base, just as it was written in the first lesson of the course. The choice of pattern does not change that either, and an observation that looks only at the output cannot tell which pattern ran. The difference is in the other columns. The online pattern reads 352 to 550 rows per request, 463.87 on average. Almost all of this is the **history of the region**; the history of the subscriber itself is eleven rows. The `region_mean` feature cannot be computed for a single subscriber, it asks for all the subscribers of the region. **Online inference is expensive not because of the model but because of the scope of the feature**, and this is why each prediction reads forty-two times as many rows. **Batch inference** computes the same region mean once and shares it among all the subscribers of the region: 6600 rows, 600 predictions, 11.00 per prediction. In return the answer ages by one period, because no reading that arrives between two runs can enter the prediction. The streaming pattern reads each row once and updates the state incrementally: 2200 rows, 3.67 per prediction. The table does **not show** its cost; of the 2200 refreshes only 600 are used, 205 state entries are held continuously, and rebuilding the state when it is lost means reading the 2200 rows from the start. The rows read column is the column where streaming looks cheapest, and the cost of streaming is not written in exactly that column. ## What the Cost Is Divided By The answer aging by a period looks like a cost. The second block measures it first: it answers the same requests from an ever older run. Then it changes the denominator of the cost per prediction. ```python # This block continues the definitions of ALL , FEATS , W , SERVING , REQUESTS , # ONLINE , read_o , N , score , THRESHOLD , PERIODS and accuracy from the first block. print("answer age (periods) changed predictions accuracy") for g in range(0, 7): t = {} for d in SERVING: j = max(PERIODS.index(d) - g, 1) for a in REQUESTS: t[(a, d)] = 1 if score(FEATS[(a, PERIODS[j])], W) > THRESHOLD else 0 chg = sum(1 for c in t if t[c] != ONLINE[c]) print(f"{g:>20} {chg:>19} {accuracy(t):>8}") dist = sorted(abs(score(FEATS[c], W) - THRESHOLD) for c in ONLINE) print(f"threshold {THRESHOLD} , distance of the decision score from the threshold: min {dist[0]:.4f}" f" , median {dist[len(dist) // 2]:.4f}") print("\n predictions used batch online stream cheapest batch unused") mean_o = read_o / N for u in (1, 2, 5, 20, 50, 200): s = [len(ALL) / u, mean_o, len(REQUESTS) / u] print(f"{u:>17} {s[0]:>7.2f} {s[1]:>9.2f} {s[2]:>6.2f}" f" {('batch', 'online', 'stream')[s.index(min(s))]:>8} {len(REQUESTS) - u:>19}") print(f"batch and online cross at {len(ALL) / mean_o:.2f} predictions used") ``` ``` answer age (periods) changed predictions accuracy 0 0 0.8067 1 0 0.8067 2 0 0.8067 3 0 0.8067 4 0 0.8067 5 0 0.8067 6 0 0.8067 threshold 40 , distance of the decision score from the threshold: min 7.6468 , median 13.6826 predictions used batch online stream cheapest batch unused 1 2200.00 463.87 200.00 stream 199 2 1100.00 463.87 100.00 stream 198 5 440.00 463.87 40.00 stream 195 20 110.00 463.87 10.00 stream 180 50 44.00 463.87 4.00 stream 150 200 11.00 463.87 1.00 stream 0 batch and online cross at 4.74 predictions used ``` The result of the staleness sweep is a single number: while the age of the answer goes from zero to six periods, **the changed predictions are 0** and the accuracy stays at 0.8067. On this workload staleness has no measurable counterpart, and "the batch pattern is free" is **not concluded** from it. The reason is one line below: the smallest distance of the decision score to the threshold is 7.6468 and its median is 13.6826. The decisions stay away from the threshold, and one more period of history does not move the eleven-period cumulative mean enough to close that distance. When the decisions approach the threshold, the same staleness moves the prediction. **A cost that cannot be measured is not ignored; it is written that it could not be measured and under which condition it would show** — this is the second claim of the course read here from the side of the cost. The second table shows that the cost per prediction is a **denominator decision**. The batch pattern reads 2200 rows in one run and produces two hundred predictions; how many of those predictions are actually read is a question of the task, not of the pattern. If a single prediction is used, the cost per prediction of the batch pattern is **2200.00**, which is more than four times the 463.87 of the online pattern; the two patterns are equal at **4.74 predictions used**. The last column is the name of the cost of the batch pattern: when one prediction is used, 199 predictions are produced and never read. Which task rests on which pattern is read from these two tables. The job that lists all the subscribers at the start of the period rests on the batch pattern: the denominator is 200, the staleness cannot be measured, and the cost drops to 11.00. A single-subscriber query that is not known in advance rests on the online pattern: the denominator is 1, and the batch pattern is more than four times as expensive there, and moreover cannot answer a key that is not in the list at all. The job that has to be triggered **at the moment the reading arrives** is done only with streaming: streaming produces 2200 triggers, the online pattern produces 600 only when a request arrives and triggers nothing if no request arrives, while the batch pattern produces the 600 at the start of the period and one period late. ## Summary - The three serving patterns give the **same answers** to the same 600 requests and the accuracy is **0.8067** in all three; the base class ratio is 0.7964, the base ratio of the test set is 0.8067 and this setup stands **0.0000** above it. The choice of pattern is not an accuracy decision, and an observation that looks only at the output cannot tell which pattern ran. - The difference is in three columns: the latency class at request time is `lookup` in the batch and streaming patterns and a `scan` of 352 to 550 rows in the online pattern; the age of the answer is 1, 0 and 0 periods; the rows read per prediction are **11.00**, **463.87** and **3.67**. - The forty-two-fold difference of the online pattern comes not from the model but from the scope of the `region_mean` feature; the history of the subscriber itself is eleven rows, that of its region is between 341 and 539 rows. - When the age of the answer is raised to six periods, the changed predictions are **0**; the reason is not the pattern but that the decision score stays at least **7.6468** away from the threshold. A cost that cannot be measured is not a measured guarantee. - The cost per prediction depends on the denominator: the batch pattern gives **2200.00** when one prediction is used and **11.00** when two hundred are used, and it becomes equal to the online pattern at **4.74** predictions used. - The streaming pattern is the cheapest in rows read, but 1600 of its 2200 refreshes are not used and 205 state entries are held continuously; this cost does not show up in the rows read column at all. ## Next Step The pattern is chosen, but within each pattern one question is still unanswered: how many requests a call should carry **together**. In the batch pattern this is how many predictions a run produces; in the online pattern it means accumulating the arriving requests and processing them at once, and the cost of accumulating is that the first request that arrived **waits**. The next lesson runs the same requests at batch sizes between one and sixty-four, counts the fall of the cost per prediction side by side with the latency of the waiting request, and repeats the same sweep on two accelerator classes.