--- title: 'Inference Scaling' source: 'https://academia.sh/en/courses/mlops-fundamentals/inference-scaling' course: 'MLOps Fundamentals' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Sweeps the batch size between one and two hundred and counts the fall in rows read per prediction and the growth in the latency of the waiting request side by side.' - 'Shows that the wait of the unfilled last group does not appear in the worst wait column.' - 'Computes the crossover point of two accelerator classes and shows that it depends on the realized group size.' - 'Shows with numbers that the gains of the cache and of request batching cannot be measured separately and multiplied.' --- # Inference Scaling Counting how request batching, caching and the choice of accelerator class are traded off between the cost per prediction and the latency of the waiting request: as the batch size goes from one to two, four, eight, sixteen, thirty-two, sixty-four, one hundred twenty-eight and two hundred over the same 600 requests, the cost per prediction falls from 463.87 to 22.00 while the worst wait rises from 0 to 199 arrivals and the average wait to 99.50; the answers are identical in all nine rows and the accuracy stays at 0.8067, so the batch size is not a quality setting. The last group of each period stays at eight requests at sizes sixteen, thirty-two and sixty-four and waits for a group that will never fill; this wait does not show in the worst wait column at all. The crossover point of the two accelerator classes is the ratio of the setup difference to the unit difference and is 80 requests, but in the table the switch shows at one hundred twenty-eight because what is measured is not the requested size but the realized group size. On a stream of 600 requests the cache, with a hit rate of 0.6783, lowers the rows read per request from 464.86 to 148.70 and together with batching gives 26.38; the two gains cannot be multiplied. The previous lesson put three serving patterns side by side and showed that the cost per prediction is a denominator decision. The online pattern was the most expensive there: 463.87 rows per request on average. There was an assumption inside that number: every request was processed **on its own**. Yet two requests that belong to the same region do not have to read the region history twice; if the two are processed together, that history is read once. This lesson turns that togetherness into a setting. Arriving requests are accumulated until a group fills, then processed in a single call. The gain is measurable and large; the price is also measurable and sits in the same place: the first request waiting for the group receives no answer until the group fills. The lesson sweeps these two numbers, then repeats the same sweep on two **accelerator** classes and under a cache. - **TD48.** The source, the pipeline, the model and the two accelerator classes are **fictional** and are generated inside the lesson. No inference service, serving framework, accelerator hardware or cloud service name is written; the accelerators are named only by their **kinds**. The seed is 20260218. - **TD49.** The label is imbalanced: the distribution is `{0: 1752, 1: 448}` and the **base class ratio is 0.7964**. The test set is 600 rows and the smallest step of accuracy is 0.0017. - **TD50.** The batch call and cache procedure was measured in the Building Applications with Model APIs course of the AI Engineering curriculum — the upper limit set by the context window, the damage that a failure spreads and the staleness of the cache lifetime were counted there. It is **not repeated**; here only the **cost per prediction** and the **wait** are counted. - **TD51.** Latency is counted in **arrivals**: the number of arrivals a request waits until its group fills. No raw milliseconds are written. - **TD52.** The history of every subscriber and every region in a group is read **once**; the whole gain of request batching comes from this sharing. - **TD53.** The two accelerator classes are written in **work units**: the general-purpose processing unit is `(60, 4)` and the accelerator is `(300, 1)`; the first number is the setup per call and the second is the unit per request. This is a **choice** and its sensitivity is swept below. - **TD54.** The serving horizon is that of the previous lesson: `2026-10`, `2026-11`, `2026-12` and two hundred requests in each period. The groups form within a period. ## Batch Size Sweep The first block sets up the source, the pipeline, the feature path and the model; then it runs the same 600 requests at nine different batch sizes. The features are independent of the weights and are computed **once**. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages of the # pipeline, the feature path, the threshold model with three weights and the two accelerator # classes are generated inside the lesson; no inference service or hardware is called. # Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if prev is not None: out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40 def label(k): return 1 if k["m3"] > THRESHOLD else 0 def features(records): """The training path sees the full history. Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def mean(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), 4) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] ALL = s3_consumption(generate_source()) FEATS = features(ALL) TRAIN_ROWS = [k for k in ALL if k["period"] in TRAIN] W = max(GRID, key=lambda w: sum( 1 for k in TRAIN_ROWS if (1 if score(FEATS[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k))) SERVING, REQUESTS = TEST, sorted(SUBSCRIBERS) by_sub, by_reg = {}, {} for k in ALL: by_sub.setdefault(k["subscriber"], []).append(k) by_reg.setdefault(k["region"], []).append(k) TRUTH = {(k["subscriber"], k["period"]): label(k) for k in ALL if k["period"] in SERVING} N = len(REQUESTS) * len(SERVING) def process_group(requests, period): """The history of every subscriber and every region in a group is read ONCE.""" i = PERIODS.index(period) subs = sorted(set(requests)) regs = sorted(set(SUBSCRIBERS[a]["region"] for a in subs)) read = sum(len(by_sub[a]) for a in subs) + sum(len(by_reg[b]) for b in regs) reg_mean = {} for b in regs: s = [k["m3"] for k in by_reg[b] if PERIODS.index(k["period"]) < i] reg_mean[b] = round(sum(s) / len(s), 4) if s else 0.0 t = {} for a in subs: s = [k["m3"] for k in by_sub[a] if PERIODS.index(k["period"]) < i] t[(a, period)] = 1 if score({"subscriber_mean": round(sum(s) / len(s), 4) if s else 0.0, "region_mean": reg_mean[SUBSCRIBERS[a]["region"]], "commercial": 1 if SUBSCRIBERS[a]["type"] == "commercial" else 0}, W) > THRESHOLD else 0 return t, read GENERAL, ACCEL = (60, 4), (300, 1) SIZES = (1, 2, 4, 8, 16, 32, 64, 128, 200) def sweep(size): read, wait, calls, all_pred, work_g, work_a, leftover = 0, 0, 0, {}, 0, 0, 0 for d in SERVING: groups = [REQUESTS[i:i + size] for i in range(0, len(REQUESTS), size)] for g in groups: t, o = process_group(g, d) all_pred.update(t) read += o wait += len(g) * (len(g) - 1) // 2 calls += 1 work_g += GENERAL[0] + GENERAL[1] * len(g) work_a += ACCEL[0] + ACCEL[1] * len(g) leftover = len(groups[-1]) if len(groups[-1]) != size else 0 return all_pred, read, wait, calls, leftover, work_g, work_a SINGLE = sweep(1)[0] WORK = {} print(" size calls unfilled rows read rows/prediction worst wait avg wait" " same answer accuracy") for b in SIZES: t, rd, wt, cl, lo, wg, wa = sweep(b) WORK[b] = (wg, wa) acc = round(sum(1 for c in TRUTH if t[c] == TRUTH[c]) / N, 4) print(f"{b:>5} {cl:>5} {lo:>8} {rd:>12} {rd / N:>15.2f} {b - 1:>15}" f" {wt / N:>11.2f} {str(all(t[c] == SINGLE[c] for c in SINGLE)):>11} {acc:>8}") ``` ``` size calls unfilled rows read rows/prediction worst wait avg wait same answer accuracy 1 600 0 278322 463.87 0 0.00 True 0.8067 2 300 0 252450 420.75 1 0.50 True 0.8067 4 150 0 203610 339.35 3 1.50 True 0.8067 8 75 0 140316 233.86 7 3.50 True 0.8067 16 39 8 87318 145.53 15 7.34 True 0.8067 32 21 8 51183 85.31 31 15.02 True 0.8067 64 12 8 31383 52.30 63 30.38 True 0.8067 128 6 72 19800 33.00 127 53.42 True 0.8067 200 3 0 13200 22.00 199 99.50 True 0.8067 ``` Two columns go in opposite directions and never turn back. The rows read per prediction fall from 463.87 to **22.00**, a factor of twenty-one; the worst wait rises from 0 to **199 arrivals** and the average wait to 99.50. The last two columns close the **quality** side of the decision: at all nine sizes the answers are identical and the accuracy stays at 0.8067 — the base class ratio is 0.7964 and the base ratio of the test set is 0.8067, so this setup stands **0.0000** above the base. **The batch size is not a quality setting; it is a setting between cost and wait.** The gain is not linear. The first doubling lowers 463.87 only to 420.75, because the regions of two requests are mostly different among five regions and the two region histories are still read separately. The gain reaches its highest speed between sizes 16 and 32 and then slows down: at size 200 all the regions are already in a single group, so there is no new history left to share. **What request batching gains is not the number of requests but the rows that the requests read in common.** The third column carries the table's own flaw. At sizes 16, 32 and 64 the last group of each period stays at **eight** requests, and at size 128 at seventy-two. These requests wait for a group that will never fill and are released only by a timeout. Their wait does **not show** in the "worst wait" column, because that column assumes the group fills. The measured worst value stands below an unmeasured assumption. ## Setup Cost and Cache Rows read is not the only name of the cost. Every call pays a setup cost, and this cost changes with the unit the call runs on. The second block compares the two accelerator classes in the same sweep, computes the crossover point and adds a stream of repeated requests at the end. ```python # This block continues the definitions of ALL , REQUESTS , SERVING , N , SIZES , WORK , # GENERAL , ACCEL , SUBSCRIBERS , PERIODS , SEED , rng and process_group from the first block. print(" size actual group general-purpose accelerator cheaper") for b in SIZES: wg, wa = WORK[b] groups = [len(REQUESTS[i:i + b]) for i in range(0, len(REQUESTS), b)] print(f"{b:>5} {len(REQUESTS) / len(groups):>16.1f} {wg / N:>15.2f} {wa / N:>12.2f}" f" {'accelerator' if wa < wg else 'general-purpose':>15}") for setup in (150, 300, 600): print(f"setup {setup:>3} -> crossover point {(setup - GENERAL[0]) / (GENERAL[1] - ACCEL[1]):.1f} requests") def request_stream(n): r = rng(SEED + 911) return [f"A{int(r() * 200) + 1:03d}" for _ in range(n)] def cost(calls, size, period="2026-12"): return sum(process_group(calls[i:i + size], period)[1] for i in range(0, len(calls), size)) STREAM = request_stream(600) DISTINCT = list(dict.fromkeys(STREAM)) print(f"\nstream {len(STREAM)} requests , distinct {len(DISTINCT)} , cache hit rate" f" {(len(STREAM) - len(DISTINCT)) / len(STREAM):.4f}") base = cost(STREAM, 1) for name, calls, b in (("neither , size 1", STREAM, 1), ("batching only", STREAM, 32), ("cache only", DISTINCT, 1), ("both together", DISTINCT, 32)): rd = cost(calls, b) print(f"{name:<24} read {rd:>7} per request {rd / len(STREAM):>7.2f}" f" factor {rd / base:.4f}") t1, t2 = cost(STREAM, 32) / base, cost(DISTINCT, 1) / base print(f"predicted by the product {t1 * t2:.4f} , actual" f" {cost(DISTINCT, 32) / base:.4f}") ``` ``` size actual group general-purpose accelerator cheaper 1 1.0 64.00 301.00 general-purpose 2 2.0 34.00 151.00 general-purpose 4 4.0 19.00 76.00 general-purpose 8 8.0 11.50 38.50 general-purpose 16 15.4 7.90 20.50 general-purpose 32 28.6 6.10 11.50 general-purpose 64 50.0 5.20 7.00 general-purpose 128 100.0 4.60 4.00 accelerator 200 200.0 4.30 2.50 accelerator setup 150 -> crossover point 30.0 requests setup 300 -> crossover point 80.0 requests setup 600 -> crossover point 180.0 requests stream 600 requests , distinct 193 , cache hit rate 0.6783 neither , size 1 read 278916 per request 464.86 factor 1.0000 batching only read 47872 per request 79.79 factor 0.1716 cache only read 89221 per request 148.70 factor 0.3199 both together read 15829 per request 26.38 factor 0.0568 predicted by the product 0.0549 , actual 0.0568 ``` The accelerator class is more expensive up to size 128. The crossover point is not a measurement but a ratio: the setup difference divided by the unit difference, that is `(300 - 60) / (4 - 1) = 80` requests. The reason the switch shows at 128 in the table is the second column. When size 64 is requested, every period splits into groups of 64, 64, 64 and 8 requests, and the **realized average group is 50**, which is below 80; at size 128 the realized average is 100, which is above 80. **The case for an accelerator lies not in its own speed but in the group size that can be filled**; with a group that cannot be filled, the accelerator is only an expensive general-purpose unit. The setup cost sweep shows this with three numbers: the crossover points of setups 150, 300 and 600 are 30, 80 and 180 requests. The last four rows add the cache. In the stream of six hundred requests there are only **193** distinct requests; the cache hit rate is 0.6783 and the rows read per request fall from 464.86 to 148.70. Request batching alone is stronger: 79.79. The two together give **26.38**. But the two factors cannot be measured separately and multiplied: the product of 0.1716 and 0.3199 predicts 0.0549 and the actual is 0.0568. The difference is small and its cause is measurable — the cache **thins** the groups. When the remaining 193 requests are split into groups of thirty-two, a single request is left in the last group, and that single request reads the whole region history on its own. **The gain of one mechanism depends on which mechanism stands next to it.** ## Summary - When the batch size is swept between one and two hundred, the rows read per prediction fall **from 463.87 to 22.00**, the worst wait rises **from 0 to 199 arrivals** and the average wait to 99.50; the two columns go in opposite directions and a size cannot be chosen without writing which of them is paid. - At all nine sizes the answers are identical and the accuracy stays at 0.8067; the base class ratio is 0.7964, the base ratio of the test set is 0.8067 and this setup stands 0.0000 above it: the batch size is not a quality setting. - The gain comes not from the number of requests but from the **rows the requests read in common**; the first doubling lowers 463.87 only to 420.75, because the regions of two requests are mostly different. - At sizes 16, 32 and 64 the last group of each period stays at eight requests and waits for a group that will never fill; this wait does not show in the "worst wait" column at all, because that column assumes the group fills. - The crossover point of the accelerator class is `(300 - 60) / (4 - 1) = 80` requests, but in the table the switch shows at 128: what is measured is not the requested size but the **realized group size** (50 when 64 is requested, 100 when 128 is requested). - On a stream of 600 requests the cache lowers 464.86 to 148.70 with a hit rate of 0.6783; together with request batching it gives 26.38, while the product of the two factors predicts 25.52 — the cache thins the groups. ## Next Step In five lessons the model was packaged, placed in a pattern and scaled. In all five there was a silent assumption: the model sits at an endpoint that is **reachable over the network** from the place where the request arrives. When the network drops, the cost per prediction of this setup is undefined, because there is no prediction. The next lesson moves the model to the source of the request, to the device itself; it measures what shrinking takes from the accuracy and in which task it takes it first, and counts the independence gained by running at the edge as answered requests.