Lesson 11 / 12

Inference Scaling

Counting how request batching, caching and the choice of accelerator class are traded off between the cost per prediction and the latency of the waiting request: as the batch size goes from one to two, four, eight, sixteen, thirty-two, sixty-four, one hundred twenty-eight and two hundred over the same 600 requests, the cost per prediction falls from 463.87 to 22.00 while the worst wait rises from 0 to 199 arrivals and the average wait to 99.50; the answers are identical in all nine rows and the accuracy stays at 0.8067, so the batch size is not a quality setting. The last group of each period stays at eight requests at sizes sixteen, thirty-two and sixty-four and waits for a group that will never fill; this wait does not show in the worst wait column at all. The crossover point of the two accelerator classes is the ratio of the setup difference to the unit difference and is 80 requests, but in the table the switch shows at one hundred twenty-eight because what is measured is not the requested size but the realized group size. On a stream of 600 requests the cache, with a hit rate of 0.6783, lowers the rows read per request from 464.86 to 148.70 and together with batching gives 26.38; the two gains cannot be multiplied.

Published October 14, 2026 Updated October 21, 2026

In this lesson

Sweeps the batch size between one and two hundred and counts the fall in rows read per prediction and the growth in the latency of the waiting request side by side.
Shows that the wait of the unfilled last group does not appear in the worst wait column.
Computes the crossover point of two accelerator classes and shows that it depends on the realized group size.
Shows with numbers that the gains of the cache and of request batching cannot be measured separately and multiplied.

Contents

The previous lesson put three serving patterns side by side and showed that the cost per prediction is a denominator decision. The online pattern was the most expensive there: 463.87 rows per request on average. There was an assumption inside that number: every request was processed on its own. Yet two requests that belong to the same region do not have to read the region history twice; if the two are processed together, that history is read once.

This lesson turns that togetherness into a setting. Arriving requests are accumulated until a group fills, then processed in a single call. The gain is measurable and large; the price is also measurable and sits in the same place: the first request waiting for the group receives no answer until the group fills. The lesson sweeps these two numbers, then repeats the same sweep on two accelerator classes and under a cache.

TD48. The source, the pipeline, the model and the two accelerator classes are fictional and are generated inside the lesson. No inference service, serving framework, accelerator hardware or cloud service name is written; the accelerators are named only by their kinds. The seed is 20260218.
TD49. The label is imbalanced: the distribution is {0: 1752, 1: 448} and the base class ratio is 0.7964. The test set is 600 rows and the smallest step of accuracy is 0.0017.
TD50. The batch call and cache procedure was measured in the Building Applications with Model APIs course of the AI Engineering curriculum — the upper limit set by the context window, the damage that a failure spreads and the staleness of the cache lifetime were counted there. It is not repeated; here only the cost per prediction and the wait are counted.
TD51. Latency is counted in arrivals: the number of arrivals a request waits until its group fills. No raw milliseconds are written.
TD52. The history of every subscriber and every region in a group is read once; the whole gain of request batching comes from this sharing.
TD53. The two accelerator classes are written in work units: the general-purpose processing unit is (60, 4) and the accelerator is (300, 1); the first number is the setup per call and the second is the unit per request. This is a choice and its sensitivity is swept below.
TD54. The serving horizon is that of the previous lesson: 2026-10, 2026-11, 2026-12 and two hundred requests in each period. The groups form within a period.

Batch Size Sweep

The first block sets up the source, the pipeline, the feature path and the model; then it runs the same 600 requests at nine different batch sizes. The features are independent of the weights and are computed once.

# FICTIONAL. The meter readings of the measurement network, the first three stages of the
# pipeline, the feature path, the threshold model with three weights and the two accelerator
# classes are generated inside the lesson; no inference service or hardware is called.
# Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out = []
    for k in records:
        i = PERIODS.index(k["period"])
        prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if prev is not None:
            out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out


TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


def features(records):
    """The training path sees the full history. Independent of the weights , computed once."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if d < i]
        return round(sum(chosen) / len(chosen), 4) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


ALL = s3_consumption(generate_source())
FEATS = features(ALL)
TRAIN_ROWS = [k for k in ALL if k["period"] in TRAIN]
W = max(GRID, key=lambda w: sum(
    1 for k in TRAIN_ROWS if (1 if score(FEATS[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k)))

SERVING, REQUESTS = TEST, sorted(SUBSCRIBERS)
by_sub, by_reg = {}, {}
for k in ALL:
    by_sub.setdefault(k["subscriber"], []).append(k)
    by_reg.setdefault(k["region"], []).append(k)
TRUTH = {(k["subscriber"], k["period"]): label(k) for k in ALL if k["period"] in SERVING}
N = len(REQUESTS) * len(SERVING)


def process_group(requests, period):
    """The history of every subscriber and every region in a group is read ONCE."""
    i = PERIODS.index(period)
    subs = sorted(set(requests))
    regs = sorted(set(SUBSCRIBERS[a]["region"] for a in subs))
    read = sum(len(by_sub[a]) for a in subs) + sum(len(by_reg[b]) for b in regs)
    reg_mean = {}
    for b in regs:
        s = [k["m3"] for k in by_reg[b] if PERIODS.index(k["period"]) < i]
        reg_mean[b] = round(sum(s) / len(s), 4) if s else 0.0
    t = {}
    for a in subs:
        s = [k["m3"] for k in by_sub[a] if PERIODS.index(k["period"]) < i]
        t[(a, period)] = 1 if score({"subscriber_mean": round(sum(s) / len(s), 4) if s else 0.0,
                                     "region_mean": reg_mean[SUBSCRIBERS[a]["region"]],
                                     "commercial": 1 if SUBSCRIBERS[a]["type"] == "commercial" else 0},
                                    W) > THRESHOLD else 0
    return t, read


GENERAL, ACCEL = (60, 4), (300, 1)
SIZES = (1, 2, 4, 8, 16, 32, 64, 128, 200)


def sweep(size):
    read, wait, calls, all_pred, work_g, work_a, leftover = 0, 0, 0, {}, 0, 0, 0
    for d in SERVING:
        groups = [REQUESTS[i:i + size] for i in range(0, len(REQUESTS), size)]
        for g in groups:
            t, o = process_group(g, d)
            all_pred.update(t)
            read += o
            wait += len(g) * (len(g) - 1) // 2
            calls += 1
            work_g += GENERAL[0] + GENERAL[1] * len(g)
            work_a += ACCEL[0] + ACCEL[1] * len(g)
        leftover = len(groups[-1]) if len(groups[-1]) != size else 0
    return all_pred, read, wait, calls, leftover, work_g, work_a


SINGLE = sweep(1)[0]
WORK = {}
print(" size  calls  unfilled     rows read  rows/prediction       worst wait     avg wait"
      "  same answer  accuracy")
for b in SIZES:
    t, rd, wt, cl, lo, wg, wa = sweep(b)
    WORK[b] = (wg, wa)
    acc = round(sum(1 for c in TRUTH if t[c] == TRUTH[c]) / N, 4)
    print(f"{b:>5}  {cl:>5}  {lo:>8}  {rd:>12}  {rd / N:>15.2f}  {b - 1:>15}"
          f"  {wt / N:>11.2f}  {str(all(t[c] == SINGLE[c] for c in SINGLE)):>11}  {acc:>8}")

 size  calls  unfilled     rows read  rows/prediction       worst wait     avg wait  same answer  accuracy
    1    600         0        278322           463.87                0         0.00         True    0.8067
    2    300         0        252450           420.75                1         0.50         True    0.8067
    4    150         0        203610           339.35                3         1.50         True    0.8067
    8     75         0        140316           233.86                7         3.50         True    0.8067
   16     39         8         87318           145.53               15         7.34         True    0.8067
   32     21         8         51183            85.31               31        15.02         True    0.8067
   64     12         8         31383            52.30               63        30.38         True    0.8067
  128      6        72         19800            33.00              127        53.42         True    0.8067
  200      3         0         13200            22.00              199        99.50         True    0.8067

Two columns go in opposite directions and never turn back. The rows read per prediction fall from 463.87 to 22.00, a factor of twenty-one; the worst wait rises from 0 to 199 arrivals and the average wait to 99.50. The last two columns close the quality side of the decision: at all nine sizes the answers are identical and the accuracy stays at 0.8067 — the base class ratio is 0.7964 and the base ratio of the test set is 0.8067, so this setup stands 0.0000 above the base. The batch size is not a quality setting; it is a setting between cost and wait.

The gain is not linear. The first doubling lowers 463.87 only to 420.75, because the regions of two requests are mostly different among five regions and the two region histories are still read separately. The gain reaches its highest speed between sizes 16 and 32 and then slows down: at size 200 all the regions are already in a single group, so there is no new history left to share. What request batching gains is not the number of requests but the rows that the requests read in common.

The third column carries the table’s own flaw. At sizes 16, 32 and 64 the last group of each period stays at eight requests, and at size 128 at seventy-two. These requests wait for a group that will never fill and are released only by a timeout. Their wait does not show in the “worst wait” column, because that column assumes the group fills. The measured worst value stands below an unmeasured assumption.

Setup Cost and Cache

Rows read is not the only name of the cost. Every call pays a setup cost, and this cost changes with the unit the call runs on. The second block compares the two accelerator classes in the same sweep, computes the crossover point and adds a stream of repeated requests at the end.

# This block continues the definitions of ALL , REQUESTS , SERVING , N , SIZES , WORK ,
# GENERAL , ACCEL , SUBSCRIBERS , PERIODS , SEED , rng and process_group from the first block.
print(" size      actual group  general-purpose   accelerator          cheaper")
for b in SIZES:
    wg, wa = WORK[b]
    groups = [len(REQUESTS[i:i + b]) for i in range(0, len(REQUESTS), b)]
    print(f"{b:>5}  {len(REQUESTS) / len(groups):>16.1f}  {wg / N:>15.2f}  {wa / N:>12.2f}"
          f"  {'accelerator' if wa < wg else 'general-purpose':>15}")
for setup in (150, 300, 600):
    print(f"setup {setup:>3} -> crossover point {(setup - GENERAL[0]) / (GENERAL[1] - ACCEL[1]):.1f} requests")


def request_stream(n):
    r = rng(SEED + 911)
    return [f"A{int(r() * 200) + 1:03d}" for _ in range(n)]


def cost(calls, size, period="2026-12"):
    return sum(process_group(calls[i:i + size], period)[1]
               for i in range(0, len(calls), size))


STREAM = request_stream(600)
DISTINCT = list(dict.fromkeys(STREAM))
print(f"\nstream {len(STREAM)} requests , distinct {len(DISTINCT)} , cache hit rate"
      f" {(len(STREAM) - len(DISTINCT)) / len(STREAM):.4f}")
base = cost(STREAM, 1)
for name, calls, b in (("neither , size 1", STREAM, 1), ("batching only", STREAM, 32),
                       ("cache only", DISTINCT, 1), ("both together", DISTINCT, 32)):
    rd = cost(calls, b)
    print(f"{name:<24} read {rd:>7}  per request {rd / len(STREAM):>7.2f}"
          f"  factor {rd / base:.4f}")
t1, t2 = cost(STREAM, 32) / base, cost(DISTINCT, 1) / base
print(f"predicted by the product {t1 * t2:.4f} , actual"
      f" {cost(DISTINCT, 32) / base:.4f}")

 size      actual group  general-purpose   accelerator          cheaper
    1               1.0            64.00        301.00  general-purpose
    2               2.0            34.00        151.00  general-purpose
    4               4.0            19.00         76.00  general-purpose
    8               8.0            11.50         38.50  general-purpose
   16              15.4             7.90         20.50  general-purpose
   32              28.6             6.10         11.50  general-purpose
   64              50.0             5.20          7.00  general-purpose
  128             100.0             4.60          4.00      accelerator
  200             200.0             4.30          2.50      accelerator
setup 150 -> crossover point 30.0 requests
setup 300 -> crossover point 80.0 requests
setup 600 -> crossover point 180.0 requests

stream 600 requests , distinct 193 , cache hit rate 0.6783
neither , size 1         read  278916  per request  464.86  factor 1.0000
batching only            read   47872  per request   79.79  factor 0.1716
cache only               read   89221  per request  148.70  factor 0.3199
both together            read   15829  per request   26.38  factor 0.0568
predicted by the product 0.0549 , actual 0.0568

The accelerator class is more expensive up to size 128. The crossover point is not a measurement but a ratio: the setup difference divided by the unit difference, that is (300 - 60) / (4 - 1) = 80 requests. The reason the switch shows at 128 in the table is the second column. When size 64 is requested, every period splits into groups of 64, 64, 64 and 8 requests, and the realized average group is 50, which is below 80; at size 128 the realized average is 100, which is above 80. The case for an accelerator lies not in its own speed but in the group size that can be filled; with a group that cannot be filled, the accelerator is only an expensive general-purpose unit. The setup cost sweep shows this with three numbers: the crossover points of setups 150, 300 and 600 are 30, 80 and 180 requests.

The last four rows add the cache. In the stream of six hundred requests there are only 193 distinct requests; the cache hit rate is 0.6783 and the rows read per request fall from 464.86 to 148.70. Request batching alone is stronger: 79.79. The two together give 26.38. But the two factors cannot be measured separately and multiplied: the product of 0.1716 and 0.3199 predicts 0.0549 and the actual is 0.0568. The difference is small and its cause is measurable — the cache thins the groups. When the remaining 193 requests are split into groups of thirty-two, a single request is left in the last group, and that single request reads the whole region history on its own. The gain of one mechanism depends on which mechanism stands next to it.

Summary

When the batch size is swept between one and two hundred, the rows read per prediction fall from 463.87 to 22.00, the worst wait rises from 0 to 199 arrivals and the average wait to 99.50; the two columns go in opposite directions and a size cannot be chosen without writing which of them is paid.
At all nine sizes the answers are identical and the accuracy stays at 0.8067; the base class ratio is 0.7964, the base ratio of the test set is 0.8067 and this setup stands 0.0000 above it: the batch size is not a quality setting.
The gain comes not from the number of requests but from the rows the requests read in common; the first doubling lowers 463.87 only to 420.75, because the regions of two requests are mostly different.
At sizes 16, 32 and 64 the last group of each period stays at eight requests and waits for a group that will never fill; this wait does not show in the “worst wait” column at all, because that column assumes the group fills.
The crossover point of the accelerator class is (300 - 60) / (4 - 1) = 80 requests, but in the table the switch shows at 128: what is measured is not the requested size but the realized group size (50 when 64 is requested, 100 when 128 is requested).
On a stream of 600 requests the cache lowers 464.86 to 148.70 with a hit rate of 0.6783; together with request batching it gives 26.38, while the product of the two factors predicts 25.52 — the cache thins the groups.

Next Step

In five lessons the model was packaged, placed in a pattern and scaled. In all five there was a silent assumption: the model sits at an endpoint that is reachable over the network from the place where the request arrives. When the network drops, the cost per prediction of this setup is undefined, because there is no prediction. The next lesson moves the model to the source of the request, to the device itself; it measures what shrinking takes from the accuracy and in which task it takes it first, and counts the independence gained by running at the edge as answered requests.

To keep your progress and take notes, Log in