Serving Patterns

The previous lesson packaged the model: the runtime was pinned, the interface contract was written, and the number of calls that dropped when no contract was written was counted. What the package accepts and what it returns is now written down. The only thing not written is when this package runs. The same package can run once at the start of a period and write two hundred predictions to a table, can run once whenever a request arrives, or can refresh itself whenever a new reading lands. The weights are the same in all three.

These three placements are called a serving pattern. The difference between them is not in the model but in the moment and the scope at which inference is done. This lesson runs the three side by side on the same workload and, instead of a single accuracy number, writes three numbers separately: the work done at the moment the request arrives, the age of the newest reading the answer rests on, and the cost per prediction.

• TD41. The source, the pipeline and the model are fictional and are generated inside the lesson. No serving framework, inference service, accelerator or cloud service is called; all three patterns are written here. The seed is 20260218.
• TD42. The label is imbalanced: the distribution is {0: 1752, 1: 448} and the base class ratio is 0.7964. No accuracy is read without writing how far above this ratio it stands. The test set is 600 rows, so the smallest step of accuracy is 0.0017.
• TD43. In this course the training pipeline is handled on a single machine; partitioning and shuffling were measured in the Distributed Data Processing course and are not repeated.
• TD44. The cost is counted as rows read; no raw milliseconds are written. The latency is written as a class: lookup is the reading of a single stored record at request time; scan is the reading of the history of the subscriber and of its region at request time.
• TD45. Freshness is counted in periods: the age of the answer is the number of periods that have to pass before the newest reading is reflected in the prediction.
• TD46. The serving horizon is the test periods, 2026-10, 2026-11, 2026-12, and in each period one request arrives for each of the two hundred subscribers. 600 requests in total.
• TD47. The batch pattern runs once per period. This is a choice and is swept below.

Three Patterns Side by Side

The first block sets up the source, the three stages of the pipeline up to consumption, the feature path and the fictional model; then it runs the three patterns with the same weights. The features are computed once and are independent of the weights.

# FICTIONAL. The meter readings of the measurement network, the first three stages of the
# pipeline, the feature path and the threshold model with three weights are generated inside
# the lesson; no learning library, serving framework or inference service is called.
# Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out = []
    for k in records:
        i = PERIODS.index(k["period"])
        prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if prev is not None:
            out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out


TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


def features(records, window=None, digits=4):
    """The training path sees the full history. Independent of the weights , computed once."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def train(records, feats):
    subset = [k for k in records if k["period"] in TRAIN]
    best, best_w = -1.0, None
    for w in GRID:
        d = sum(1 for k in subset
                if (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k))
        if d / len(subset) > best:
            best, best_w = d / len(subset), w
    return {"weights": best_w, "train_rows": len(subset), "candidates": len(GRID)}


ALL = s3_consumption(generate_source())
FEATS = features(ALL)
MODEL = train(ALL, FEATS)
W = MODEL["weights"]
SERVING, REQUESTS = TEST, sorted(SUBSCRIBERS)


def batch_run(records, period, w):
    """The whole history is scanned once , all predictions are produced and stored."""
    feats = features(records)
    return ({k["subscriber"]: (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0)
             for k in records if k["period"] == period}, len(records))


def online_request(idx, sub, period, w):
    """At request time the history of the subscriber and of its region is scanned."""
    by_sub, by_reg = idx
    i, reg = PERIODS.index(period), by_sub[sub][0]["region"]
    sub_vals = [k["m3"] for k in by_sub[sub] if PERIODS.index(k["period"]) < i]
    reg_vals = [k["m3"] for k in by_reg[reg] if PERIODS.index(k["period"]) < i]
    o = {"subscriber_mean": round(sum(sub_vals) / len(sub_vals), 4) if sub_vals else 0.0,
         "region_mean": round(sum(reg_vals) / len(reg_vals), 4) if reg_vals else 0.0,
         "commercial": 1 if by_sub[sub][0]["type"] == "commercial" else 0}
    return (1 if score(o, w) > THRESHOLD else 0), len(by_sub[sub]) + len(by_reg[reg])


def stream_run(records, w):
    """As each reading arrives the state is updated incrementally and the prediction is refreshed."""
    sub_state, reg_state, pred, read, refreshes = {}, {}, {}, 0, 0
    for k in sorted(records, key=lambda k: (PERIODS.index(k["period"]), k["subscriber"])):
        read += 1
        sub, reg = k["subscriber"], k["region"]
        o = {"subscriber_mean": round(sub_state[sub][0] / sub_state[sub][1], 4) if sub in sub_state else 0.0,
             "region_mean": round(reg_state[reg][0] / reg_state[reg][1], 4) if reg in reg_state else 0.0,
             "commercial": 1 if k["type"] == "commercial" else 0}
        pred[(sub, k["period"])] = 1 if score(o, w) > THRESHOLD else 0
        sub_state.setdefault(sub, [0, 0]); reg_state.setdefault(reg, [0, 0])
        sub_state[sub][0] += k["m3"]; sub_state[sub][1] += 1
        reg_state[reg][0] += k["m3"]; reg_state[reg][1] += 1
        refreshes += 1
    return pred, read, refreshes, len(sub_state) + len(reg_state)


by_sub, by_reg = {}, {}
for k in ALL:
    by_sub.setdefault(k["subscriber"], []).append(k)
    by_reg.setdefault(k["region"], []).append(k)
IDX = (by_sub, by_reg)

TRUTH = {(k["subscriber"], k["period"]): label(k) for k in ALL if k["period"] in SERVING}
N = len(REQUESTS) * len(SERVING)


def accuracy(t):
    return round(sum(1 for c in TRUTH if t[c] == TRUTH[c]) / N, 4)


BATCH, read_b = {}, 0
for d in SERVING:
    t, o = batch_run(ALL, d, W)
    read_b += o
    for a in REQUESTS:
        BATCH[(a, d)] = t[a]

ONLINE, read_o, min_read, max_read = {}, 0, 10 ** 9, 0
for d in SERVING:
    for a in REQUESTS:
        p, o = online_request(IDX, a, d, W)
        ONLINE[(a, d)], read_o = p, read_o + o
        min_read, max_read = min(min_read, o), max(max_read, o)

STREAM, read_s, refreshes, state = stream_run(ALL, W)
print(f"training rows {MODEL['train_rows']} , weights {W} , candidates {MODEL['candidates']}")
print(f"serving horizon {len(SERVING)} periods x {len(REQUESTS)} subscribers = {N} requests")
print("pattern    request time  answer age     rows read  per prediction  accuracy")
for name, age, rd, acc in (("batch", 1, read_b, accuracy(BATCH)),
                           ("online", 0, read_o, accuracy(ONLINE)),
                           ("stream", 0, read_s, accuracy(STREAM))):
    print(f"{name:<9}  {'scan' if name == 'online' else 'lookup':>12}  {age:>10}"
          f"  {rd:>12}  {rd / N:>14.2f}  {acc:>8}")
print(f"rows read per online request: min {min_read} , max {max_read}")
print(f"stream: refreshes {refreshes} , used {N} , state entries held {state}")

training rows 1600 , weights (0.6, 0.4, 0) , candidates 27
serving horizon 3 periods x 200 subscribers = 600 requests
pattern    request time  answer age     rows read  per prediction  accuracy
batch            lookup           1          6600           11.00    0.8067
online             scan           0        278322          463.87    0.8067
stream           lookup           0          2200            3.67    0.8067
rows read per online request: min 352 , max 550
stream: refreshes 2200 , used 600 , state entries held 205

The last column is 0.8067 in all three rows. The same weights, the same feature path, the same 600 requests; all three patterns give the same answers. The label is imbalanced and the base class ratio is 0.7964; the base ratio of the test set itself is also 0.8067, so this setup stands 0.0000 above the base, just as it was written in the first lesson of the course. The choice of pattern does not change that either, and an observation that looks only at the output cannot tell which pattern ran.

The difference is in the other columns. The online pattern reads 352 to 550 rows per request, 463.87 on average. Almost all of this is the history of the region; the history of the subscriber itself is eleven rows. The region_mean feature cannot be computed for a single subscriber, it asks for all the subscribers of the region. Online inference is expensive not because of the model but because of the scope of the feature, and this is why each prediction reads forty-two times as many rows.

Batch inference computes the same region mean once and shares it among all the subscribers of the region: 6600 rows, 600 predictions, 11.00 per prediction. In return the answer ages by one period, because no reading that arrives between two runs can enter the prediction. The streaming pattern reads each row once and updates the state incrementally: 2200 rows, 3.67 per prediction. The table does not show its cost; of the 2200 refreshes only 600 are used, 205 state entries are held continuously, and rebuilding the state when it is lost means reading the 2200 rows from the start. The rows read column is the column where streaming looks cheapest, and the cost of streaming is not written in exactly that column.

What the Cost Is Divided By

The answer aging by a period looks like a cost. The second block measures it first: it answers the same requests from an ever older run. Then it changes the denominator of the cost per prediction.

# This block continues the definitions of ALL , FEATS , W , SERVING , REQUESTS ,
# ONLINE , read_o , N , score , THRESHOLD , PERIODS and accuracy from the first block.
print("answer age (periods)  changed predictions  accuracy")
for g in range(0, 7):
    t = {}
    for d in SERVING:
        j = max(PERIODS.index(d) - g, 1)
        for a in REQUESTS:
            t[(a, d)] = 1 if score(FEATS[(a, PERIODS[j])], W) > THRESHOLD else 0
    chg = sum(1 for c in t if t[c] != ONLINE[c])
    print(f"{g:>20}  {chg:>19}  {accuracy(t):>8}")

dist = sorted(abs(score(FEATS[c], W) - THRESHOLD) for c in ONLINE)
print(f"threshold {THRESHOLD} , distance of the decision score from the threshold: min {dist[0]:.4f}"
      f" , median {dist[len(dist) // 2]:.4f}")

print("\n predictions used    batch     online  stream  cheapest         batch unused")
mean_o = read_o / N
for u in (1, 2, 5, 20, 50, 200):
    s = [len(ALL) / u, mean_o, len(REQUESTS) / u]
    print(f"{u:>17}  {s[0]:>7.2f}  {s[1]:>9.2f}  {s[2]:>6.2f}"
          f"  {('batch', 'online', 'stream')[s.index(min(s))]:>8}  {len(REQUESTS) - u:>19}")
print(f"batch and online cross at {len(ALL) / mean_o:.2f} predictions used")

answer age (periods)  changed predictions  accuracy
                   0                    0    0.8067
                   1                    0    0.8067
                   2                    0    0.8067
                   3                    0    0.8067
                   4                    0    0.8067
                   5                    0    0.8067
                   6                    0    0.8067
threshold 40 , distance of the decision score from the threshold: min 7.6468 , median 13.6826

 predictions used    batch     online  stream  cheapest         batch unused
                1  2200.00     463.87  200.00    stream                  199
                2  1100.00     463.87  100.00    stream                  198
                5   440.00     463.87   40.00    stream                  195
               20   110.00     463.87   10.00    stream                  180
               50    44.00     463.87    4.00    stream                  150
              200    11.00     463.87    1.00    stream                    0
batch and online cross at 4.74 predictions used

The result of the staleness sweep is a single number: while the age of the answer goes from zero to six periods, the changed predictions are 0 and the accuracy stays at 0.8067. On this workload staleness has no measurable counterpart, and “the batch pattern is free” is not concluded from it. The reason is one line below: the smallest distance of the decision score to the threshold is 7.6468 and its median is 13.6826. The decisions stay away from the threshold, and one more period of history does not move the eleven-period cumulative mean enough to close that distance. When the decisions approach the threshold, the same staleness moves the prediction. A cost that cannot be measured is not ignored; it is written that it could not be measured and under which condition it would show — this is the second claim of the course read here from the side of the cost.

The second table shows that the cost per prediction is a denominator decision. The batch pattern reads 2200 rows in one run and produces two hundred predictions; how many of those predictions are actually read is a question of the task, not of the pattern. If a single prediction is used, the cost per prediction of the batch pattern is 2200.00, which is more than four times the 463.87 of the online pattern; the two patterns are equal at 4.74 predictions used. The last column is the name of the cost of the batch pattern: when one prediction is used, 199 predictions are produced and never read.

Which task rests on which pattern is read from these two tables. The job that lists all the subscribers at the start of the period rests on the batch pattern: the denominator is 200, the staleness cannot be measured, and the cost drops to 11.00. A single-subscriber query that is not known in advance rests on the online pattern: the denominator is 1, and the batch pattern is more than four times as expensive there, and moreover cannot answer a key that is not in the list at all. The job that has to be triggered at the moment the reading arrives is done only with streaming: streaming produces 2200 triggers, the online pattern produces 600 only when a request arrives and triggers nothing if no request arrives, while the batch pattern produces the 600 at the start of the period and one period late.

Summary

• The three serving patterns give the same answers to the same 600 requests and the accuracy is 0.8067 in all three; the base class ratio is 0.7964, the base ratio of the test set is 0.8067 and this setup stands 0.0000 above it. The choice of pattern is not an accuracy decision, and an observation that looks only at the output cannot tell which pattern ran.
• The difference is in three columns: the latency class at request time is lookup in the batch and streaming patterns and a scan of 352 to 550 rows in the online pattern; the age of the answer is 1, 0 and 0 periods; the rows read per prediction are 11.00, 463.87 and 3.67.
• The forty-two-fold difference of the online pattern comes not from the model but from the scope of the region_mean feature; the history of the subscriber itself is eleven rows, that of its region is between 341 and 539 rows.
• When the age of the answer is raised to six periods, the changed predictions are 0; the reason is not the pattern but that the decision score stays at least 7.6468 away from the threshold. A cost that cannot be measured is not a measured guarantee.
• The cost per prediction depends on the denominator: the batch pattern gives 2200.00 when one prediction is used and 11.00 when two hundred are used, and it becomes equal to the online pattern at 4.74 predictions used.
• The streaming pattern is the cheapest in rows read, but 1600 of its 2200 refreshes are not used and 205 state entries are held continuously; this cost does not show up in the rows read column at all.

Next Step

The pattern is chosen, but within each pattern one question is still unanswered: how many requests a call should carry together. In the batch pattern this is how many predictions a run produces; in the online pattern it means accumulating the arriving requests and processing them at once, and the cost of accumulating is that the first request that arrived waits. The next lesson runs the same requests at batch sizes between one and sixty-four, counts the fall of the cost per prediction side by side with the latency of the waiting request, and repeats the same sweep on two accelerator classes.