Edge and On-Device Inference

The previous lesson made the request cheaper on the server side: when the batch size rose to two hundred, the rows read per prediction fell to 22.00. Five lessons packaged, placed and scaled the model; beneath all five lay the same silent assumption: the model sits at an endpoint that is reachable over the network from the place where the request is born. When that network goes down, the cost per prediction is undefined, because there is no prediction.

This lesson moves the model to the source of the request. What moves is small: three weights. What cannot move is the history that produces the feature those weights multiply. The lesson first counts that history, then measures what dropping it and quantization take from the two tasks, and finally writes the independence of the edge as answered requests.

• TD55. The source, the pipeline, the model and the shrinking steps are fictional; no edge device, runtime, accelerator hardware or cloud service name is written. The seed is 20260218.
• TD56. The label is imbalanced: {0: 1752, 1: 448}, the base class ratio is 0.7964; the test set is 600 rows and the step is 0.0017.
• TD57. The quantization and distillation procedure was measured in the Building Applications with Model APIs course; it is not repeated and its numbers are inputs.
• TD58. Shrinking has two steps: because the region history is not on the device, region_mean is dropped and the model is retrained on a grid of nine candidates; then the features are reduced to level distinct values.
• TD59. Two tasks are measured: task A has two classes (threshold 40), task B has three tiers (thresholds 20 and 40).
• TD60. The network outage rate p means that a share p of the requests receive no answer at all from the center; the edge answers all of them. The one number that compares the two in a common unit is the correct-answer count (answered requests times accuracy).

The History That Cannot Move to the Device

The first block sets up the source, pipeline and tasks, trains the two models separately and sweeps the level.

# FICTIONAL. The meter readings of the measurement network, the first three stages of the
# pipeline, the feature path, the threshold model with three weights and the shrinking steps
# are generated inside the lesson; no edge device, runtime or hardware is called.
# Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out = []
    for k in records:
        i = PERIODS.index(k["period"])
        prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if prev is not None:
            out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out


TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40


def features(records):
    """The training path sees the full history. Independent of the weights , computed once."""
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if d < i]
        return round(sum(chosen) / len(chosen), 4) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
EDGE_GRID = [w for w in GRID if w[1] == 0.0]      # the region history is not on the device

ALL = s3_consumption(generate_source())
FEATS = features(ALL)
M3 = {(k["subscriber"], k["period"]): k["m3"] for k in ALL}
TRAIN_KEYS = [c for c in M3 if c[1] in TRAIN]
TEST_KEYS = [c for c in M3 if c[1] in TEST]
N = len(TEST_KEYS)
LOW_S, HIGH_S = min(o["subscriber_mean"] for o in FEATS.values()), max(o["subscriber_mean"] for o in FEATS.values())
LOW_R, HIGH_R = min(o["region_mean"] for o in FEATS.values()), max(o["region_mean"] for o in FEATS.values())


def task_a(v):
    return 1 if v > THRESHOLD else 0


def task_b(v):
    return 0 if v <= 20 else (1 if v <= THRESHOLD else 2)


def train(task, grid, feats):
    return max(grid, key=lambda w: sum(1 for c in TRAIN_KEYS
                                       if task(score(feats[c], w)) == task(M3[c])))


def accuracy(task, w, feats):
    return round(sum(1 for c in TEST_KEYS if task(score(feats[c], w)) == task(M3[c])) / N, 4)


def share(task):
    return round(max(sum(1 for c in TEST_KEYS if task(M3[c]) == s) for s in (0, 1, 2)) / N, 4)


def quantize_features(level):
    def q(x, low, high):
        step = (high - low) / (level - 1)
        return low + round((x - low) / step) * step

    return {c: {"subscriber_mean": q(o["subscriber_mean"], LOW_S, HIGH_S),
                "region_mean": q(o["region_mean"], LOW_R, HIGH_R),
                "commercial": o["commercial"]} for c, o in FEATS.items()}


WA, WB = train(task_a, GRID, FEATS), train(task_b, GRID, FEATS)
WAK, WBK = train(task_a, EDGE_GRID, FEATS), train(task_b, EDGE_GRID, FEATS)
region_sizes = [sum(1 for c in M3 if SUBSCRIBERS[c[0]]["region"] == b) for b in REGION]
print(f"largest class share in the serving horizon: task A {share(task_a)} , task B {share(task_b)}")
print(f"center model  A {WA} accuracy {accuracy(task_a, WA, FEATS)}"
      f" | B {WB} accuracy {accuracy(task_b, WB, FEATS)}")
print(f"edge model    A {WAK} accuracy {accuracy(task_a, WAK, FEATS)}"
      f" | B {WBK} accuracy {accuracy(task_b, WBK, FEATS)}")
print(f"number written to the device 5 (3 weights + 2 sums of the subscriber) ;"
      f" dropped region history {min(region_sizes)}-{max(region_sizes)} rows")

base_a = {c: task_a(score(FEATS[c], WA)) for c in TEST_KEYS}
base_b = {c: task_b(score(FEATS[c], WB)) for c in TEST_KEYS}
print("\nlevel  distinct values  A changed  A accuracy  B changed  B accuracy")
for s in (256, 16, 12, 8, 4, 2):
    q_feats = quantize_features(s)
    ta = {c: task_a(score(q_feats[c], WA)) for c in TEST_KEYS}
    tb = {c: task_b(score(q_feats[c], WB)) for c in TEST_KEYS}
    fd = (len(set(o["subscriber_mean"] for o in q_feats.values()))
          + len(set(o["region_mean"] for o in q_feats.values())))
    print(f"{s:>5}  {fd:>15}  {sum(1 for c in TEST_KEYS if ta[c] != base_a[c]):>9}"
          f"  {accuracy(task_a, WA, q_feats):>10}  {sum(1 for c in TEST_KEYS if tb[c] != base_b[c]):>9}"
          f"  {accuracy(task_b, WB, q_feats):>10}")

largest class share in the serving horizon: task A 0.8067 , task B 0.4833
center model  A (0.6, 0.4, 0) accuracy 0.8067 | B (0.6, 0.4, 4) accuracy 0.4833
edge model    A (0.6, 0.0, 0) accuracy 0.8067 | B (1.0, 0.0, 4) accuracy 0.4733
number written to the device 5 (3 weights + 2 sums of the subscriber) ; dropped region history 341-539 rows

level  distinct values  A changed  A accuracy  B changed  B accuracy
  256              199          0      0.8067          0      0.4833
   16               21          0      0.8067          2        0.48
   12               16          0      0.8067          0      0.4833
    8               11          0      0.8067          5      0.4783
    4                7          0      0.8067          0      0.4833
    2                4        402      0.3833        600       0.225

The fourth line is the first sentence of the lesson: the number written to the device is five, and what is dropped is the region history of 341 to 539 rows. What cannot move to the edge is not the model but the history that produces the feature. The cost splits: in task A the accuracy stays at 0.8067, in task B it falls to 0.4733.

Placing the two accuracies next to the baseline changes the name of the cost. The label is imbalanced and the base class ratio is 0.7964; in the serving horizon the largest class of task A holds a share of 0.8067 and the largest tier of task B holds 0.4833, and the center gives exactly that share in both. The zero drop in task A comes from this: there is nothing there for shrinking to take away. The edge model, however, falls 0.0100 below the baseline in task B, six steps worse than a procedure that says the largest tier for every row. The cost of shrinking is not the drop in accuracy but which side of the baseline one stays on.

Quantization confirms the same order independently. Task A does not move a single prediction down to level 4; task B moves two predictions at level 16 and five at level 8. These movements are just above the 0.0017 resolution and the sweep is not monotonic, so the point where degradation begins is written as “between 16 and 2”. At level 2 both fall below the baseline: task A 0.3833 (0.4234 below), task B 0.2250 (0.2583 below). Shrinking erases the distinction, erases the finest task first, and once it goes far enough it carries the model below the baseline.

The Request Answered Under an Outage

What shrinking takes away was written; what it gives was not.

# This block continues the definitions of N , FEATS , WA , WB , WAK , WBK , task_a , task_b ,
# accuracy , quantize_features , rng and SEED from the first block.
mA, mB = accuracy(task_a, WA, FEATS), accuracy(task_b, WB, FEATS)
kA, kB = accuracy(task_a, WAK, FEATS), accuracy(task_b, WBK, FEATS)
r = rng(SEED + 733)
outage = [r() for _ in range(N)]
print("outage rate  answered  center A  edge A  center B  edge B")
for p in (0.00, 0.02, 0.05, 0.10, 0.20):
    y = sum(1 for x in outage if x >= p)
    print(f"{p:>11.2f}  {y:>8}  {round(y * mA):>8}  {round(N * kA):>6}"
          f"  {round(y * mB):>8}  {round(N * kB):>6}")
print(f"crossover point: task A {1 - kA / mA:.4f} , task B {1 - kB / mB:.4f}")
print("when the edge model is quantized  level 8: task B", accuracy(task_b, WBK, quantize_features(8)),
      ", level 4:", accuracy(task_b, WBK, quantize_features(4)))

outage rate  answered  center A  edge A  center B  edge B
       0.00       600       484     484       290     284
       0.02       589       475     484       285     284
       0.05       566       457     484       274     284
       0.10       536       432     484       259     284
       0.20       485       391     484       234     284
crossover point: task A 0.0000 , task B 0.0207
when the edge model is quantized  level 8: task B 0.4767 , level 4: 0.4533

The edge columns are fixed, and this is the one real gain of the edge: whatever the outage rate, all 600 requests are answered, while at the center the answered requests fall to 485. The correct answers fall from 484 to 391 in task A and from 290 to 234 in task B. The two setups are equal at an outage rate of 0.0000 in task A and 0.0207 in task B: above this rate, the edge model that has fallen below the baseline still produces more correct answers. Independence is not an accuracy number; it enters the same table only through the correct-answer count. The last line combines both steps: 0.4767 at level 8, 0.4533 at level 4.

Summary

• The number written to the device is five; what cannot move is the region history of 341 to 539 rows.
• Dropping the history leaves task A unchanged and lowers task B to 0.4733. The base class ratio is 0.7964 and the largest tier of task B is 0.4833: the edge model is 0.0100 below the baseline, and the zero in A is not robustness but an absence of measurement.
• Quantization confirms the same order: task A does not move a single prediction down to level 4, task B moves two at level 16 and five at level 8; at level 2 the two collapse to 0.3833 and 0.2250.
• Whatever the outage rate, the edge answers all 600 requests; at a rate of 0.20 the correct answers of the center fall from 484 to 391 and from 290 to 234. The two setups are equal at an outage rate of 0.0000 in task A and 0.0207 in task B, and the case for the edge is that rate.

Course Wrap-Up

The rows carry no common unit; the metric each is read by sits next to the lesson name, and the difference in parentheses is pinned setup minus baseline.

The table is a single rule written twelve times. A model’s number is meaningful together with the path that produced it; a number that cannot be reproduced is not a measurement but a memory. In What Is MLOps the same job gives three different numbers, and the weights are the same in all three; in Data and Model Versioning the same name carries three different contents; in Model Registry runs with the same weights report 0.7950 and 0.8067. What changes is not the model but by which path the number was produced.

The second claim reads from the same table: pinning also has a cost, and a pinning whose contribution cannot be shown gives an unmeasured assurance. In Environment Pinning three items are enough and the contribution of the grid and the rounding is unmeasurable; in Experiment Tracking a ledger that holds only the metric shortens only one of five searches; in Continuous Integration for Machine Learning two of the four runs stopped by the data gate would have given the same model anyway; in Serving Patterns, unmeasurable staleness is no assurance of the pattern but a result of the decision score standing 7.6468 from the threshold. In four rows, the pinning that was paid for has not been repaid, or cannot be shown to have been.

The third claim becomes sharpest in a single row. In Feature Stores, with the same triple of weights, the accuracy is 0.8067 by the training path and 0.7800 by the serving path; the model never changed, and all of the 0.0267 comes from the history by which the feature was computed. Model Packaging is its silent twin: when the threshold field drops out of the package, the number of dropped calls does not change, and 1200 of the 1200 calls change their answer. Testing the model is not enough; the feature path in serving and the declaration of the package must be tested too.

Over sixty-one lessons data was taken in, moved, stored, partitioned, tied to a model, made reproducible, packaged, served, scaled and moved to the device. None of them asked about what comes after the model leaves. The world changes and the input starts to drift; the label arrives weeks later, so today’s accuracy cannot be measured today; and when the model will retire is written nowhere. The next course treats the model in production not as a delivery but as an object of observation, and measures these three questions.