Model Packaging

The run that passed the gates of the previous lesson now counted as having produced a model, but what was produced was still not written down. The model was a dictionary of three weights and a threshold; the calling side used it by guessing the names, types and order of the fields.

This lesson turns that dictionary into a package. Packaging means putting next to the weights everything the caller has to know: which fields are expected with which type, which parameter is carried in the package, which field comes back in return. This declaration is called the interface contract. The question of the lesson is who breaks when no contract is written: when a field name or type changes, how many calls drop, how many give a wrong answer without dropping at all, and how many of these the contract catches before release.

• TD21. The source, the pipeline, the model and the package are fictional. No packaging format, container runtime, serving framework or machine learning library name is written. The seed is 20260218.
• TD22. The data schema between producer and consumer and whom a schema change breaks were measured in the Introduction to Data Engineering course and are not repeated. The contract here is the contract not of the data but of the interface of the model.
• TD23. The package carries four things: the weights, the threshold, the declaration of the input fields and the name of the output field. The runtime, the source separation and the form of deployment are outside this lesson.
• TD24. A call counts as dropped if a field the package expects is missing from the record or if the type of the field does not enter arithmetic. Every call whose field is present and whose type matches passes.
• TD25. There are three callers and all three send the same 600 test rows; 1800 calls in total.
• TD26. The callers differ at three points: one reads the field names from the declaration of the package, one has hard-coded the names in its own code, one has hard-coded the names and carries one field as text.
• TD27. The contract is the published interface; both the package and the caller conform to it. Two forms of it are compared: one that covers only the input and one that also covers the output and the parameter.
• TD28. The label distribution is imbalanced; the base class ratio is written in this lesson too.
• TD29. The baseline is packaging without a contract: no declaration, no pre-release validation.
• TD30. Resolution: the test set is 600 rows, the step of accuracy is 0.0017; the step of the call count is 1 call.
• TD31. Time is not written as raw milliseconds; the measures are calls, records and declarations.

What Is Inside the Package

The first block sets up the source, the first three stages of the pipeline and the model; none of it is specific to this lesson.

# FICTIONAL. Source, pipeline, model and package are defined inside the lesson; no real
# machine learning library, packaging format or runtime is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


TRAIN, TEST, THRESHOLD = PERIODS[1:9], PERIODS[9:], 40


def label(k):
    return 1 if k["m3"] > THRESHOLD else 0


def features(records, window=None, digits=4):
    ordered = sorted(records, key=lambda k: PERIODS.index(k["period"]))
    sub, reg = {}, {}
    for k in ordered:
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        chosen = [v for d, v in series if (d < i and (window is None or d >= i - window))]
        return round(sum(chosen) / len(chosen), digits) if chosen else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def train(records, feats):
    subset = [k for k in records if k["period"] in TRAIN]
    best, best_w = -1.0, None
    for w in GRID:
        d = sum(1 for k in subset
                if (1 if score(feats[(k["subscriber"], k["period"])], w) > THRESHOLD else 0) == label(k))
        if d / len(subset) > best:
            best, best_w = d / len(subset), w
    return {"weights": best_w, "train_accuracy": round(best, 4),
            "train_rows": len(subset)}


def evaluate(records, model, feats):
    subset = [k for k in records if k["period"] in TEST]
    d = sum(1 for k in subset
            if (1 if score(feats[(k["subscriber"], k["period"])], model["weights"]) > THRESHOLD else 0)
            == label(k))
    return round(d / len(subset), 4)

The second block packages the model and sets up its three versions. v1 is the package in release, v2 is a package that changed the name of one input field, and v3 is a package that does not carry the threshold field at all. The three callers send the same 600 rows and the dropped calls are counted.

# ---- Packaging: weights , threshold and interface declaration in a single unit
SOURCE = generate_source()
V, _ = s1_screen(SOURCE)
V, _ = s2_dedupe(V)
ROWS, _ = s3_consumption(V)
FEATS = features(ROWS)
MODEL = train(ROWS, FEATS)
TEST_SET = [k for k in ROWS if k["period"] in TEST]
DISTRIBUTION = {e: sum(1 for k in ROWS if label(k) == e) for e in (0, 1)}
TEST_BASE = round(sum(1 for k in TEST_SET if label(k) == 0) / len(TEST_SET), 4)
print("label distribution:", DISTRIBUTION, " base class ratio:", round(DISTRIBUTION[0] / len(ROWS), 4))
print("weights:", MODEL["weights"], " training rows:", MODEL["train_rows"],
      " test rows:", len(TEST_SET),
      f" test accuracy: {evaluate(ROWS, MODEL, FEATS):.4f}  test base class ratio: {TEST_BASE:.4f}")

PACKAGE_V1 = {"weights": MODEL["weights"], "threshold": THRESHOLD, "output": "high",
              "input": ("subscriber_mean", "region_mean", "commercial")}
PACKAGE_V2 = dict(PACKAGE_V1, input=("subscriber_average", "region_mean", "commercial"))
PACKAGE_V3 = {a: v for a, v in PACKAGE_V1.items() if a != "threshold"}
VERSIONS = [("v1 reference", PACKAGE_V1), ("v2 field name", PACKAGE_V2), ("v3 no threshold", PACKAGE_V3)]
DEFAULT_THRESHOLD = 0


def call(package, record):
    w, g = package["weights"], package["input"]
    o = {a: record[a] for a in g}                               # missing field -> KeyError
    p = w[0] * o[g[0]] + w[1] * o[g[1]] + w[2] * o[g[2]]        # type mismatch -> TypeError
    return {package["output"]: 1 if p > package.get("threshold", DEFAULT_THRESHOLD) else 0}


def c_adaptive(package, o):
    """Reads the field names from the declaration of the package."""
    return dict(zip(package["input"], (o["subscriber_mean"], o["region_mean"], o["commercial"])))


def c_fixed_name(package, o):
    """Has hard-coded the field names in its own code."""
    return {"subscriber_mean": o["subscriber_mean"], "region_mean": o["region_mean"],
            "commercial": o["commercial"]}


def c_fixed_type(package, o):
    """Has hard-coded the names and carries the commercial field as text."""
    return dict(c_fixed_name(package, o), commercial=str(o["commercial"]))


CALLERS = [("adaptive", c_adaptive), ("fixed_name", c_fixed_name), ("fixed_type", c_fixed_type)]
print(f"{'version':<16}{'calls':>7}{'dropped':>9}{'passed':>7}{'high':>8}"
      + "".join(f"{a:>12}" for a, _ in CALLERS))
for vname, pkg in VERSIONS:
    dropped, passed, high, broken = 0, 0, 0, []
    for cname, cfn in CALLERS:
        d = 0
        for k in TEST_SET:
            try:
                high += call(pkg, cfn(pkg, FEATS[(k["subscriber"], k["period"])]))[pkg["output"]]
                passed += 1
            except (KeyError, TypeError):
                d += 1
        dropped += d
        broken.append(d)
    print(f"{vname:<16}{len(TEST_SET) * 3:>7}{dropped:>9}{passed:>7}{high:>8}"
          + "".join(f"{x:>12}" for x in broken))

label distribution: {0: 1752, 1: 448}  base class ratio: 0.7964
weights: (0.6, 0.4, 0)  training rows: 1600  test rows: 600  test accuracy: 0.8067  test base class ratio: 0.8067
version           calls  dropped passed    high    adaptive  fixed_name  fixed_type
v1 reference       1800      600   1200       0           0           0         600
v2 field name      1800     1200    600       0           0         600         600
v3 no threshold    1800      600   1200    1200           0           0         600

The Caller That Breaks When No Contract Is Written

The first two lines write the model itself and carry the warning repeated in every lesson of this course. The label distribution is 1752 to 448, that is, the base class ratio is 0.7964; in the test set this ratio is 0.8067 and the test accuracy of the model is also 0.8067. Accuracy misleads when it is read alone: the packaged model produces, in the test set, exactly the share of the majority class. All the call counts below are calls of this model.

The baseline row is v1, and even there 600 of the 1800 calls drop. What breaks them is not a version change but a type mismatch that has been there from the start: the fixed_type caller carries the commercial field as text and the score computation cannot enter arithmetic. In a setup without a contract this mismatch comes out only in production, call by call.

The v2 row gives the cost of the field name change. The package starts to expect subscriber_average instead of subscriber_mean and the dropped calls rise from 600 to 1200. Two callers that have hard-coded the names break at once, because the record they send does not have the field the package is looking for. The only caller left standing is the one that reads the names from the declaration of the package, and all 600 of its calls pass. A package’s declaring its field names and the callers’ reading that declaration are two separate decisions; without the second the first is of no use.

The v3 row is the most dangerous one, and no observation that looks at the number of dropped calls can see it. The threshold field has been taken out of the package; when the caller cannot find the field it falls back on its own default, that is, on 0. The number of dropped calls does not change: in v3 as in v1, 600 calls drop and 1200 pass. The only column that changes is the last. All 1200 of the 1200 calls that pass return the answer “high”, whereas the number of highs returned in v1 was 0. The model is the same, the weights are the same, the caller is the same; every answer has flipped because of a single missing parameter. There is not a single row that lands in the error log.

What the Contract Catches Before Release

The third block writes the contract. The contract is the published interface: both the package and the caller conform to it, and the two look not at each other but at the contract. Two forms of it are compared: a contract that declares only the input fields, and a contract that also declares the output and the parameter.

# ---- Interface contract: the published interface. Both package and caller conform to it.
TYPES = {"float": float, "integer": int}
CONTRACT_INPUT = {"input": {"subscriber_mean": "float", "region_mean": "float",
                            "commercial": "integer"}}
CONTRACT_FULL = dict(CONTRACT_INPUT, output="high", parameters=("threshold",))


def validate(contract, package, samples):
    findings = []
    if set(package["input"]) != set(contract["input"]):
        findings.append("package: input field name")
    if "output" in contract and package["output"] != contract["output"]:
        findings.append("package: output field name")
    for p in contract.get("parameters", ()):
        if p not in package:
            findings.append(f"package: missing parameter {p}")
    for name, sample in samples:
        for a, t in contract["input"].items():
            if a not in sample:
                findings.append(f"{name}: missing field {a}")
            elif type(sample[a]) is not TYPES[t]:
                findings.append(f"{name}: {a} type")
    return findings


SAMPLE = FEATS[(TEST_SET[0]["subscriber"], TEST_SET[0]["period"])]
print(f"{'version':<16}{'input contract':>18}{'full contract':>14}   findings of the full contract")
for vname, pkg in VERSIONS:
    samples = [(cname, cfn(pkg, SAMPLE)) for cname, cfn in CALLERS]
    g = validate(CONTRACT_INPUT, pkg, samples)
    t = validate(CONTRACT_FULL, pkg, samples)
    print(f"{vname:<16}{len(g):>18}{len(t):>14}   {t}")
print("records read before release:", len(CALLERS),
      " calls in the setup without a contract:", len(TEST_SET) * 3,
      " declarations in the contract:", len(CONTRACT_FULL["input"]) + 1 + len(CONTRACT_FULL["parameters"]))

version             input contract full contract   findings of the full contract
v1 reference                     1             1   ['fixed_type: commercial type']
v2 field name                    3             3   ['package: input field name', 'adaptive: missing field subscriber_mean', 'fixed_type: commercial type']
v3 no threshold                  1             2   ['package: missing parameter threshold', 'fixed_type: commercial type']
records read before release: 3  calls in the setup without a contract: 1800  declarations in the contract: 5

In all three versions the number of dropped calls falls to 0, because the mismatch is found before release and the version is not published. The measured difference is where the mismatch is seen: in the setup without a contract among the 1800 calls, in the setup with a contract in 3 sample records. The validation reads a single record per caller.

The v2 row shows what the contract is, too. Of the three findings, one belongs to the package and two to the callers. One of them is surprising: the adaptive caller, which reads the declaration of the package, never dropped in production, yet it produces a record that is incompatible according to the contract. Both are right. That caller conforms to the package and does not conform to the contract; because the package deviated from the contract and the caller followed the deviation silently. The job of the contract is not to compare the package with the caller, but to compare both of them with the published interface.

The v3 row writes that the contract itself can be defective. The contract that declares only the input fields produces a single finding in v3, and that finding is irrelevant: the type mismatch that the fixed_type caller has had from the start. It does not see the missing threshold field, because there is no such thing as a parameter in its contract. The contract that also declares the output and the parameter produces two findings in the same version, and the first is exactly the missing parameter. The input contract catches two of the three versions as well; the only thing that catches the silent break that flips 1200 answers is the declaration of the parameter.

The cost of the contract can be counted too. Five declarations are written: three input fields, one output field, one parameter. A validation step is added before every release, and that step asks for one sample record per caller. A contract with an item left out gives an unmeasured assurance where it does not cover: the input contract looked green on v3.

The rule of the course applies to the interface too: a number that cannot be reproduced is not a measurement but a memory. The answer a call returns cannot be reproduced if the interface that produced it is not written down; the 1200 answers of v3 are the proof.

Summary

• Packaging means putting next to the weights everything the caller has to know: the weights, the threshold, the input declaration and the output field. This declaration is the interface contract.
• The label distribution is 1752 to 448, the base class ratio is 0.7964; the test accuracy of the packaged model, 0.8067, is exactly the base ratio in the test set.
• In the setup without a contract, even in the baseline version 600 of the 1800 calls drop; when the field name changes the dropped calls rise to 1200, and the only caller left standing is the one that reads the names from the declaration of the package.
• When the threshold field falls out of the package, the number of dropped calls does not change: as in v1, 600 calls drop, 1200 calls pass and all 1200 of them change their answer; the number of highs, which was 0 in v1, becomes 1200 in v3.
• In the setup with a contract, the dropped calls are 0 in all three versions; the mismatch is seen in 3 sample records instead of 1800 calls, and its cost is 5 declarations and one validation step.
• The contract that covers only the input catches two of the three versions; the only item that catches the silent break is the declaration of the parameter.

Next Step

In these three lessons the model was attached to a pipeline, passed through gates and packaged with an interface. The assumption common to all three can be written in one sentence: the caller calls whenever it wants and the answer comes at once. The 1800 calls of this lesson were counted the same way, as if all of them ran at the same time and at the same cost. Yet running a packaged model once at the end of the night for twelve periods and running it for a single row in the middle of a request are not the same thing. The next lesson separates the serving patterns: batch, online and streaming inference. The measure is the latency class, the staleness and the cost per prediction of the three patterns.