Shadow and Canary Deployment

The previous topic measured how the degradation is seen in six lessons: an input drift produced an alert but did not lower accuracy, a concept drift lowered accuracy without showing in any proxy metric, the only measure that saw it arrived two periods late, and the label collected from production was wrong. The common flaw of the six lessons is this: none of them asked what to do once it is seen.

The first thing done once it is seen is to change the model. Changing is itself a risk: the new model runs in production for the first time, and the only thing known about it is its behavior on the training set. This lesson puts the new model into release in three different placements and counts a single question: how much the share sent to the new model moves the result, and how much it moves the measure. Release staging itself was built in the Continuous Integration and Delivery course of the DevOps and Platform Engineering curriculum, and rollback and staged release were measured in the Resilience and Reliability course of the System Design and Distributed Systems curriculum; neither is repeated here. The question here is not about deployment but about the measure.

• RG1. The source, the pipeline and the model are fictional and are generated inside the lesson. No deployment tool, router, experiment platform or monitoring product is called. Seed 20260218.
• RG2. The label is tied to a rule that the features see: subscriber_mean + 6 x commercial > 30. The decision threshold is a setting separate from this rule and is searched during training; the training criterion is balanced accuracy, that is, the mean of recall and specificity. All three were established in the first lesson of this course.
• RG3. In the monitored periods the base class ratio is 0.5342; no accuracy is read without this ratio being written next to it. There are 200 rows per period, so the smallest step of a period’s accuracy is 0.0050.
• RG4. The production data has a data drift: after 2026-09 commercial consumption grows by 0.06 per period. The rule has not changed.
• RG5. The release window is a single period: 2026-12, 200 requests. In the canary, the share of requests counted from the start goes to the new model; this is a fictional split.
• RG6. In the shadow placement the new model scores the same request in parallel with the old model, the result is only recorded, and no response returns to the user.
• RG7. Cost is counted in requests and rows; raw time is not written.

Two Candidates the Training Cannot Separate

The first block sets up the source, the consumption stage, the feature path and the fictional model; then it retrains on the drifted data in production and takes the first two ranks of the search as release candidates.

# FICTIONAL. The meter readings of the measurement network, the consumption stage of the
# pipeline, the feature path and the three-weight threshold model are generated inside
# the lesson; no learning library, deployment tool, experiment platform or monitoring
# product is called.
# Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def rows():
    """Source readings and consumption from the period difference: the first three stages of the pipeline."""
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e})
    t = {(k["subscriber"], k["period"]): k["index"] for k in records}
    return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])])
            for k in records if PERIODS.index(k["period"])]


def features(records):
    """Independent of the weights and computed ONCE; it sees the full history."""
    sub, reg = {}, {}
    for k in sorted(records, key=lambda k: PERIODS.index(k["period"])):
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        s = [v for d, v in series if d < i]
        return round(sum(s) / len(s), 4) if s else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6   # the RULE that generates the label , in other words the concept
MONITORED = PERIODS[6:]                   # six periods monitored in production
GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)      # the decision threshold is A SEPARATE setting


def label(o, threshold=RULE_THRESHOLD):
    return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def predict(o, w, decision):
    return 1 if score(o, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision):
    """Training criterion: the mean of recall and specificity."""
    tp = tn = fp = fn = 0
    for k in subset:
        o = feats[(k["subscriber"], k["period"])]
        p, g = predict(o, w, decision), label(o)
        if p and g:
            tp += 1
        elif p:
            fp += 1
        elif g:
            fn += 1
        else:
            tn += 1
    recall = tp / (tp + fn) if tp + fn else 0.0
    specificity = tn / (tn + fp) if tn + fp else 0.0
    return (recall + specificity) / 2, round(recall, 4), round(specificity, 4)


def train(records, feats, periods):
    """Weights and decision threshold are searched together: 27 x 5 = 135 candidates."""
    subset = [k for k in records if k["period"] in periods]
    ranked = sorted(((balanced_accuracy(subset, feats, w, dc)[0], w, dc)
                     for w in GRID for dc in DECISION_GRID), key=lambda a: -a[0])
    best = ranked[0]
    return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4),
            "ranked": ranked, "rows": len(subset)}


def drift(records, start="2026-09", slope=0.06):
    """FICTIONAL data drift: commercial consumption grows by `slope` every period."""
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


ALL = rows()
FEATS = features(ALL)
BASELINE = train(ALL, FEATS, PERIODS[1:9])
PRODUCTION = drift(ALL)                   # the input has drifted in production
FEATS_D = features(PRODUCTION)
bal, recall, specificity = balanced_accuracy([k for k in ALL if k["period"] in PERIODS[1:9]], FEATS,
                                             BASELINE["weights"], BASELINE["decision"])
monitored_rows = [k for k in ALL if k["period"] in MONITORED]
labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows]
print(f"baseline model {BASELINE['weights']} decision threshold {BASELINE['decision']}"
      f" balanced accuracy {BASELINE['balanced']} recall {recall:.4f} specificity {specificity:.4f}")
print(f"monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(labels) / len(labels), 4)}"
      f" , 200 rows per period , accuracy step 0.0050")

# Two candidates: the retrained model and the second rank of the same search.
RETRAINED = train(PRODUCTION, FEATS_D, PERIODS[5:11])
A = {"name": "A retrained", "weights": RETRAINED["weights"], "decision": RETRAINED["decision"],
     "train": RETRAINED["balanced"]}
second = RETRAINED["ranked"][1]
B = {"name": "B second of the search", "weights": second[1], "decision": second[2],
     "train": round(second[0], 4)}
last = [k for k in PRODUCTION if k["period"] == "2026-12"]
for m in (A, B):
    m["disagreement"] = sum(1 for k in last
                            if predict(FEATS_D[(k["subscriber"], k["period"])], m["weights"], m["decision"])
                            != predict(FEATS_D[(k["subscriber"], k["period"])], BASELINE["weights"],
                                       BASELINE["decision"]))
    print(f"candidate {m['name']} {m['weights']}/{m['decision']}"
          f" training balanced accuracy {m['train']} , prediction disagreement {m['disagreement']} on 2026-12")

baseline model (0.6, 0.0, 4) decision threshold 18 balanced accuracy 0.9864 recall 1.0000 specificity 0.9728
monitoring 1200 rows , base class ratio 0.5342 , 200 rows per period , accuracy step 0.0050
candidate A retrained (0.6, 0.0, 4)/18 training balanced accuracy 0.98 , prediction disagreement 0 on 2026-12
candidate B second of the search (0.8, 0.4, 4)/34 training balanced accuracy 0.9741 , prediction disagreement 11 on 2026-12

The two candidates stand at 0.9800 and 0.9741 in training. The 0.0059 between them is a difference of about seven rows in the 1200 rows of the six-period training window; no training report can rank these two with confidence. Their situations in production, however, are different: A carries exactly the same weights and the same decision threshold as the old model, so its prediction disagreement on 2026-12 is 0 and releasing it cannot change anything. B, on the other hand, departs from the old model on 11 predictions in the same period. The whole subject of the release decision is these 11 predictions.

What the Share Does to the Measure

The second block runs the three placements on the same 200 requests. The sent column is the number of requests sent to the new model, served is the accuracy of the responses returned to the user, measured rows is the number of rows on which a measure of the candidate can be collected, and extra wrong is the number of requests the old model got right but the new model served wrongly.

# This block continues the BASELINE, PRODUCTION, FEATS_D, A, B, last, predict,
# label and score definitions of the first block.
def release(old, new, feats, subset, share, shadow=False):
    """Shadow: no request goes to the new model , the measurement is still taken."""
    n = 0 if shadow else int(len(subset) * share)
    measured = len(subset) if shadow else n
    served = cand = extra_wrong = 0
    for j, k in enumerate(subset):
        o = feats[(k["subscriber"], k["period"])]
        g = label(o)
        p_old = predict(o, old["weights"], old["decision"])
        p_new = predict(o, new["weights"], new["decision"])
        served += ((p_new if j < n else p_old) == g)
        extra_wrong += (j < n and p_new != g and p_old == g)
        cand += ((shadow or j < n) and p_new == g)
    return (n, round(served / len(subset), 4), measured, round(cand / measured, 4),
            round(1 / measured, 4), extra_wrong)


old_accuracy = sum(1 for k in last
                   if predict(FEATS_D[(k["subscriber"], k["period"])], BASELINE["weights"],
                              BASELINE["decision"]) == label(FEATS_D[(k["subscriber"], k["period"])]))
print(f"2026-12 requests {len(last)} , old model accuracy {old_accuracy / len(last)}")
print("cand  placement      sent         served  measured rows      measure  resolution  extra wrong")
for m in (A, B):
    for name, share, g in (("shadow", 0.0, True), ("canary 0.10", 0.10, False),
                           ("canary 0.50", 0.50, False), ("full rollout", 1.00, False)):
        n, h, ms, cm, rs, ew = release(BASELINE, m, FEATS_D, last, share, g)
        print(f"{m['name'][0]}     {name:<12}  {n:>5}  {h:>13}  {ms:>13}  {cm:>11}"
              f"  {rs:>10}  {ew:>11}")

2026-12 requests 200 , old model accuracy 0.98
cand  placement      sent         served  measured rows      measure  resolution  extra wrong
A     shadow            0           0.98            200         0.98       0.005            0
A     canary 0.10      20           0.98             20          1.0        0.05            0
A     canary 0.50     100           0.98            100         0.99        0.01            0
A     full rollout    200           0.98            200         0.98       0.005            0
B     shadow            0           0.98            200        0.965       0.005            0
B     canary 0.10      20           0.98             20          1.0        0.05            0
B     canary 0.50     100           0.97            100         0.97        0.01            3
B     full rollout    200          0.965            200        0.965       0.005            7

The A rows are the same in all four placements: served accuracy 0.9800, extra wrong 0. This is not a success but a measure of an emptiness. For a candidate whose prediction disagreement is zero, a canary release means staging a change that can change nothing; the risk carried is zero, and the information gained is zero as well. Even the 1.0000 in A’s canary 0.10 row is not information: it is a number measured on twenty rows, and the old model’s value on the same twenty rows is also 1.0000.

The B rows carry the real reading. In the shadow placement 0 requests go to the new model, the served accuracy stays at the old model’s 0.9800, no user sees the output of the new model, and the extra wrong is 0. Even so, the candidate is measured on all 200 rows and its real value is read: 0.9650, that is 0.0150 worse than the old model. This difference is three steps of the period resolution and lies above the band.

When the canary share is 0.10 the table turns around. 20 requests go to the new model, but now the candidate can be measured only on those 20 rows: the resolution of the measure goes from 0.0050 to 0.0500, ten times coarser. On those twenty rows B’s measure is 1.0000. A candidate whose real value is 0.9650 looks flawless at the most common canary share. The served accuracy also stays at 0.9800, so someone looking at the release dashboard sees that nothing has changed. The trace of the bad candidate has dropped below the measure.

When the share is raised to 0.50 the measure begins to recover — resolution 0.0100, the candidate’s measure 0.9700 — but its cost becomes visible too: 3 requests that the old model got right are answered wrongly. A share of 1.00 is not a canary but a full rollout; the measure returns to its full resolution and the extra wrong goes up to 7. The rollback of a canary is pulling the share down to 0, and this does not bring back the requests that were served wrongly. The shadow has nothing to roll back, because no request ever went.

From this comes a single rule with two sides. In a canary the resolution of the measure and the risk carried depend on the same number, the share, and move in opposite directions: reducing the share reduces the risk and also blinds the measure. The shadow breaks this link — its resolution is the same as that of the full rollout (0.0050) and the requests going to the new model are 0. The price the shadow pays is in another column and is not in this table: every request is scored twice, so 400 inferences are made for 200 requests. The only thing the shadow cannot measure is the behavior caused by the response itself; since nobody sees the score, nobody influences the result either.

Summary

• The same training search leaves two candidates at 0.9800 and 0.9741 and cannot separate them; in production one is a copy whose prediction disagreement with the old model is 0, and the other is a real change that departs on 11 predictions on 2026-12. The base class ratio is 0.5342 and the step of a period’s accuracy is 0.0050.
• The shadow placement sends 0 requests to the new model and still measures the candidate on all 200 rows: its real value is 0.9650, 0.0150 below the old model and as much as three steps of the resolution.
• The canary share divides the measure as well. At a share of 0.10 the candidate is measured on only 20 rows, the resolution goes up to 0.0500, and the candidate that is in reality bad looks like 1.0000; since the served accuracy stays at 0.9800, someone looking at the dashboard sees nothing.
• Raising the share fixes the measure but writes its cost: at 0.50 3 requests and at 1.00 7 requests that the old model got right are answered wrongly. A share of 1.00 is not a canary but a full rollout, and the rollback does not bring these requests back.
• In a canary the resolution and the risk depend on the same share and go in opposite directions; the shadow breaks this link and pays its cost in the number of inferences: 400 scorings for 200 requests.

Next Step

The shadow read the candidate’s real value in a single period as 0.9650 and found it 0.0150 below the old model’s 0.9800. This number looks sufficient for a decision, but an unanswered question sits underneath it: how small can a difference measured with 200 rows become before it can no longer be read? The next lesson sweeps the comparison of two models over the number of periods, counts within which band the difference of the same candidate moves from period to period, and calculates the smallest difference this set size can distinguish. Every comparison that remains below the number found is inconclusive, and the lesson shows an example of this together with its mark.