Deployment Patterns

Placement and scaling decisions have already been delegated to the system: the scheduler decides which node gets which pod, and the scaling controller decides how many replicas stay up. Both rest on a shared assumption — a working version already exists somewhere. Two decisions remain undelegated. How does a new version enter the cluster, and who checks whether a replica that has come up is actually serving. This lesson measures the first one.

Release mechanics were measured earlier: the request dropped when the pool changes, in-flight requests recovered through connection draining, the difference between hard and graceful shutdown. Those measurements are not repeated here. This lesson’s question is one step further: if not a single request drops during the rollout, do the patterns still diverge from each other? They do — but not at the point of interruption. They diverge at the point where the new version is broken. That every rollout produces some escaped defects is a measured result; this lesson ties that result to pattern choice and runs all three patterns against the same faulty-release sequence.

The delegated decision is this: a controller, not a human, decides when to stop the old version and how much traffic to open to the new one. The controller needs four information items for this: which replica is on which version, whether the replica is ready, what the error rate is, and what to do once the threshold is crossed. If the last item is undefined, the controller keeps advancing — not because it does not know, but because it was never told to stop.

• OO1. The cluster runs the fictional regional measurement network’s verification service with twelve replicas; six hundred requests arrive per round. Both the network and the figures are fictional.
• OO2. The faulty version errors out on six percent of the requests it receives. The defect is present from the first request and stays constant through the version’s lifetime; defects that fade with warm-up are outside this model.
• OO3. The observer checks at the end of the round. Delays shorter than a round, metric-collection lag, and the alert channel’s own latency are not modeled.
• OO4. The alarm threshold is one and a half percent, chosen above the cluster’s ordinary error share; where the threshold comes from is not this lesson’s subject.
• OO5. Over the rollback rounds, the new version’s share declines to zero in equal steps; the rollback itself produces no errors.
• OO6. The horizon is twelve rounds. If the alarm has not fired within that time, the defect is considered to have escaped.

Measurement Setup

What is measured is not a real cluster but a model built with node. The model carries three things: what share a pattern produces round by round, how requests are distributed according to that share, and what the observer sees at the end of the round. The random generator is hand-written, and its seed is visible.

// pattern.mjs — round-based model of three deployment patterns; not a real cluster
export const SEED = 20260801;
export const SERVICE = 12, ROUND_REQUESTS = 600, ERROR_RATE = 0.06, THRESHOLD = 0.015, HORIZON = 12;

export function generator(seed) {                  // linear congruential generator, seed visible
  let x = seed >>> 0;
  return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; };
}

// Each step: replicas serving the new version, replicas serving the old version,
// total replicas kept active at that moment, rollback cost in rounds if the alarm fires this round.
export const rolling = { name: 'rolling update', plan: [
  { newer: 3, older: 9, active: 15, rollback: 1 }, { newer: 6, older: 6, active: 15, rollback: 2 },
  { newer: 9, older: 3, active: 15, rollback: 3 }, { newer: 12, older: 0, active: 12, rollback: 4 } ] };

export const blueGreen = { name: 'blue-green', plan: [
  { newer: 0, older: 12, active: 24, rollback: 0 }, { newer: 12, older: 0, active: 24, rollback: 1 },
  { newer: 12, older: 0, active: 12, rollback: 3 } ] };

export const canary = (k, observe = 3) => ({ name: `canary (sample ${k})`, plan: [
  ...Array.from({ length: observe }, () => ({ newer: k, older: 12, active: 12 + k, rollback: 1 })),
  { newer: 12, older: 0, active: 24, rollback: 1 }, { newer: 12, older: 0, active: 12, rollback: 3 } ] });

export function run({ name, plan }, labeled = false) {
  const rnd = generator(SEED);
  let i = 0, alarm = 0, remaining = 0, share = 0, cost = 0, peak = SERVICE;
  let errors = 0, touched = 0, step = { newer: 0, older: SERVICE, active: SERVICE, rollback: 0 };
  for (let round = 1; round <= HORIZON; round += 1) {
    if (remaining > 0) { step = { ...step, newer: Math.max(0, step.newer - share), older: SERVICE }; remaining -= 1; }
    else if (alarm) step = { newer: 0, older: SERVICE, active: SERVICE, rollback: 0 };
    else { step = plan[Math.min(i, plan.length - 1)]; i += 1; }
    peak = Math.max(peak, step.active);
    let roundErrors = 0, roundTouched = 0;
    const ratio = step.newer / (step.newer + step.older);
    for (let r = 0; r < ROUND_REQUESTS; r += 1) {
      if (rnd() >= ratio) continue;                // request went to the old version
      roundTouched += 1;
      if (rnd() < ERROR_RATE) roundErrors += 1;
    }
    errors += roundErrors; touched += roundTouched;
    if (!alarm) {                                  // the observer checks at the end of the round
      const observed = labeled ? (roundTouched ? roundErrors / roundTouched : 0) : roundErrors / ROUND_REQUESTS;
      const sufficient = !labeled || roundTouched >= 100;
      if (sufficient && observed > THRESHOLD) { alarm = round; cost = step.rollback; remaining = cost; share = Math.ceil(step.newer / Math.max(1, cost)); }
    }
  }
  return { name, alarm, cost, window: alarm ? alarm - 1 : HORIZON, peak, errors, touched };
}

The entire difference between the three patterns’ plans collects into three columns. The rolling update opens the new version in shares of three and keeps three extra replicas active through the transition. Blue-green brings the new version fully up before opening any traffic to it — its share is zero in the first round, with twenty-four replicas active — then switches over in a single step. Canary opens a small sample to traffic, waits a few rounds, and promotes if the alarm does not fire.

Three Patterns, One Faulty Release

// run1.mjs — the same faulty-release sequence is run against all three patterns
import { run, rolling, blueGreen, canary, SEED, SERVICE, ROUND_REQUESTS, ERROR_RATE, THRESHOLD, HORIZON } from './pattern.mjs';

console.log(`seed ${SEED} | ${SERVICE} replicas, ${ROUND_REQUESTS} requests per round, ${HORIZON}-round horizon`);
console.log(`faulty version: ${ERROR_RATE * 100}% of requests to the new version error out`);
console.log(`observer: alarms once the cluster-wide error rate exceeds ${THRESHOLD * 100}%\n`);
console.log('pattern                 alarm round  undetected    faulty req    touched    peak repl  rollback rounds');
for (const k of [rolling, blueGreen, canary(2)]) {
  const s = run(k);
  console.log(`${s.name.padEnd(23)} ${String(s.alarm || '-').padEnd(11)} ${String(s.window).padEnd(13)} ` +
    `${String(s.errors).padEnd(13)} ${String(s.touched).padEnd(10)} ${String(s.peak).padEnd(11)} ${s.cost || '-'}`);
}

seed 20260801 | 12 replicas, 600 requests per round, 12-round horizon
faulty version: 6% of requests to the new version error out
observer: alarms once the cluster-wide error rate exceeds 1.5%

pattern                 alarm round  undetected    faulty req    touched    peak repl  rollback rounds
rolling update          2           1             33            555        15          2
blue-green              2           1             30            600        24          1
canary (sample 2)       4           3             53            851        24          1

All three patterns carried the same defect, and all three detected it, but their costs piled up in different places.

The rolling update gave the new version a one-quarter share in the first round; at that share, the cluster’s total error rate did not clear the threshold, so the alarm fired in the second round. Blue-green gave no share at all in the first round and the full share in the second — the alarm fired in the same round, but in that single round all six hundred requests went to the faulty version. Over the same span, the rolling update touched five hundred fifty-five requests; the difference looks small only because the rolling update was also splitting the load roughly in half by the second round.

The real divergence is in two columns. Peak capacity: the rolling update got through with fifteen replicas, blue-green with twenty-four — meaning the cluster had to find a full extra set of room for the new version at cutover. Rollback cost runs the other way: for blue-green, rollback is a single round, because the old version is still standing; for the rolling update, a defect caught in the second round costs two rounds of rollback, and would have cost four had it been caught in the fourth round. As the rolling update advances, rollback grows more expensive; for blue-green, the expense jumps all at once — in the round the old environment is torn down, rollback cost leaps from one round to three.

The canary row is this table’s surprise. A two-replica sample kept the cluster’s total error rate under the threshold for all three observation rounds: with two of fourteen serving replicas faulty, the total error rate sits around one percent, and the threshold is one and a half. The observer saw nothing, the canary was promoted, and the fourth round opened it fully. The escaped defect touched eight hundred fifty-one requests — the worst of the three patterns. When the sample is chosen wrong, the canary stops being the pattern that catches the defect and becomes the one that delays catching it and then opens it up fully.

Sample Size

Which is why the canary’s one setting has to be swept. The same model is run with two observers: the first watches the cluster’s total error rate, the second splits requests by version and watches only the new version’s error rate. The second one demands an extra information item — which version each request went to — and needs to see enough requests in the round before it can decide.

// run2.mjs — sample size is swept for the canary; the second observer splits requests by version
import { run, canary, ROUND_REQUESTS } from './pattern.mjs';

console.log('sample  cluster-wide observer                     per-version observer');
console.log('repl    alarm  undetected    faulty  touched    alarm  undetected    faulty  touched');
for (const k of [1, 2, 3, 4, 6, 8]) {
  const t = run(canary(k)), l = run(canary(k), true);
  const fmt = (s) => `${String(s.alarm || '-').padEnd(6)} ${String(s.window).padEnd(13)} ` +
    `${String(s.errors).padEnd(7)} ${String(s.touched).padEnd(10)}`;
  console.log(`${String(k).padEnd(6)} ${fmt(t)} ${fmt(l)}`);
}
console.log(`\nrequests reaching the canary during a round of observation: for sample k, ${ROUND_REQUESTS} x k/(12+k)`);
console.log('the per-version observer does not decide without seeing at least 100 requests in the round');
for (const k of [1, 2, 3, 4, 6, 8]) {
  console.log(`  sample ${k}: ${Math.round(ROUND_REQUESTS * k / (12 + k))} req/round` +
    ` -> ${ROUND_REQUESTS * k / (12 + k) >= 100 ? 'can decide' : 'sample too small'}`);
}

sample  cluster-wide observer                     per-version observer
repl    alarm  undetected    faulty  touched    alarm  undetected    faulty  touched
1      4      3             42      730        4      3             42      730       
2      4      3             53      851        4      3             53      851       
3      4      3             55      948        1      0             7       107       
4      3      2             24      461        1      0             8       147       
6      1      0             11      191        1      0             11      191       
8      1      0             14      243        1      0             14      243       

requests reaching the canary during a round of observation: for sample k, 600 x k/(12+k)
the per-version observer does not decide without seeing at least 100 requests in the round
  sample 1: 46 req/round -> sample too small
  sample 2: 86 req/round -> sample too small
  sample 3: 120 req/round -> can decide
  sample 4: 150 req/round -> can decide
  sample 6: 200 req/round -> can decide
  sample 8: 240 req/round -> can decide

Read on its own, the left half leads to the conclusion that the canary is a bad pattern. For samples of one, two, and three replicas, the alarm fired in the fourth round — that is, after promotion; the number of touched requests grew as the sample grew, because the same missed span carried a larger observed share. At four replicas the alarm moved to the third round, at six and eight replicas to the first. The lowest total damage is not at eight but at six: the eight-replica sample caught the defect in the first round too, but that single round still touched two hundred forty-three requests, against one hundred ninety-one for the six-replica sample.

The right half gives the sweep’s real result. The per-version observer fired the alarm in the first round for samples of three and four replicas; the faulty-request count fell to seven and eight. At one and two replicas, the right half stayed identical to the left — because at those sample sizes only forty-six and eighty-six requests per round reach the new version, below the observer’s decision threshold. The small sample’s problem is not only the metric being diluted; it is the smallness of the sample itself. A labeled metric solves the dilution problem; it does not solve the sample problem.

The setting worth reading off this is three replicas: it catches the defect in the first round, touches one hundred seven requests, and the extra capacity it keeps active is thirteen replicas, not fifteen. Its cost is that every request must carry a version label and the observer must keep a separate series for that label — one more information item.

What the Pattern Hides

None of the three patterns is “safer”; all three pile the same three quantities into different places. Blue-green makes rollback cheap and pays for it in capacity: twenty-four replicas, a full extra set of room, sitting idle for the whole cutover window. The rolling update makes capacity cheap and pays for it in rollback: the further it advances, the longer the way back. Canary shrinks the blast radius and pays for it in observation time — nothing advances for three rounds.

What stays silent is that all three assume a threshold has been set correctly. In the model, every round the alarm did not fire, the controller moved on to the plan’s next step; no pattern ever said “I am not sure, let me wait.” The two-replica canary’s promotion was not a malfunction — it was behavior consistent with the plan: the observer saw nothing crossing the threshold, so the controller advanced. This is exactly where the delegated decision’s cost sits — the controller does not know the error rate, it knows the number the observer reported. If that number is diluted, the wrong decision comes not from wrong information but from a missing distinction.

Summary

• All three patterns were run against the same faulty release: the rolling update caught it in the second round with thirty-three faulty requests, blue-green in the second round with thirty, and the two-replica canary in the fourth round with fifty-three.
• Peak capacity and rollback cost run in opposite directions: blue-green needs twenty-four replicas and rolls back in a single round; the rolling update gets through with fifteen replicas, but rollback grows to two, three, or four rounds the further it has advanced.
• For blue-green, the cheap rollback window ends in the round the old environment is torn down; after that round, rollback jumps from one round to three.
• For canary, a small sample misses the defect and lets it get promoted: the two-replica sample touched eight hundred fifty-one requests, the worst of the three patterns.
• The observer’s distinction matters before sample size does: the per-version metric moved the alarm from the fourth round to the first for the three-replica sample, cutting faulty requests from fifty-five to seven; at one and two replicas it changed nothing, because the sample sat below the decision threshold.
• The delegated decision is “advance or stop”; the controller knows the number the observer reported, not the error rate itself, and if that number is diluted, the wrong decision comes from a missing distinction.

Next Step

Throughout this lesson’s model, a replica could serve the moment it entered traffic. In reality, a process spends an interval between coming up and being ready: it builds its connection pool, warms its cache, polls its dependencies. If the controller does not know about this interval, it opens the replica to traffic too early; or, the reverse, it mistakes a slow-starting replica for a failed one and kills it. The next lesson measures this interval with three separate checks: it sweeps the thresholds of the checks that distinguish whether a replica is alive, can take requests, and is still starting up, and counts how many unnecessary restarts and how many faulty requests the wrong setting produces.