Staged Rollout

The promotion gate was a single decision: the artifact either passed or it did not, and when it passed, it opened to everyone at once. Promotion does not have to happen in a single step. A new version can first open to a small percentage of requests, be watched for a while, and then have its share raised. This turns the gate from a single decision into a sequence of decisions.

The rollout mechanics themselves — how the share shifts, how many replicas sit on which version, how many requests an escaped defect touches — are an already-measured subject and are not repeated here; they are this lesson’s input. This lesson’s question sits one step further back: who raises the share, and on what evidence. At the end of every step, a metric is read, compared against a threshold, and one of two decisions is made: raise the percentage, or roll back. That is the gate. What gets measured is this gate’s three numbers — which defect it catches, which clean version it wrongfully stops, and how many minutes it adds to the rollout.

• RS13. The fictional regional measurement network’s billing service receives five hundred requests per minute. Time-of-day variation, regional difference, and request mix are not modeled. Both the network and the figures are fictional.
• RS14. The broken version has passed the promotion gates; its defect is visible only in production traffic. Two severities are measured: a pronounced defect errors on 1.6% of requests, a faint defect on 0.8%. The baseline error rate is 0.4%.
• RS15. The metric is a single item: the new version’s error rate. Latency, saturation, and business metrics, and a multi-metric decision, are outside the model.
• RS16. The window is read once, at the end of the step. A sliding window, a consecutive-violation counter, and metric-aggregation lag are not modeled.
• RS17. Rollback is instant and free. The round cost of rollback is not this lesson’s subject.
• RS18. Every setting is run two hundred times; the rates are the share of these two hundred runs. The seed is visible.

Measurement Setup

What is measured is not a real rollout but a model set up with node. The model climbs a ladder step by step, generates the requests falling in the window one by one at each step, and makes the decision at the end of the step. The random generator is written by hand, and its seed is visible.

// rollout.mjs — model of the decision side of a staged rollout; not real traffic, a model
export const SEED = 20260903;
export const REQ_MIN = 500, BASE = 0.004, PRONOUNCED = 0.016, FAINT = 0.008;
export const WINDOW = 10, MULTIPLIER = 2.5, REPEAT = 200;

export function generator(seed) {                  // linear congruential generator, seed visible
  let x = seed >>> 0;
  return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; };
}

export const ladders = {                            // percentage steps
  fast: [5, 50, 100], balanced: [1, 5, 25, 50, 100], careful: [1, 2, 5, 10, 25, 50, 100],
};

// One rollout: the ladder climbs step by step, the metric window is read at the end of each step.
// The decision gate is here: if the error rate seen in the window exceeds the threshold, roll back; if not, raise the percentage.
export function attempt(rnd, ladder, window, threshold, rate) {
  let touched = 0;
  for (let i = 0; i < ladder.length; i += 1) {
    const n = Math.round(REQ_MIN * window * ladder[i] / 100);
    let h = 0;
    for (let r = 0; r < n; r += 1) if (rnd() < rate) h += 1;
    touched += n;
    if (n > 0 && h / n > threshold) return { decision: 'roll back', step: i + 1, touched };
  }
  return { decision: 'ok', step: ladder.length, touched };
}

export function run(ladder, rate, window = WINDOW, multiplier = MULTIPLIER) {
  const threshold = BASE * multiplier, rnd = generator(SEED);
  let stopped = 0, step = 0, touched = 0;
  const distribution = ladder.map(() => 0);
  for (let k = 0; k < REPEAT; k += 1) {
    const s = attempt(rnd, ladder, window, threshold, rate);
    if (s.decision === 'roll back') { stopped += 1; step += s.step; distribution[s.step - 1] += 1; }
    touched += s.touched;
  }
  return { distribution, rate: stopped / REPEAT, avgStep: stopped ? step / stopped : 0,
    avgTouched: Math.round(touched / REPEAT), duration: ladder.length * window, threshold };
}

The same run is called for three separate versions. For the pronounced- and faint-defect versions, the resulting rate is what the gate catches; called with the baseline rate, that same number is what the gate wrongly holds — because that version has no defect.

Three Ladders

// run1.mjs — three ladders, three versions: pronounced-defect, faint-defect, clean
import { run, ladders, SEED, REQ_MIN, BASE, PRONOUNCED, FAINT, WINDOW, MULTIPLIER, REPEAT } from './rollout.mjs';

console.log(`seed ${SEED} | ${REQ_MIN} req/min, window ${WINDOW} min, ${REPEAT} repeats`);
console.log(`baseline error rate ${BASE * 100}% | pronounced defect ${PRONOUNCED * 100}% | faint defect ${FAINT * 100}%`);
console.log(`decision threshold: ${MULTIPLIER} times baseline, i.e. ${BASE * MULTIPLIER * 100}%\n`);
console.log('ladder     percentage steps       pronounced defect          faint defect      clean');
console.log('                                  stopped step   touched    stopped touched    wrongly stopped');
for (const [label, m] of Object.entries(ladders)) {
  const b = run(m, PRONOUNCED), s = run(m, FAINT), t = run(m, BASE);
  console.log(`${label.padEnd(10)} ${m.join('-').padEnd(23)} ${`${(b.rate * 100).toFixed(0)}%`.padEnd(8)} ` +
    `${b.avgStep.toFixed(2).padEnd(7)} ${String(b.avgTouched).padEnd(10)} ${`${(s.rate * 100).toFixed(0)}%`.padEnd(8)} ` +
    `${String(s.avgTouched).padEnd(11)} ${`${(t.rate * 100).toFixed(0)}%`.padEnd(9)} ${b.duration} min`);
}
console.log('\nrequests falling in the metric window per step, and the smallest observable non-zero rate:');
for (const p of [1, 2, 5, 10, 25, 50, 100]) {
  const n = REQ_MIN * WINDOW * p / 100;
  console.log(`  ${String(p).padStart(3)}% share: ${String(n).padStart(5)} req/window, one failure = ${(100 / n).toFixed(2)}%` +
    ` -> threshold ${(BASE * MULTIPLIER * 100).toFixed(1)}% ${100 / n > BASE * MULTIPLIER * 100 ? 'indistinguishable' : 'distinguishable'}`);
}

seed 20260903 | 500 req/min, window 10 min, 200 repeats
baseline error rate 0.4% | pronounced defect 1.6% | faint defect 0.8%
decision threshold: 2.5 times baseline, i.e. 1%

ladder     percentage steps       pronounced defect          faint defect      clean
                                  stopped step   touched    stopped touched    wrongly stopped
fast       5-50-100                100%     1.23    825        40%      5375        9%        30 min
balanced   1-5-25-50-100           100%     1.60    306        70%      3423        27%       50 min
careful    1-2-5-10-25-50-100      100%     1.72    188        82%      2403        32%       70 min

requests falling in the metric window per step, and the smallest observable non-zero rate:
    1% share:    50 req/window, one failure = 2.00% -> threshold 1.0% indistinguishable
    2% share:   100 req/window, one failure = 1.00% -> threshold 1.0% distinguishable
    5% share:   250 req/window, one failure = 0.40% -> threshold 1.0% distinguishable
   10% share:   500 req/window, one failure = 0.20% -> threshold 1.0% distinguishable
   25% share:  1250 req/window, one failure = 0.08% -> threshold 1.0% distinguishable
   50% share:  2500 req/window, one failure = 0.04% -> threshold 1.0% distinguishable
  100% share:  5000 req/window, one failure = 0.02% -> threshold 1.0% distinguishable

On the pronounced defect, all three ladders scored perfectly: the version was stopped in all two hundred of the two hundred runs. Where they diverge is touched-request count. The fast ladder opened its first step at 5%, so it touched 825 requests; the careful ladder started at 1% and stayed at 188. The cost is duration: thirty minutes against seventy.

The faint-defect column turns the table around. A version with a 0.8% error rate sits below the threshold; the gate catches it only when noise happens to push it over. The fast ladder stopped this version in four of ten runs, the careful ladder in eight. But when it fails to stop it, the touched-request count is 5,375 for the fast ladder and 2,403 for the careful one. On a defect the gate misses, a fine-grained ladder still pays off — because even at the moment of missing it, the share stays small.

The last column writes the bill. The careful ladder stopped a clean version in three of ten runs. Thirty-two percent. Why is answered by the table below: a 1% share, in a ten-minute window, gets fifty requests. The smallest observable non-zero error rate at fifty requests is 2% — twice the decision threshold. At that step, a single baseline failure alone triggers the gate. There, the gate is not measuring the version; it is measuring noise.

Window and Threshold

// run2.mjs — on the balanced ladder, the metric window and the decision threshold are scanned separately
import { run, ladders, BASE, PRONOUNCED, FAINT, WINDOW, MULTIPLIER } from './rollout.mjs';
const m = ladders.balanced;

console.log(`ladder ${m.join('-')} | threshold multiplier ${MULTIPLIER} fixed`);
console.log('window   pronounced stopped  faint stopped  clean wrongly stopped  touched   rollout duration');
for (const p of [5, 10, 20, 40, 80]) {
  const b = run(m, PRONOUNCED, p), s = run(m, FAINT, p), t = run(m, BASE, p);
  console.log(`${`${p} min`.padEnd(9)} ${`${(b.rate * 100).toFixed(0)}%`.padEnd(21)} ${`${(s.rate * 100).toFixed(0)}%`.padEnd(15)} ` +
    `${`${(t.rate * 100).toFixed(0)}%`.padEnd(21)} ${String(b.avgTouched).padEnd(9)} ${b.duration} min`);
}
console.log(`\nwindow ${WINDOW} min fixed | threshold = baseline x multiplier`);
console.log('multiplier  threshold  pronounced stopped  faint stopped  clean wrongly stopped  touched');
for (const c of [1.5, 2, 2.5, 3, 4, 6]) {
  const b = run(m, PRONOUNCED, WINDOW, c), s = run(m, FAINT, WINDOW, c), t = run(m, BASE, WINDOW, c);
  console.log(`${String(c).padEnd(11)} ${`${(BASE * c * 100).toFixed(1)}%`.padEnd(10)} ${`${(b.rate * 100).toFixed(0)}%`.padEnd(21)} ` +
    `${`${(s.rate * 100).toFixed(0)}%`.padEnd(15)} ${`${(t.rate * 100).toFixed(0)}%`.padEnd(21)} ${b.avgTouched}`);
}

const t2 = run(m, BASE), b2 = run(m, PRONOUNCED);
console.log(`\nwindow ${WINDOW} min, threshold multiplier ${MULTIPLIER}: which step the stop happened at`);
console.log(`  percentage step:          ${m.map((x) => `${x}%`.padStart(6)).join('')}`);
console.log(`  clean version stopped:    ${t2.distribution.map((x) => String(x).padStart(6)).join('')}`);
console.log(`  pronounced defect stopped:${b2.distribution.map((x) => String(x).padStart(6)).join('')}`);

ladder 1-5-25-50-100 | threshold multiplier 2.5 fixed
window   pronounced stopped  faint stopped  clean wrongly stopped  touched   rollout duration
5 min     100%                  64%             20%                   341       25 min
10 min    100%                  70%             27%                   306       50 min
20 min    100%                  48%             9%                    563       100 min
40 min    100%                  40%             5%                    770       200 min
80 min    100%                  34%             2%                    890       400 min

window 10 min fixed | threshold = baseline x multiplier
multiplier  threshold  pronounced stopped  faint stopped  clean wrongly stopped  touched
1.5         0.6%       100%                  100%            53%                   236
2           0.8%       100%                  91%             26%                   306
2.5         1.0%       100%                  70%             27%                   306
3           1.2%       100%                  45%             23%                   565
4           1.6%       94%                   37%             19%                   1373
6           2.4%       31%                   7%              2%                    6333

window 10 min, threshold multiplier 2.5: which step the stop happened at
  percentage step:              1%    5%   25%   50%  100%
  clean version stopped:        37    16     0     0     0
  pronounced defect stopped:   100    79    21     0     0

The window scan does not give a straight curve. Going from a five-minute to a ten-minute window, wrongful stopping rises from 20% to 27%, then falls. The reason is in the decision rule: at a 1% share, a smaller window brings fewer requests, but a single failure still crosses the threshold. Growing the window first raises the odds of “seeing one failure”; but once the window grows large enough to require two failures, the curve turns. At twenty minutes, wrongful stopping drops to 9%, at eighty minutes to 2% — in exchange, rollout duration climbs from one hundred minutes to four hundred, and touched requests rise from 563 to 890. The price of waiting is the requests touched while waiting.

The threshold scan shows a clean trade-off. Dropping the multiplier to 1.5 catches all of the faint defect, but stops more than half of clean versions. Raising it to 6 drops wrongful stopping to 2%; but this time even the pronounced defect is caught only a third of the time, and touched requests jump to 6,333. The rows between 2 and 2.5 stand out: wrongful stopping stays almost the same while faint-defect catch rate rises from 70% to 91%. Pulling the threshold from 2.5 down to 2 is a gain with no offsetting cost.

The reason is in the last block. Of the clean version’s fifty-three wrongful stops, thirty-seven happened at the first step, sixteen at the second; past the third step, not one. Small-share steps, not the threshold, produce the wrongful stops. Moving the threshold does not fix those steps, because the number measured there already jumps around independently of the threshold. The pronounced-defect row concentrates in the same place: a hundred runs stopped at the first step, seventy-nine at the second. So the first two steps produce nearly all of both the decisions and the mistakes; the remaining three steps only add duration.

The Gate’s Three Numbers

What it catches: the pronounced-defect version was stopped in all two hundred of the two hundred runs. What only it catches: every one of these defects — by definition, the broken version has already passed the promotion gates, and its defect is visible only in production traffic. The percentage-raising gate is the first gate to turn real traffic into a metric; its marginal value is therefore one hundred percent.

What it misses: 18% to 60% of the faint defect depending on the ladder; even 69% of the pronounced defect once the threshold is pulled to 6. What it wrongly holds: 27% of clean versions on the balanced ladder, 32% on the careful ladder; that drops to 2% at an eighty-minute window. The time it adds: thirty, fifty, and seventy minutes per ladder; up to four hundred minutes once the window is widened.

None of these three numbers can be read off the pattern’s name. The sentence “we did a canary” is empty until it says which metric the gate watches, over what window, at what threshold — and the worst of these three settings is trying to make the decision at a share too small for the decision to be made at all.

Summary

• Three ladders were run on the same defect severities: all three stopped the pronounced defect across two hundred runs, but touched requests dropped from 825 to 188, and duration rose from thirty minutes to seventy.
• For the faint defect, the stop rate ranged from 40% to 82%; even when the gate missed it, the fine-grained ladder cut touched requests from 5,375 to 2,403.
• Wrongful rollback was measured: 27% on the balanced ladder, 32% on the careful one. All fifty-three wrongful stops happened in the first two steps; past the third, not one.
• The window scan is not one-directional: going from five to ten minutes raised wrongful stopping, past twenty it fell; at eighty minutes it reached 2%, at the cost of a four-hundred-minute rollout.
• In the threshold scan, a multiplier of 2 instead of 2.5 raised faint-defect catch rate from 70% to 91% without raising wrongful stopping; raising the threshold further grew missed defects fast.
• A 1% share, in a ten-minute window, gets fifty requests; a single failure reads as 2%, the threshold is 1%. At that step the gate measures noise, not the version — a small share limits blast radius, it does not deliver a decision.

Next Step

This entire measurement leaned on one assumption: rollback is instant and free. The moment the gate said “roll back,” nothing happened in the model. In reality, rollback has a round, a cost, and a limit. If the old version is still standing, rollback is a matter of flipping a single switch; if it is not, rollback means a fresh release. The next lesson measures the scheme that keeps two full environments standing side by side: rollback’s round cost, the resource bill of keeping two environments up continuously, the state-carrying requests cut off at the moment of the switch, and — when the data layer is shared — what rollback actually fails to undo.