Gradual Rollout and Rollback

A change tested in preview reaches every user at once when it goes to production. So does every flaw preview could not catch: load behaviors that show up only under real traffic, edge cases found only in real data, failures seen only on certain devices.

This lesson takes release out of being a single switch flip. A small share of traffic is routed to the new version, the error rate is watched against a threshold, and the previous deployment is returned to once the threshold is crossed. Its question is this: by what measures are the share, the threshold, and the rollback time chosen?

The Four Parts of a Staged Rollout

Rollout plan. The share of traffic routed to the new version increases in steps, with observation time left between them. Each step is a decision point: continue, or revert.

Sticky assignment. Which version a user sees is set by a bucket derived from their identity, and it does not change for the rest of the rollout. If assignment were done per request, the same user would flip between the two versions; that has three concrete consequences — the hashed names the document carries start requesting the other version’s chunks, flows left mid-session break, and measurement gets noisy. The bucket count sets the share’s resolution: a hundred buckets means one-percent steps.

Observation window and threshold. The decision looks at the error rate of requests that land on the new version in a recent window. The window must reach at least one sample-size floor; a decision made before that is noise.

Rollback. Zeroing the share once the threshold is crossed. Its precondition is that the deployment to return to is still live in production — the protected-deployments rule from the Preview Deployments lesson does its work here.

Running the Same Flaw Through Four Rollout Forms

// staged-rollout.mjs -- percentage-share rollout, sticky assignment, threshold-based rollback.
// A virtual clock is used; Date.now() is never called, so the result is the same on every run.
import { createHash } from "node:crypto";

// Deterministic generator (splitmix32). The seed is fixed, so the sequence is the same on every run.
function generator(seed) {
  let s = seed >>> 0;
  return () => {
    s = (s + 0x9e3779b9) >>> 0;
    let z = s;
    z = Math.imul(z ^ (z >>> 16), 0x21f0aaad) >>> 0;
    z = Math.imul(z ^ (z >>> 15), 0x735a2d97) >>> 0;
    return ((z ^ (z >>> 15)) >>> 0) / 4294967296;
  };
}

const DURATION = 3600;     // s
const RATE = 40;           // req/s
const USERS = 5000;
const BASE_ERROR = 0.004;  // old version's baseline error rate
const PLAN = [[0, 1], [300, 5], [600, 25], [1200, 50], [1800, 100]];
const WINDOW = 120;        // s, observation window
const MIN_SAMPLE = 200;    // minimum new-version requests required in the window
const THRESHOLD = 0.02;    // window error-rate threshold
const STEP = 30;           // s, evaluation interval

// The threshold must be cleared past sampling noise: the lower bound of the rate is checked.
const lowerBound = (failed, n) => {
  const p = failed / n;
  return p - 2 * Math.sqrt((p * (1 - p)) / n);
};

// Sticky assignment: a user identity is pinned to a bucket between 0 and 99.
// The same user stays on the same side throughout the rollout.
const bucket = (user) =>
  createHash("sha256").update("user-" + user).digest().readUInt32BE(0) % 100;

const BUCKET = Array.from({ length: USERS }, (_, i) => bucket(i));
const shareAt = (t) => PLAN.filter(([at]) => at <= t).at(-1)[1];

function rollout({ newError, allAtOnce, manual }) {
  // Two separate streams: user selection and the error draw must not affect each other.
  const rUser = generator(20_260_412);
  const rError = generator(78_345_921);
  const events = [];   // { t, isNew, hasError, user }
  let rollback = null, nextEvaluation = STEP;
  const log = [];
  let lastShare = null;

  for (let n = 0; n < DURATION * RATE; n++) {
    const t = n / RATE;
    if (manual !== undefined && rollback === null && t >= manual) {
      rollback = manual;
      log.push([rollback, "detected manually and rolled back"]);
    }
    while (t >= nextEvaluation) {
      if (rollback === null && manual === undefined) {
        const windowEvents = events.filter((o) => o.isNew && o.t > nextEvaluation - WINDOW);
        const failed = windowEvents.filter((o) => o.hasError).length;
        const n = windowEvents.length;
        if (n >= MIN_SAMPLE && lowerBound(failed, n) > THRESHOLD) {
          rollback = nextEvaluation;
          log.push([rollback, "rolled back, window error rate " +
            ((100 * failed) / n).toFixed(1) + "% (" + failed + "/" + n +
            "), lower bound " + (100 * lowerBound(failed, n)).toFixed(1) + "%"]);
        }
      }
      nextEvaluation += STEP;
    }
    const share = rollback !== null ? 0 : allAtOnce ? 100 : shareAt(t);
    if (share !== lastShare && rollback === null) {
      log.push([t, "share " + share + "%"]);
      lastShare = share;
    }
    const user = Math.floor(rUser() * USERS);
    const isNew = BUCKET[user] < share;
    const hasError = rError() < (isNew ? newError : BASE_ERROR);
    events.push({ t, isNew, hasError, user });
  }

  const newEvents = events.filter((o) => o.isNew);
  const failed = newEvents.filter((o) => o.hasError);
  return {
    rollback,
    newRequests: newEvents.length,
    errors: failed.length,
    users: new Set(failed.map((o) => o.user)).size,
    log,
  };
}

console.log("load: " + (DURATION * RATE).toLocaleString("en-US") + " requests / " + DURATION +
  " s, " + USERS.toLocaleString("en-US") + " users, sticky assignment");
console.log("plan: " + PLAN.map(([t, p]) => t + " s " + p + "%").join(", "));
console.log("observation: " + WINDOW + " s window, at least " + MIN_SAMPLE +
  " samples, threshold " + (100 * THRESHOLD).toFixed(1) + "%, evaluated every " + STEP + " s");
console.log("old version's baseline error rate: " + (100 * BASE_ERROR).toFixed(1) + "%");

for (const [name, opts] of [
  ["healthy new version (error rate 0.5%)", { newError: 0.005 }],
  ["flawed new version (error rate 6.0%)", { newError: 0.06 }],
]) {
  const s = rollout(opts);
  console.log("\nscenario: " + name);
  for (const [t, message] of s.log)
    console.log("  " + (t.toFixed(0) + " s").padStart(8) + "  " + message);
  console.log("  result: " + (s.rollback === null ? "rollout completed" : "rolled back") +
    ", " + s.newRequests.toLocaleString("en-US") + " requests to the new version, " +
    s.errors + " errors, " + s.users + " distinct users affected");
}

console.log("\nsame flaw (6.0%), four rollout and detection forms:");
console.log("  " + "form".padEnd(34) + "detected".padStart(9) + "errors".padStart(8) +
  "affected users".padStart(21));
for (const [name, opts] of [
  ["staged + automatic monitoring", { newError: 0.06 }],
  ["all-at-once + automatic monitoring", { newError: 0.06, allAtOnce: true }],
  ["staged + manual detection (600 s)", { newError: 0.06, manual: 600 }],
  ["all-at-once + manual detection (600 s)", { newError: 0.06, allAtOnce: true, manual: 600 }],
]) {
  const s = rollout(opts);
  console.log("  " + name.padEnd(34) + (s.rollback + " s").padStart(9) +
    String(s.errors).padStart(8) +
    (s.users + " (" + ((100 * s.users) / USERS).toFixed(1) + "%)").padStart(21));
}

$ node staged-rollout.mjs
load: 144,000 requests / 3600 s, 5,000 users, sticky assignment
plan: 0 s 1%, 300 s 5%, 600 s 25%, 1200 s 50%, 1800 s 100%
observation: 120 s window, at least 200 samples, threshold 2.0%, evaluated every 30 s
old version's baseline error rate: 0.4%

scenario: healthy new version (error rate 0.5%)
       0 s  share 1%
     300 s  share 5%
     600 s  share 25%
    1200 s  share 50%
    1800 s  share 100%
  result: rollout completed, 91,166 requests to the new version, 471 errors, 444 distinct users affected

scenario: flawed new version (error rate 6.0%)
       0 s  share 1%
     300 s  share 5%
     390 s  rolled back, window error rate 7.4% (15/202), lower bound 3.7%
  result: rolled back, 365 requests to the new version, 22 errors, 21 distinct users affected

same flaw (6.0%), four rollout and detection forms:
  form                               detected  errors       affected users
  staged + automatic monitoring         390 s      22            21 (0.4%)
  all-at-once + automatic monitoring     30 s      67            67 (1.3%)
  staged + manual detection (600 s)     600 s      44            42 (0.8%)
  all-at-once + manual detection (600 s)    600 s    1490         1275 (25.5%)

The output makes four points.

A small share cannot decide. The flawed version was not detected while its share was one percent. The reason is arithmetic: one percent of forty requests per second is 0.4 requests per second; in a hundred-twenty-second window that does not even reach fifty requests, short of the two-hundred minimum-sample floor. Detection happened right after the window filled once the share reached five percent. The first step’s share is set not by risk appetite but by the sample size a decision needs.

The threshold is defined together with noise. The decision looks at the error rate’s lower bound, not its point value: in the flawed version, the rate is 7.4 percent, its lower bound 3.7 percent, and the 2 percent threshold is exceeded by that lower bound. A rule that looked at the point value would also find windows in the healthy version that cross the threshold now and then — seeing five errors in two hundred samples is not unusual for a version whose true rate is 0.5 percent. The lower-bound condition filters out these false rollbacks.

Staged rollout slows detection but shrinks impact. In an all-at-once rollout, the flaw is detected in thirty seconds; staged, in three hundred ninety. The number of affected users, by contrast, differs by about three times under automatic monitoring and about thirty times under manual detection. What staged rollout sells is not speed, it is margin for error.

Without automatic monitoring, staged rollout alone is not enough. The last two rows apply the same ten-minute detection delay to both rollout forms: forty-two users are affected in the staged case, one thousand two hundred seventy-five — a quarter of all users — in the all-at-once case. The rollout plan and monitoring work together; without one, the other falls short.

The Condition for Being Revertible

Rollback is not rebuilding and republishing the old source; that would be a new release and would take as long as a build. For rollback to be short, two conditions are required: the deployment to return to is still live in production, and which one is live is determined by a single pointer.

The pointer’s change must be atomic: it is written to a temporary file, then moved onto the old one with rename(2). This call either happens entirely or not at all; a reader sees either the old value or the new one, never half of it.

// switch.mjs -- atomic switch between version-labeled deployments.
// A temporary pointer file is written first, then moved onto it with rename(2).
import { writeFileSync, renameSync, readFileSync } from "node:fs";

const target = process.argv[2];
const t0 = process.hrtime.bigint();
writeFileSync("release/current.new", target + "\n");
renameSync("release/current.new", "release/current");
const t1 = process.hrtime.bigint();

console.log("switch -> " + target + "   write + rename " +
  (Number(t1 - t0) / 1e6).toFixed(3) + " ms   pointer " +
  readFileSync("release/current", "utf8").trim());

// reader.mjs -- reads continuously while the switch is happening; counts broken or mixed reads.
import { readFileSync } from "node:fs";

const count = new Map();
let broken = 0, inconsistent = 0;
for (let i = 0; i < 120_000; i++) {
  try {
    const version = readFileSync("release/current", "utf8").trim();
    const body = readFileSync("release/version/" + version + "/document.txt", "utf8").trim();
    if (body !== version) inconsistent += 1;
    count.set(version, (count.get(version) ?? 0) + 1);
  } catch {
    broken += 1;
  }
}
console.log("read   : " + [...count].sort().map(([k, v]) => k + "=" + v).join("  ") +
  "   broken=" + broken + "   inconsistent=" + inconsistent);

#!/usr/bin/env bash
# Two version-labeled deployments, atomic switch, and rollback.
rm -rf release
mkdir -p release/version/2026.03 release/version/2026.04
echo "2026.03" > release/version/2026.03/document.txt
echo "2026.04" > release/version/2026.04/document.txt
echo "2026.03" > release/current
echo "start: $(cat release/current)"

node reader.mjs &
reader=$!
sleep 0.3
node switch.mjs 2026.04
wait "$reader"

node reader.mjs &
reader=$!
sleep 0.3
node switch.mjs 2026.03
wait "$reader"

start: 2026.03
switch -> 2026.04   write + rename 0.250 ms   pointer 2026.04
read   : 2026.03=18088  2026.04=101912   broken=0   inconsistent=0
switch -> 2026.03   write + rename 0.292 ms   pointer 2026.03
read   : 2026.03=101743  2026.04=18257   broken=0   inconsistent=0

Read counts depend on the machine and the load; they change on every run. Two fields do not change: no read breaks, and no read comes back with content that disagrees with the pointer. Every one of the hundred twenty thousand reads sees either the old version or the new one.

The switch itself takes under a millisecond. From this it also follows where rollback time comes from: it comes not from the pointer changing but from the change propagating. The invalidation from the Static Hosting lesson neither propagates instantly nor reaches the user’s own cache; giving the document a short freshness window is why it also sets the upper bound on rollback time. Thanks to hashed names, the old version’s assets are still live, so rollback requires no new asset download.

Non-Revertible Changes

Reverting code is not reverting what that code did. Three classes of change fall outside this rule, and staged rollout’s protection does not work for them.

Schema changes. If the new version removed a field, the reverted old version looks for that field and does not find it. The solution is splitting the change into revertible steps: expand-contract. First, the schema is expanded to a shape both versions can run against — the new field is added, the old one stays. Then the code is released and stability is waited for. Only after that, in a separate release, is the old field contracted. Each step has its own revertibility.

Data written in a new format. If the old version cannot read records the new version produces, rollback leaves the data unreadable. The rule is that read support enters one release ahead of the write change: first the version that can read both formats ships, then the version that writes the new format.

Side effects that have already left. A notification sent, a payment call made, or a record created at a third party cannot be reverted. The only path for these is rolling forward: shipping a new version that fixes the flaw. All staged rollout provides here is having kept the number of users the side effect reached small.

Together these three give one design rule: revertibility is a property, it does not come for free. The moment a change becomes non-revertible, staged rollout only shrinks the impact; it does not make the fix easier.

Summary

• Staged rollout has four parts: rollout plan, sticky assignment, observation window and threshold, rollback. The bucket count sets the share’s resolution.
• The first step’s share is set not by risk appetite but by the sample size a decision needs; in the example, a one-percent share could not fill the window and so could not decide.
• The threshold is applied to the error rate’s lower bound, not its point value; a rule that looked at the point value would also roll back healthy versions.
• Staged rollout slows detection but shrinks impact; the gap grows as detection delay grows — forty-two users in the example against one thousand two hundred seventy-five.
• Rollback is returning to a protected deployment through a single pointer; the switch made with rename(2) takes under a millisecond, and what sets the duration is the change propagating.
• Schema changes are made in an expand-contract sequence, and write-format changes ship with read support one release ahead; side effects that have already left are answered only by rolling forward.

Next Step

Every deployment form up to this point shared the same assumption: the output is put at an address, the user opens that address with a browser, and the new version takes effect on the next request. There are targets where this assumption does not hold. The same build output can be placed inside a desktop shell, inside a web view embedded in a native body, or in a package distributed from an app store. Addresses then have to be relative, the update channel goes through a review process, and rollback is counted not in the milliseconds measured in this lesson but in days. The final lesson takes up these targets and the constraints they bring.