Two-Environment Deployment

The previous measurement leaned on one assumption: rollback is instant and free. The moment the gate said “roll back,” nothing happened in the model. This lesson lifts that assumption. Rollback has a round, a price, and a limit — and most of that price gets paid even when rollback never turns out to be needed.

Two-environment deployment’s promise fits one sentence: the new version stands up beside the old one, traffic switches over with a single flip, and if something goes wrong, the switch flips back. What that promise costs is that two full environments doing the same work stand together for a while. In this lesson the gate is the rollback decision: how many rounds the decision takes, under what condition it drops to a single round, and what that single round fails to cover are measured.

• RS19. The fictional regional measurement network’s work-order service runs with sixteen copies; sixty releases are measured, and one deployment round is three minutes. Both the network and the figures are fictional.
• RS20. Thirty percent of releases are defective. The defect’s time-to-appear is exponentially distributed, with a mean of twenty-five minutes. Which metric surfaces the defect is a separate measure’s subject; only the moment of appearance is modeled here.
• RS21. The second environment stands up twelve minutes before the switch flips, and stays up for the rollback window. Once the window closes, rollback means a fresh release: five rounds.
• RS22. The staged scheme climbs five steps in fifty minutes with four extra copies; rollback takes as many rounds as the steps climbed by then.
• RS23. At the moment of cutover there are three classes of work unit, and their durations are exponentially distributed. Session affinity carrying over to the new environment is not modeled; what gets measured is how many are cut off when it does not carry over.
• RS24. The data layer is shared between the two environments. The write rate is 140 rows per minute; 35% of rows are updated after rollback. Migration ordering and backward-compatible writes are not this lesson’s subject.

Measurement Setup

What is measured is not a real cluster but a model set up with node. The model carries three separate things: a release sequence’s resource cost and rollback rounds, the work units in flight at the moment the switch flips, and the rows written to the shared data layer. The random generator is written by hand, and its seed is visible.

// twoenv.mjs — cost and cutover model of two-environment deployment; not a real cluster, a model
export const SEED = 20260904;
export const COPIES = 16, RELEASES = 60, ROUND_MIN = 3, PREP_MIN = 12, REDEPLOY_ROUNDS = 5;
export const STAGED_EXTRA = 4, STAGED_DURATION = 50, STAGED_STEPS = 5;
export const DEFECT_MEAN = 25, WINDOW_MIN = 30;    // the defect's time-to-appear is exponentially distributed

export function generator(seed) {                  // linear congruential generator, seed visible
  let x = seed >>> 0;
  return () => { x = (Math.imul(x, 1664525) + 1013904223) >>> 0; return x / 4294967296; };
}
export const exponential = (rnd, mean) => -mean * Math.log(1 - rnd());

// Release sequence: whether each release is defective, and in which minute the defect appears.
export function releases(rnd, brokenRate = 0.30) {
  return Array.from({ length: RELEASES }, (_, i) => ({
    no: i + 1, broken: rnd() < brokenRate, appear: exponential(rnd, DEFECT_MEAN),
  }));
}

// Two-environment scheme: the old environment is kept up for the window after the switch is flipped.
export function twoEnv(list, window) {
  let replicaMin = 0, round = 0, inWindow = 0, outOfWindow = 0;
  for (const y of list) {
    replicaMin += COPIES * (PREP_MIN + window);    // how long the second environment stays up
    if (!y.broken) continue;
    if (y.appear <= window) { round += 1; inWindow += 1; }
    else { round += REDEPLOY_ROUNDS; outOfWindow += 1; }
  }
  const broken = list.filter((y) => y.broken).length;
  return { name: `two-environment (window ${window} min)`, replicaMin, inWindow, outOfWindow,
    avgRound: broken ? round / broken : 0, rollbackMin: broken ? round * ROUND_MIN / broken : 0 };
}

// Staged scheme: the extra copies stay up for the rollout's duration; rollback takes as many rounds as were climbed by then.
export function staged(list) {
  let replicaMin = 0, round = 0;
  for (const y of list) {
    replicaMin += STAGED_EXTRA * STAGED_DURATION;
    if (!y.broken) continue;
    const step = Math.min(STAGED_STEPS, 1 + Math.floor(y.appear / (STAGED_DURATION / STAGED_STEPS)));
    round += step;
  }
  const broken = list.filter((y) => y.broken).length;
  return { name: 'staged (single environment)', replicaMin, inWindow: 0, outOfWindow: broken,
    avgRound: broken ? round / broken : 0, rollbackMin: broken ? round * ROUND_MIN / broken : 0 };
}

// The moment of cutover: units in flight when the switch is flipped. Three classes, exponential duration.
export const CLASSES = [
  { name: 'short request', rate: 300, mean: 0.04 },  // rate: units starting per minute
  { name: 'session', rate: 8, mean: 12 },
  { name: 'long-running job', rate: 0.5, mean: 25 },
];

export function inFlight(rnd, cls, T = 240) {       // T minutes of warm-up, switch flips at T
  const open = [];
  for (let min = 0; min < T; min += 1) {
    for (let k = 0; k < cls.rate; k += 1) {
      const start = min + rnd(), duration = exponential(rnd, cls.mean);
      if (start + duration > T) open.push({ start, end: start + duration });
    }
  }
  return open;
}

// Shared data layer: the new version writes in the new format, the rollback code reverts, it does not revert rows.
export function dataLayer(rnd, openMin, writeRate = 140, updateRate = 0.35) {
  let newFormat = 0, laterUpdated = 0;
  for (let min = 0; min < openMin; min += 1) {
    for (let k = 0; k < writeRate; k += 1) {
      newFormat += 1;
      if (rnd() < updateRate) laterUpdated += 1;    // updated by the old version after rollback
    }
  }
  return { newFormat, lost: laterUpdated, silent: newFormat - laterUpdated };
}

The model’s setting is window: how many minutes the old environment stays up after the switch flips. This single number sets both rollback’s round cost and the scheme’s standing bill — and it sets them in opposite directions.

Resource Cost and Rollback Round

// run1.mjs — the resource cost and rollback round cost of two schemes; the rollback window is scanned
import { generator, releases, twoEnv, staged, SEED, COPIES, RELEASES, ROUND_MIN, WINDOW_MIN, DEFECT_MEAN } from './twoenv.mjs';

const l = releases(generator(SEED));
const broken = l.filter((y) => y.broken).length;
console.log(`seed ${SEED} | ${RELEASES} releases, ${COPIES} copies, ${broken} releases defective`);
console.log(`the defect's time-to-appear is exponential, mean ${DEFECT_MEAN} min | one round ${ROUND_MIN} min\n`);
console.log('scheme                             replica-min  in window  out of window  avg round  avg rollback');
for (const s of [twoEnv(l, WINDOW_MIN), staged(l)]) {
  console.log(`${s.name.padEnd(34)} ${String(s.replicaMin).padEnd(12)} ${String(s.inWindow).padEnd(10)} ` +
    `${String(s.outOfWindow).padEnd(14)} ${s.avgRound.toFixed(2).padEnd(10)} ${s.rollbackMin.toFixed(1)} min`);
}
console.log('\nthe rollback window is scanned (two-environment scheme):');
console.log('window   replica-min  single switch  redeploy  avg round  avg rollback   extra replica-min');
for (const p of [0, 5, 15, 30, 60, 120, 240]) {
  const s = twoEnv(l, p), a = staged(l);
  console.log(`${`${p} min`.padEnd(8)} ${String(s.replicaMin).padEnd(12)} ${String(s.inWindow).padEnd(14)} ` +
    `${String(s.outOfWindow).padEnd(9)} ${s.avgRound.toFixed(2).padEnd(10)} ${`${s.rollbackMin.toFixed(1)} min`.padEnd(14)} ${s.replicaMin - a.replicaMin}`);
}

seed 20260904 | 60 releases, 16 copies, 18 releases defective
the defect's time-to-appear is exponential, mean 25 min | one round 3 min

scheme                             replica-min  in window  out of window  avg round  avg rollback
two-environment (window 30 min)    40320        11         7              2.56       7.7 min
staged (single environment)        12000        0          18             2.78       8.3 min

the rollback window is scanned (two-environment scheme):
window   replica-min  single switch  redeploy  avg round  avg rollback   extra replica-min
0 min    11520        0              18        5.00       15.0 min       -480
5 min    16320        3              15        4.33       13.0 min       4320
15 min   25920        8              10        3.22       9.7 min        13920
30 min   40320        11             7         2.56       7.7 min        28320
60 min   69120        17             1         1.22       3.7 min        57120
120 min  126720       18             0         1.00       3.0 min        114720
240 min  241920       18             0         1.00       3.0 min        229920

The top table puts a limit on the promise. At a thirty-minute window, only eleven of the eighteen defective releases were rolled back with a single switch; seven were noticed after the window closed and needed a five-round redeploy. The average rollback round is 2.56, and 2.78 in the staged scheme. The difference between the two schemes at this setting is eight percent. In exchange, the two-environment scheme burned three times the resources: 40,320 replica-minutes against 12,000.

The phrase “instant rollback” becomes a number once the window is kept longer than the defect’s time-to-appear. The scan shows the price. At a sixty-minute window, seventeen of the eighteen defects were rolled back in a single round, and the average round dropped to 1.22 — an extra 57,120 replica-minutes. At one hundred twenty minutes, every one of them dropped to a single round, at an extra cost of 114,720 replica-minutes. Every round gained carries a price: at the thirty-minute window, forty-four rounds were shaved off, and that cost 655 replica-minutes per round.

The first row says the same thing from the other direction. At a zero-minute window, the two-environment scheme is cheaper than the staged scheme (11,520 against 12,000), because the second environment stands up only for the twelve-minute prep. But in that scheme, not one release could be rolled back with a single switch; the average round climbed to five. Shutting down the old environment shuts down two-environment deployment’s one reason to exist.

Work Cut Off at the Moment of Cutover

// run2.mjs — state-carrying requests cut off at the moment of cutover, and the shared data layer
import { generator, inFlight, dataLayer, CLASSES, SEED, COPIES } from './twoenv.mjs';

const open = CLASSES.map((s) => ({ s, list: inFlight(generator(SEED + s.rate), s) }));
console.log('units in flight at the moment the switch is flipped:');
for (const { s, list } of open) console.log(`  ${s.name.padEnd(18)} ${String(list.length).padStart(5)} units (average duration ${s.mean} min)`);

console.log('\nthe drain duration is scanned: a unit still running at the end of the duration gets cut');
console.log('drain     short request  session  long-running job  total cut  two environments together');
for (const d of [0, 0.5, 1, 5, 15, 30]) {
  const k = open.map(({ list }) => list.filter((u) => u.end > 240 + d).length);
  console.log(`${`${d} min`.padEnd(9)} ${String(k[0]).padEnd(15)} ${String(k[1]).padEnd(8)} ${String(k[2]).padEnd(17)} ` +
    `${String(k[0] + k[1] + k[2]).padEnd(10)} ${2 * COPIES * d} replica-min`);
}

const v = dataLayer(generator(SEED + 11), 11);
console.log('\nshared data layer: the switch was flipped, rollback happened at minute 11');
console.log(`  rows written in the new format: ${v.newFormat}`);
console.log(`  rollback round: 1 (code) | rows reverted for data: 0`);
console.log(`  rows updated by the old version after rollback, dropping the new field: ${v.lost}`);
console.log(`  rows still carrying the new field, invisible to the old version: ${v.silent}`);

units in flight at the moment the switch is flipped:
  short request         10 units (average duration 0.04 min)
  session               89 units (average duration 12 min)
  long-running job      24 units (average duration 25 min)

the drain duration is scanned: a unit still running at the end of the duration gets cut
drain     short request  session  long-running job  total cut  two environments together
0 min     10              89       24                123        0 replica-min
0.5 min   0               88       24                112        16 replica-min
1 min     0               86       24                110        32 replica-min
5 min     0               68       20                88         160 replica-min
15 min    0               31       14                45         480 replica-min
30 min    0               7        5                 12         960 replica-min

shared data layer: the switch was flipped, rollback happened at minute 11
  rows written in the new format: 1540
  rollback round: 1 (code) | rows reverted for data: 0
  rows updated by the old version after rollback, dropping the new field: 491
  rows still carrying the new field, invisible to the old version: 1049

At the moment the switch flipped, 123 work units were in flight. Ten of them were short requests, eighty-nine sessions, twenty-four long-running jobs. A half-minute drain saved every short request — and only one session. A one-minute drain brought the eighty-nine down to eighty-six, and touched none of the long-running jobs.

The reason is a difference in scale. The drain duration is tuned to the short request’s lifetime; a session lives twelve minutes, a long-running job twenty-five — two to three orders of magnitude longer. Bringing 123 units down to twelve needs a thirty-minute drain, and for those thirty minutes, two versions write to the same data layer together. The cost of a seamless cutover is not just 960 replica-minutes; it is a half hour during which two versions write at once.

One more calculation is needed. If every cutover cuts off eighty-nine sessions, then wrongfully rolling back a clean version means two cutovers: 178 sessions and forty-eight long-running jobs. The cost of a wrongful rollback can outweigh the cost of the defect itself — and how often a wrongful rollback happens depends not on the rollback decision, but on the setting of the gate that came before it.

What Rollback Does Not Undo

The last block shows the single switch’s limit. The switch flipped, the defect was seen at minute eleven, rollback completed in one round. The code returned to the old version. In those eleven minutes, 1,540 rows had been written to the shared data layer in the new format; rollback undid zero of them.

There are two separate kinds of damage. 491 rows were updated by the old version after rollback; because the old version does not recognize the new field, that field dropped during the write and the information was lost for good. The remaining 1,049 rows keep carrying their new field; the old version does not see them, so they stay silent — but once the new version comes back, there will be a format split between these two sets.

Two-environment deployment reduces rolling back the code to a single round. When the data layer is shared, rollback’s scope is limited to code, and that limit never shows up in the round cost: a one-round rollback can leave behind a 1,540-row residue.

The Gate’s Three Numbers

What it catches: all eighteen defective releases were eventually rolled back. What only it catches: eleven releases rolled back in a single round at the thirty-minute window — possible only while the old environment is still standing — for a total of forty-four rounds saved, or 132 minutes. The marginal value’s price is 655 replica-minutes per round.

What it misses: the seven releases noticed after the window closed, and the rows written to the data layer during every rollback — 1,540 rows in the example release, 491 of them a permanent loss. What it wrongly holds: rolling back a clean version means two cutovers, 178 sessions, and forty-eight long-running jobs. The time it adds: twelve minutes of prep per release plus the window duration; up to thirty minutes of drain for a seamless cutover; 40,320 replica-minutes in total.

Summary

• At a thirty-minute window, eleven of the eighteen defective releases were rolled back with a single switch, seven with a five-round redeploy; the average round is 2.56, and 2.78 in the staged scheme.
• The resource cost is threefold: 40,320 replica-minutes against 12,000. Raising the window to one hundred twenty minutes drops every release to a single round, at an extra cost of 114,720 replica-minutes.
• At a zero-minute window, the two-environment scheme comes out cheaper than the staged one, but no release remains rollback-able with a single switch: shutting down the old environment shuts down the pattern’s reason to exist.
• At the moment of cutover, 123 work units were in flight. A half-minute drain saved every short request, and only one of eighty-nine sessions; getting down to twelve means a thirty-minute drain and two versions writing together for half an hour.
• In the shared data layer, a one-round rollback undid zero of 1,540 rows; 491 of them were updated by the old version and lost for good.
• Wrongfully rolling back a clean version means two cutovers, 178 sessions, and forty-eight interrupted long-running jobs; this number depends not on the rollback gate, but on the setting of the gate before it.

Next Step

Every measurement in this lesson rested on a single necessity: turning on new behavior requires putting the new version into traffic, and turning it off requires pulling it out. As long as releasing and enabling behavior are the same operation, rollback is always a deployment operation — it has a round, a cutover moment, and a data residue. These two can be separated. If behavior is carried in the code and kept off with a record, turning it on and off is no longer a deployment but a setting change; rollback’s round cost approaches zero. The next lesson measures this separation: the rounds it saves, the enabling decision’s own gate, and the debt every record left off accumulates over time.