Rollback Plan

Turning off a flag takes one round when what needs rolling back is a behavior. If the defect sits in a change that was never flagged, if the configuration also changed, or if data was transformed along with the release, the way back is not a single switch. There are ordered steps, the steps have preconditions, and after some of them the way back closes.

What this lesson measures is not release mechanics but the plan itself. Zero-downtime release schemes and migration-versus-release ordering were measured in M16/K08 Server Security and Going to Production; the question asked here is how many steps and rounds it takes to come back from those schemes. Three scenarios are run, every step’s round is counted, which step is the point of no return is found, and how many rounds an unrehearsed plan costs on its first run is measured.

• RS31. Rollback is a plan: ordered steps, each step with one round and zero or more preconditions. Steps cannot be skipped.
• RS32. A round is the unit of time paid at the moment of the incident; in the model, every round is the same length. A round’s minute equivalent varies by environment; the ratio does not.
• RS33. The config record does not keep versions: an overwritten value cannot be recovered.
• RS34. A rehearsal is running the plan start to finish in an environment outside the incident. It finds permanent gaps; it does not find ones tied to the moment of the incident — on-call, authority, that night’s team.
• RS35. A rollback left halfway leaves a mixed state: some nodes are on the old version, some on the new. Getting out of this state adds a fixed round.
• RS36. The fiction is the regional measurement network’s billing service; recovery is underway from a broken release. The service, and the plan and attempt ledgers below, are all fictional.

The Plan’s Three Scenarios

The same incident produces three separate rollback plans. If only the code changed, the plan is short. If a configuration value also changed along with the release, two steps get added. If the release carried a data migration, three steps get added. The executor below runs the plan step by step, gets stuck on a step whose precondition is not met, and adds the repair round.

// plan.mjs — rollback is modeled as a plan and run across three scenarios; the plan is a MODEL
// Regional measurement network fiction: recovering from the billing service's broken release.

const STATE = {                          // the real state at the moment of the incident
  'previous-artifact-in-storage': true,  // the build artifact inside the retention policy still stands
  'health-endpoint-compatible': true,    // the old version's health endpoint matches the new check
  'pre-migration-backup': true,          // a snapshot backup was taken before the migration
  'config-previous-version': false,      // the config record does not keep versions
  'oncall-authority': false,             // rollback authority sits only with the day team
  'reverse-migration-script': false,     // the migration's reverse step was never written
};
const REPAIR = { 'config-previous-version': 3, 'oncall-authority': 2, 'reverse-migration-script': 6 };

// [name, round, preconditions, is the way back open after this step]
const CODE = [
  ['defect confirmed', 1, ['health-endpoint-compatible'], true],
  ['rollback decision and authority', 1, ['oncall-authority'], true],
  ['previous build artifact selected', 1, ['previous-artifact-in-storage'], true],
  ['nodes reverted to the old artifact', 2, [], true],
  ['health and consistency verified', 1, ['health-endpoint-compatible'], true],
];
const CONFIG = [
  ['config written back to previous value', 2, ['config-previous-version'], false],
  ['config propagated to nodes', 1, [], true],
];
const DATA = [
  ['writes stopped', 1, [], true],
  ['reverse migration runs', 3, ['reverse-migration-script'], false],
  ['writes reopened, counts compared', 2, ['pre-migration-backup'], true],
];
const SCENARIO = {
  'code only': CODE,
  'code + config': [...CODE.slice(0, 4), ...CONFIG, CODE[4]],
  'code + data': [...CODE.slice(0, 4), ...DATA, CODE[4]],
};

function run(plan, rehearsed) {
  const state = { ...STATE };
  if (rehearsed) for (const k of Object.keys(REPAIR)) state[k] = true;  // rehearsal found it first, fixed it first
  let round = 0, extra = 0;
  const stuck = [];
  plan.forEach(([name, t, cond], i) => {
    for (const k of cond) {
      if (!state[k]) { stuck.push([i + 1, name, k, REPAIR[k]]); extra += REPAIR[k]; state[k] = true; }
    }
    round += t;
  });
  const d = plan.findIndex(([, , , g]) => !g);
  return { steps: plan.length, round, extra, stuck, pointOfNoReturn: d < 0 ? '—' : String(d + 1) };
}

console.log('scenario             steps  unrehearsed stuck  extra round  unrehearsed  rehearsed  point of no return');
for (const [name, plan] of Object.entries(SCENARIO)) {
  const y = run(plan, false), p = run(plan, true);
  console.log(`${name.padEnd(19)} ${String(y.steps).padEnd(5)} ${String(y.stuck.length).padEnd(19)} ` +
    `${String(y.extra).padEnd(13)} ${String(y.round + y.extra).padEnd(13)} ${String(p.round + p.extra).padEnd(10)} ${y.pointOfNoReturn}`);
}
console.log('');
console.log('code + data, first unrehearsed run — step by step');
const kv = run(SCENARIO['code + data'], false);
for (const [no, name, cond, repairRound] of kv.stuck) {
  console.log(`  step ${no} stuck: ${name.padEnd(38)} missing precondition: ${cond} (+${repairRound} round)`);
}
console.log(`  total ${kv.steps} steps, ${kv.round} plan rounds, ${kv.extra} repair rounds`);
console.log(`  no way back after step ${kv.pointOfNoReturn}: the reverse migration overwrites the data`);

scenario             steps  unrehearsed stuck  extra round  unrehearsed  rehearsed  point of no return
code only           5     1                   2             8             6          —
code + config       7     2                   5             14            9          5
code + data         8     2                   8             20            12         6

code + data, first unrehearsed run — step by step
  step 2 stuck: rollback decision and authority        missing precondition: oncall-authority (+2 round)
  step 6 stuck: reverse migration runs                 missing precondition: reverse-migration-script (+6 round)
  total 8 steps, 12 plan rounds, 8 repair rounds
  no way back after step 6: the reverse migration overwrites the data

Three scenarios come from the same incident and take five, seven, eight steps. The rise in step count looks small; the rise in round count does not. In the unrehearsed run, an eight-round plan climbs to twenty rounds — two and a half times. All of that difference comes not from the plan’s steps, but from the steps’ preconditions.

The real number here is that rollback does not carry the same weight as release. A release goes through the pipeline: its stages are ordered, its preconditions are supplied by the pipeline, and every step has already run hundreds of times before. Rollback, by contrast, usually runs for the first time. This asymmetry between the two directions of the same system shows up in the three scenarios’ unrehearsed column.

Three of the model’s six preconditions are not met, and all three are unmet for the same reason: none of them is required in the forward direction. The config record keeping history is not required for release — the latest value is enough going forward. The on-call engineer’s rollback authority is not required for release — the pipeline does the releasing. The migration’s reverse script is not required for release — the migration runs forward. All three are needed only on the way back, and something needed only on the way back is something no one notices until the way back is taken. The moment a missing precondition is discovered is, most of the time, the moment it is needed the most.

The Point of No Return

Not every step in a rollback plan is reversible. In the code-only scenario there is no point of no return: every step moves to the next state without destroying the previous one; someone who wants to change their mind at round eight can release forward to the new version again.

In the config scenario, the point of no return is step five. Because the config record does not keep versions, writing over the previous value means losing the current one. The value lost is the new version’s value, and once rollback fails and someone wants to go back to the new version, that value is no longer there. The step itself takes one second; its irreversibility comes not from its duration, but from the fact that where it writes keeps no history.

In the data scenario, the point of no return is step six. The reverse migration converts the records accumulated in the new schema back to the old format and overwrites them. After this step, the rest of the plan becomes mandatory: if steps seven and eight do not run, the system is in neither the old state nor the new one. That is why the eight-step plan’s last three steps count as a single unit.

Where the point of no return sits inside the plan is direct information for the decision. A rollback that has reached step five or six is no longer “abandonable”; the decision that has to be made before that point is whether rollback is truly necessary. Knowing up to which step a plan is free also says when the decision has to be made.

The Cost of an Unrehearsed Plan

The unrehearsed and rehearsed columns run the same plans, with the same step count, in the same order. The only difference is whether the preconditions are found before the incident or during it. In the code-only scenario the difference is two rounds, in the data scenario eight.

The unrehearsed run’s step-by-step breakdown shows where it gets stuck. At step two, the on-call engineer lacks rollback authority: two rounds pass while someone else is reached and the authority is opened up. At step six, the migration’s reverse script has not been written: six rounds pass while the script gets written during the incident. Where the second stuck point falls is worth noting — right on top of the point of no return. The script gets written at the exact step where the way back closes, and it runs having never been tested.

Rehearsal is not free. The plan’s twelve rounds get paid during rehearsal too, plus the repair of the gaps. The difference is when the payment happens: rehearsal rounds pass outside the incident, at a chosen time, with no one kept waiting. The unrehearsed plan’s eight repair rounds, by contrast, pass inside the incident window, while the defect is still running in production. The same work, at a different price.

Rehearsal’s limit is also part of the measurement. The script not being written gets found by rehearsal, because it is a permanent gap. The on-call engineer’s authority is not: if the rehearsal runs during the day, it passes with the day team’s authority, and the gap that opens up on the night shift stays invisible.

The Gate: Pre-Step Precondition Check

Checking every plan step’s precondition before the step starts is a gate: a step that does not pass does not start. This gate, too, is evaluated on three numbers. A ledger of twelve rollback attempts is run in two modes — checked and unchecked.

// check.mjs — the precondition check is a gate; twelve rollback attempts run in two modes
// [attempt, scenario, actually missing preconditions, what the check wrongly calls "missing"]  MODEL
const ATTEMPT = [
  ['O-01', 'code only', [], []],
  ['O-02', 'code only', ['oncall-authority'], []],
  ['O-03', 'code + config', ['config-previous-version'], []],
  ['O-04', 'code only', [], ['previous-artifact-in-storage']],
  ['O-05', 'code + data', ['reverse-migration-script'], []],
  ['O-06', 'code only', [], []],
  ['O-07', 'code + config', ['oncall-authority', 'config-previous-version'], []],
  ['O-08', 'code only', [], []],
  ['O-09', 'code + data', ['pre-migration-backup'], []],
  ['O-10', 'code only', ['oncall-authority'], []],
  ['O-11', 'code + config', [], ['config-previous-version']],
  ['O-12', 'code + data', [], []],
];
const REPAIR = { 'config-previous-version': 3, 'oncall-authority': 2, 'reverse-migration-script': 6,
  'pre-migration-backup': 4, 'previous-artifact-in-storage': 2 };
const CHECKED = new Set(['previous-artifact-in-storage', 'health-endpoint-compatible',
  'config-previous-version', 'oncall-authority', 'reverse-migration-script']);
const REHEARSAL_CATCHES = new Set(['config-previous-version', 'reverse-migration-script', 'previous-artifact-in-storage']);
const HALFWAY = 5;    // the round cost of getting out of a half-finished rollback's mixed state
const CHECK = 1;      // the round the precondition check adds per plan

const count = { caught: 0, onlyThis: 0, missed: 0, wrong: 0, wrongRounds: 0,
  uncheckedExtra: 0, checkedExtra: 0 };
for (const [, , missing, wrongMissing] of ATTEMPT) {
  const repair = missing.reduce((t, k) => t + REPAIR[k], 0);
  count.uncheckedExtra += repair + (missing.length ? HALFWAY : 0);

  const seen = missing.filter((k) => CHECKED.has(k));
  const missed = missing.filter((k) => !CHECKED.has(k));
  const wasted = wrongMissing.reduce((t, k) => t + REPAIR[k], 0);
  count.wrongRounds += wasted;
  count.checkedExtra += CHECK + repair + (missed.length ? HALFWAY : 0) + wasted;

  if (seen.length) { count.caught += 1; if (seen.every((k) => !REHEARSAL_CATCHES.has(k))) count.onlyThis += 1; }
  if (missed.length) count.missed += 1;
  if (wrongMissing.length) count.wrong += 1;
}

console.log(`${'attempts'.padEnd(46)}: ${ATTEMPT.length}`);
console.log(`${'what the gate catches (half-finished avoided)'.padEnd(46)}: ${count.caught}`);
console.log(`${'of these, only this gate catches'.padEnd(46)}: ${count.onlyThis} (what rehearsal cannot catch)`);
console.log(`${'what the gate misses (unchecked precondition)'.padEnd(46)}: ${count.missed}`);
console.log(`${'what the gate wrongly holds'.padEnd(46)}: ${count.wrong}`);
console.log('');
console.log(`${'rounds added to the incident, unchecked mode'.padEnd(46)}: ${count.uncheckedExtra}`);
console.log(`${'rounds added to the incident, checked mode'.padEnd(46)}: ${count.checkedExtra}`);
console.log(`${'  from the check itself'.padEnd(46)}: ${ATTEMPT.length * CHECK}`);
console.log(`${'  from wrongly holding'.padEnd(46)}: ${count.wrongRounds}`);
console.log(`${"the gate's net gain".padEnd(46)}: ${count.uncheckedExtra - count.checkedExtra} rounds`);
console.log(`${'net gain per attempt'.padEnd(46)}: ${((count.uncheckedExtra - count.checkedExtra) / ATTEMPT.length).toFixed(2)} rounds`);

attempts                                      : 12
what the gate catches (half-finished avoided) : 5
of these, only this gate catches              : 2 (what rehearsal cannot catch)
what the gate misses (unchecked precondition) : 1
what the gate wrongly holds                   : 2

rounds added to the incident, unchecked mode  : 52
rounds added to the incident, checked mode    : 44
  from the check itself                       : 12
  from wrongly holding                        : 5
the gate's net gain                           : 8 rounds
net gain per attempt                          : 0.67 rounds

The gate prevented a halfway rollback in five of twelve attempts. But rehearsal would have caught three of these five too: they were permanent gaps, and running the plan start to finish once would have been enough. The two attempts only this gate catches are the ones where on-call authority happened to be missing that night — a gap tied to the moment of the incident, invisible to a rehearsal that ran during the day. The marginal value is not five, it is two.

The gate missed one attempt: the pre-migration backup not having been taken is not on the check list. The check list is itself a piece of code, and what it does not check has to be counted as much as what it does. In two attempts, the gate wrongly held — it took a precondition that had actually been met for missing, and made the attempt pay a five-round repair for nothing.

The time the gate adds is a separate line in this ledger: one round per attempt, twelve rounds across twelve attempts. This time is paid in a window where the defect keeps running in production, and in seven of the attempts it passes without finding a single gap. The five rounds of wrongly holding are paid in the same window. The gate’s gross gain is twenty-five rounds; the audit’s own cost is twelve rounds and wrongful holding’s is five — seventeen together.

The net gain is forty-four rounds against fifty-two, per-attempt two-thirds of a round. This is a thin margin, and the gate’s own cost — twelve rounds — already exceeds that gain. What keeps the gate justified is not the average but the distribution: in the attempts it catches, the cost is a five-round mixed state, and that cost is paid inside the incident window, while the defect is running. The gate’s only defensible form is splitting the check list with rehearsal: permanent gaps go to rehearsal, ones tied to the moment of the incident go to the gate.

Summary

• The same incident produced three plans: code only five steps, code with config seven steps, code with data eight steps; their unrehearsed total durations were eight, fourteen, and twenty rounds.
• There is no point of no return in the code-only scenario; it is step five in the config scenario and step six in the data scenario, after which the rest of the plan becomes mandatory.
• The unrehearsed run got stuck at two steps and added eight repair rounds; the second stuck point sat right above the point of no return, at the step where the reverse migration script got written during the incident.
• Rehearsal makes you pay for the same work, at a different time: twelve plan rounds pass outside the incident, while eight repair rounds pass while the defect is running in production.
• The precondition-check gate caught five of twelve attempts, caught two of those by itself alone, missed one, wrongly held two; its net gain was eight rounds, its own cost twelve.

Next Step

Two of the three scenarios had a point of no return, and both were tied to data. Code is reversible; configuration is reversible if it is versioned; for a schema change, it is not that simple. Some migration types are fully reversible, some look like they work but bring the data back incomplete, and some have no way back at all. The next lesson ranks migration types on an axis of reversibility, runs the expand–migrate–contract scheme step by step, and at every step counts whether the old and new versions can run at the same time.