Runbooks

The on-call rotation answered the “who” and “when” questions; classification added “at which severity”. Once the right person wakes at the right moment, one question remains in their hands: which step.

The runbook is a recurring response written down step by step. The M20/K05 Operations Handover lesson counted the cost of an unwritten runbook on the handover; here the runbook is written, and what is measured is the runbook itself: how many steps, how many decision points, how many can go to a machine, and how many are still correct some time after being written.

A Runbook Is Not a Document, It Is a Sequence of Steps

The example is once more the course’s fictional regional measurement network: the water-meter reading collector, verifier, billing, and work-order services. Both the network and the runbooks are fictional; there is no real runbook text or run history.

IM13: A runbook is written for each of four recurring responses; a step’s type is observe, verify, intervene, or decide, and a decision point can never be automated. IM14: Every step depends on an anchor — a gauge name, series name, threshold value, file path, flag name, queue name, ID field, version tag. If the anchor changes, every step that depends on it becomes invalid. IM15: In a run with no runbook, finding the step multiplies the time by 2.8, and the per-step error probability is 0.34; with a written runbook it is 0.08. Fixing an erroneous step takes 9 minutes. IM16: The probability of picking the wrong branch at a decision point is 0.18, at a cost of 12 minutes. IM17: An invalid step produces 16 minutes of delay on the first run; on later runs it is known and worked around in 3 minutes, but it counts as an error step either way. IM18: Two anchors change every period; the same runbook set is run over 40 incidents.

// runbook/runbook.mjs — the fictional regional measurement network's runbook model (model):
// there is no real runbook text, incident record, or run history. The generator is our own, the seed is visible.

export const SERVICE = ["reading-collector", "verifier", "billing", "work-order"];
export const TYPE = ["observe", "verify", "intervene", "decide"];
// The system fact a step depends on. If this anchor changes, the step becomes invalid.
export const ANCHOR = ["gauge-name", "series-name", "threshold-value", "file-path", "flag-name",
  "queue-name", "id-field", "version-tag"];

export const DISCOVERY = 2.8;        // time multiplier for finding a step with no runbook
export const ERROR = { none: 0.34, written: 0.08 };   // per-step error probability
export const FIX = 9;         // fixing an erroneous step (minutes)
export const WRONG_BRANCH = 0.18;  // probability of picking the wrong branch at a decision point
export const BRANCH_PENALTY = 12;  // the cost of the wrong branch (minutes)
export const FIRST_PENALTY = 16;   // an invalid step, first run: attempted, fails, found by hand
export const LATER_PENALTY = 3;    // the same step on later runs: known, worked around
export const CHANGE = 2;           // number of anchors that change per period

export function generator(seed) {
  let x = seed >>> 0 || 1;
  return () => { x ^= x << 13; x >>>= 0; x ^= x >>> 17; x ^= x << 5; x >>>= 0;
    return x / 4294967296; };
}
const pick = (r, a) => a[Math.floor(r() * a.length)];

// One runbook per service: 6-10 steps. Automatability depends on the step's type;
// a decision point can never be automated.
export function runbooks(seed) {
  const r = generator(seed);
  return SERVICE.map((service) => {
    const count = 6 + Math.floor(r() * 5);
    const step = [];
    for (let i = 0; i < count; i++) {
      const u = r();
      const type = u < 0.22 ? "observe" : u < 0.52 ? "verify" : u < 0.82 ? "intervene" : "decide";
      const auto = type === "decide" ? false
        : r() < (type === "intervene" ? 0.40 : 0.75);
      step.push({ type, auto, minute: 2 + Math.floor(r() * 11),
        anchor: pick(r, ANCHOR) });
    }
    return { service, step };
  });
}

// mode: "none" no runbook, "written" a written runbook, "auto" automatable steps run on a machine.
// invalid: invalid anchors. seen: invalid steps already surfaced in an earlier run;
// an invalid step produces its full penalty only on the FIRST run, later ones are worked around.
export function run(b, mode, r, invalid = new Set(), seen = new Set()) {
  let minute = 0, errorStep = 0, firstDelay = 0, skipped = 0;
  for (const [i, a] of b.step.entries()) {
    const auto = mode === "auto" && a.auto;
    if (auto) { skipped++; } else { minute += a.minute * (mode === "none" ? DISCOVERY : 1); }
    if (invalid.has(a.anchor)) {      // the runbook has fallen behind the system
      const key = `${b.service}:${i}`;
      if (seen.has(key)) { minute += LATER_PENALTY; errorStep++; continue; }
      seen.add(key);
      minute += FIRST_PENALTY; firstDelay += FIRST_PENALTY; errorStep++;
      continue;
    }
    if (auto) continue;                  // a machine-run step never errs
    if (r() < ERROR[mode === "none" ? "none" : "written"]) { minute += FIX; errorStep++; }
    if (a.type === "decide" && r() < WRONG_BRANCH) { minute += BRANCH_PENALTY; errorStep++; }
  }
  return { minute, errorStep, firstDelay, skipped };
}

// Staleness: CHANGE anchors change every period. Every step that depends on a changed anchor is invalid.
export function age(seed, period) {
  const r = generator(seed), g = new Set();
  for (let p = 0; p < period; p++)
    for (let i = 0; i < CHANGE; i++) g.add(pick(r, ANCHOR));
  return g;
}

// runbook/measure.mjs — the runbook itself: steps, decision points, automation, and staleness.
import { runbooks, run, age, generator, TYPE, ANCHOR, CHANGE } from "./runbook.mjs";

const SEED = 20261107;
const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const f1 = (x) => x.toFixed(1);
const count = (a, f) => a.filter(f).length;

const R = runbooks(SEED);
const allSteps = R.flatMap((b) => b.step);
const RUNS = 40;                      // the same runbook set is run over 40 incidents

// Runs the same incident set in one mode. Every run starts from the same seed, so the
// difference between modes comes from the mode, not from the draws.
const attempt = (mode, invalid) => {
  const r = generator(SEED + 7), seen = new Set(), record = [];
  for (let i = 0; i < RUNS; i++) record.push(run(R[i % R.length], mode, r, invalid, seen));
  const t = (f) => record.reduce((a, x) => a + f(x), 0);
  return { minute: t((x) => x.minute) / RUNS, error: t((x) => x.errorStep),
    first: t((x) => x.firstDelay), skipped: t((x) => x.skipped) };
};

const E = [-19, 7, 12, 15, 10, 11];
console.log(`seed ${SEED}: ${R.length} runbooks, ${allSteps.length} steps, ${RUNS} runs`);
console.log("\n1. runbook inventory");
write(E, "runbook", "steps", "dec. pts.", "auto steps", "minutes", "anchors");
for (const b of R)
  write(E, b.service, b.step.length, count(b.step, (a) => a.type === "decide"),
    count(b.step, (a) => a.auto), b.step.reduce((a, x) => a + x.minute, 0),
    new Set(b.step.map((a) => a.anchor)).size);
write(E, "total", allSteps.length, count(allSteps, (a) => a.type === "decide"),
  count(allSteps, (a) => a.auto), allSteps.reduce((a, x) => a + x.minute, 0),
  new Set(allSteps.map((a) => a.anchor)).size);
console.log("step type: " + TYPE.map((t) => `${t} ${count(allSteps, (a) => a.type === t)}`).join(", "));

const g2 = age(SEED + 3, 2);        // a runbook aged two periods
const M = [-22, 17, 14, 16, 16];
console.log("\n2. the same 40 incidents in three run modes (fresh runbook / aged two periods)");
write(M, "mode", "avg close min", "error steps", "auto steps", "vs. no runbook");
const fresh = {}, aged = {};
for (const [name, mode] of [["no runbook", "none"], ["written runbook", "written"],
  ["automated", "auto"]]) {
  fresh[mode] = attempt(mode, new Set());
  aged[mode] = mode === "none" ? fresh[mode] : attempt(mode, g2);
  const share = (x) => `${((x.minute / fresh.none.minute) * 100).toFixed(0)}%`;
  write(M, name, `${f1(fresh[mode].minute)} / ${f1(aged[mode].minute)}`,
    `${fresh[mode].error} / ${aged[mode].error}`, `${fresh[mode].skipped} / ${aged[mode].skipped}`,
    `${share(fresh[mode])} / ${share(aged[mode])}`);
}

const D = [-8, 15, 15, 19, 16, 13];
console.log(`\n3. the runbook's staleness (written runbook, ${CHANGE} changes per period)`);
write(D, "period", "invalid anch.", "invalid steps", "1st run delay", "avg close min",
  "error steps");
for (const period of [0, 1, 2, 3, 4]) {
  const g = age(SEED + 3, period);
  const r = attempt("written", g);
  write(D, period, `${g.size}/${ANCHOR.length}`,
    `${count(allSteps, (a) => g.has(a.anchor))}/${allSteps.length}`,
    `${r.first} min`, f1(r.minute), r.error);
}

const Q = [-47, 20, 8];
console.log("\n4. the question set the runbook answers");
write(Q, "question", "data", "answer");
const QUESTION = [
  ["has this incident happened before", "runbook index", true],
  ["which step does it close with", "runbook text", true],
  ["how many steps, how many decision points", "runbook text", true],
  ["which step can go to a machine", "step type", true],
  ["when was the runbook last run", "run record", true],
  ["is this step still valid", "-", false],
  ["which branch is correct at the decision point", "-", false],
  ["how many steps are now invalid", "-", false]];
for (const [q, d, y] of QUESTION) write(Q, q, d, y ? "yes" : "no");
console.log(`${count(QUESTION, (x) => x[2])} of ${QUESTION.length} questions are answered from ` +
  `the runbook, ${count(QUESTION, (x) => !x[2])} cannot be answered without running it`);

seed 20261107: 4 runbooks, 33 steps, 40 runs

1. runbook inventory
runbook              steps   dec. pts.     auto steps   minutes    anchors
reading-collector        6           2              2        47          3
verifier                 9           1              5        64          5
billing                 10           0              5        86          6
work-order               8           3              2        56          5
total                   33           6             14       253          8
step type: observe 7, verify 8, intervene 12, decide 6

2. the same 40 incidents in three run modes (fresh runbook / aged two periods)
mode                      avg close min   error steps      auto steps  vs. no runbook
no runbook                203.1 / 203.1     113 / 113           0 / 0     100% / 100%
written runbook             72.8 / 83.3      40 / 161           0 / 0       36% / 41%
automated                   37.5 / 49.8      25 / 154       140 / 140       18% / 24%

3. the runbook's staleness (written runbook, 2 changes per period)
period    invalid anch.  invalid steps      1st run delay   avg close min  error steps
0                   0/8           0/33              0 min            72.8           40
1                   2/8          10/33            160 min            80.6          126
2                   3/8          14/33            224 min            83.3          161
3                   5/8          20/33            320 min            88.1          214
4                   6/8          24/33            384 min            91.6          251

4. the question set the runbook answers
question                                                       data  answer
has this incident happened before                     runbook index     yes
which step does it close with                          runbook text     yes
how many steps, how many decision points               runbook text     yes
which step can go to a machine                            step type     yes
when was the runbook last run                            run record     yes
is this step still valid                                          -      no
which branch is correct at the decision point                     -      no
how many steps are now invalid                                    -      no
5 of 8 questions are answered from the runbook, 3 cannot be answered without running it

The numbers are of the measurement class; their inputs are the IM13–IM18 assumptions. Four runbooks carry 33 steps, 6 decision points, and a nominal 253 minutes. The inventory alone says two things. billing is the longest runbook (10 steps, 86 minutes) but has no decision point at all — it runs straight through. work-order is only 8 steps, but carries 3 decision points: a short runbook is not an easy runbook.

With No Runbook, With a Runbook, With a Machine

The left half of the second table closes the same 40 incidents in three modes. In the no-runbook run, average close time is 203.1 minutes with 113 error steps; finding the step multiplies the time, and every step found can go wrong. The written runbook drops close time to 72.8 minutes, that is, 36% of no runbook, and cuts error steps from 113 to 40. The gain has two sources, and they separate: dropping the discovery multiplier cuts the time, and the error probability falling from 0.34 to 0.08 cuts the mistakes.

Automation takes third place. 14 of the 33 steps can go to a machine (42%), and across 40 runs a total of 140 steps run on a machine; close time drops to 37.5 minutes, 18% of no runbook, and error steps drop to 25. This ceiling is no coincidence: 6 of the remaining 19 steps are decision points, and by assumption they cannot be automated. A runbook’s unautomatable core is a countable quantity — here, six decisions.

The Runbook’s Staleness

The third table leaves the runbook where it is and changes the system instead. In one period, two anchors change and 10 of the 33 steps go invalid; after four periods, six of eight anchors have changed and invalid steps reach 24. The invalid step’s cost on the first run reads directly: 160 minutes in the first period, 384 in the fourth.

The real finding is that the two columns degrade at different rates. Average close time rises from 72.8 to 91.6 minutes — only 26% over four periods. Error steps, on the other hand, rise from 40 to 251, a sixfold increase. The reason sits inside the assumption: an invalid step costs 16 minutes on the first run, is worked around in 3 minutes on later runs, but stays an error step every single time. A stale runbook does not slow down, it becomes unreliable; a measure that watches only the clock cannot see this.

The right half of the second table sharpens this. A runbook aged two periods pushes close time from only 72.8 to 83.3 minutes and is still far faster than no runbook (41% versus 100%). But its error steps are 161 — more than the no-runbook run’s 113 error steps. A stale runbook speeds the runner up while walking them into the wrong step more often.

Automation does not stop this decay. A step handed to a machine still depends on the same anchor: after two periods, the automated mode’s error steps rise from 25 to 154, landing almost exactly where the written runbook’s 161 sits. Close time rises from 37.5 to 49.8 minutes. Automation speeds up the step; it does not verify the step’s anchor.

What the Runbook Does Not Know

The fourth table shows that five of eight questions are answered from the runbook: whether the incident has happened before, which step it closes with, how many steps and decision points there are, which step can go to a machine, and when the runbook was last run. The three left unanswered reduce to a single gap — the runbook does not know its own validity. “Is this step still valid”, “which branch is correct at the decision point”, and “how many steps are now invalid” can only be answered once the runbook is run, that is, in the middle of an incident. The delay between 160 and 384 minutes in the third table is the bill for these three questions.

Summary

• Four runbooks carry 33 steps, 6 decision points, and a nominal 253 minutes; 14 of the steps (42%) can go to a machine. The longest runbook has no decision point at all, and the 8-step runbook has three.
• The same 40 incidents close in 203.1 minutes with 113 error steps with no runbook, and in 72.8 minutes (36%) with 40 error steps with a written runbook.
• Automation drops close time to 37.5 minutes (18%) and error steps to 25; what sets the ceiling is the 6 decision points that cannot be automated.
• When two anchors change in one period, 10 of the 33 steps go invalid, and after four periods 24 do; the delay invalid steps produce on the first run rises from 160 to 384 minutes.
• Staleness degrades time and trust at different rates: close time rises from 72.8 to 91.6 minutes (26%) while error steps rise from 40 to 251 (sixfold).
• A runbook aged two periods is still more than twice as fast as no runbook (83.3 versus 203.1 minutes) but errs far more than no runbook (161 versus 113 error steps); automation does not change this (154).

Next Step

A runbook writes how an incident closes, not why it opened. The third table’s invalid steps also do not correct themselves; someone has to see them and update the runbook. The next lesson looks at what happens after the incident closes: a blameless review of the incident, tracking the action items that come out of the review, and an item left open reproducing the same incident.