--- title: Runbooks source: 'https://academia.sh/en/courses/devops-observability/runbooks' course: 'Observability and Operations' language: en updated: '2026-09-13T05:08:22+00:00' license: 'CC BY-SA 4.0' objectives: - 'Models the runbook as a sequence of steps, and counts decision points and automatable steps' - 'Runs the same incident set in three modes: with no runbook, with a written runbook, and automated' - "Measures the runbook's staleness: how many steps go invalid, how many minutes of delay on the first run, and where error steps land" --- # Runbooks The runbook itself is measured: four runbooks' 33 steps, 6 decision points, and 14 automatable steps; the same 40 incidents closing in 203.1 minutes with 113 error steps with no runbook, and in 72.8 minutes with 40 error steps with a written runbook; a runbook aged two periods pushing the close time to only 83.3 minutes while raising error steps to 161; and after four periods, 24 of 33 steps going invalid and producing 384 minutes of delay on first runs. The on-call rotation answered the "who" and "when" questions; classification added "at which severity". Once the right person wakes at the right moment, one question remains in their hands: **which step**. **The runbook** is a recurring response written down step by step. The M20/K05 `Operations Handover` lesson counted the cost of an **unwritten runbook** on the handover; here the runbook is written, and what is measured is **the runbook itself**: how many steps, how many decision points, how many can go to a machine, and how many are still correct some time after being written. ## A Runbook Is Not a Document, It Is a Sequence of Steps The example is once more the course's fictional **regional measurement network**: the water-meter reading collector, verifier, billing, and work-order services. Both the network and the runbooks are fictional; there is no real runbook text or run history. **IM13:** A runbook is written for each of four recurring responses; a step's type is **observe**, **verify**, **intervene**, or **decide**, and a decision point can never be automated. **IM14:** Every step depends on an **anchor** — a gauge name, series name, threshold value, file path, flag name, queue name, ID field, version tag. If the anchor changes, every step that depends on it becomes invalid. **IM15:** In a run with no runbook, finding the step multiplies the time by 2.8, and the per-step error probability is 0.34; with a written runbook it is 0.08. Fixing an erroneous step takes 9 minutes. **IM16:** The probability of picking the wrong branch at a decision point is 0.18, at a cost of 12 minutes. **IM17:** An invalid step produces 16 minutes of delay on the first run; on later runs it is known and worked around in 3 minutes, but it counts as an error step either way. **IM18:** Two anchors change every period; the same runbook set is run over 40 incidents. ```js // runbook/runbook.mjs — the fictional regional measurement network's runbook model (model): // there is no real runbook text, incident record, or run history. The generator is our own, the seed is visible. export const SERVICE = ["reading-collector", "verifier", "billing", "work-order"]; export const TYPE = ["observe", "verify", "intervene", "decide"]; // The system fact a step depends on. If this anchor changes, the step becomes invalid. export const ANCHOR = ["gauge-name", "series-name", "threshold-value", "file-path", "flag-name", "queue-name", "id-field", "version-tag"]; export const DISCOVERY = 2.8; // time multiplier for finding a step with no runbook export const ERROR = { none: 0.34, written: 0.08 }; // per-step error probability export const FIX = 9; // fixing an erroneous step (minutes) export const WRONG_BRANCH = 0.18; // probability of picking the wrong branch at a decision point export const BRANCH_PENALTY = 12; // the cost of the wrong branch (minutes) export const FIRST_PENALTY = 16; // an invalid step, first run: attempted, fails, found by hand export const LATER_PENALTY = 3; // the same step on later runs: known, worked around export const CHANGE = 2; // number of anchors that change per period export function generator(seed) { let x = seed >>> 0 || 1; return () => { x ^= x << 13; x >>>= 0; x ^= x >>> 17; x ^= x << 5; x >>>= 0; return x / 4294967296; }; } const pick = (r, a) => a[Math.floor(r() * a.length)]; // One runbook per service: 6-10 steps. Automatability depends on the step's type; // a decision point can never be automated. export function runbooks(seed) { const r = generator(seed); return SERVICE.map((service) => { const count = 6 + Math.floor(r() * 5); const step = []; for (let i = 0; i < count; i++) { const u = r(); const type = u < 0.22 ? "observe" : u < 0.52 ? "verify" : u < 0.82 ? "intervene" : "decide"; const auto = type === "decide" ? false : r() < (type === "intervene" ? 0.40 : 0.75); step.push({ type, auto, minute: 2 + Math.floor(r() * 11), anchor: pick(r, ANCHOR) }); } return { service, step }; }); } // mode: "none" no runbook, "written" a written runbook, "auto" automatable steps run on a machine. // invalid: invalid anchors. seen: invalid steps already surfaced in an earlier run; // an invalid step produces its full penalty only on the FIRST run, later ones are worked around. export function run(b, mode, r, invalid = new Set(), seen = new Set()) { let minute = 0, errorStep = 0, firstDelay = 0, skipped = 0; for (const [i, a] of b.step.entries()) { const auto = mode === "auto" && a.auto; if (auto) { skipped++; } else { minute += a.minute * (mode === "none" ? DISCOVERY : 1); } if (invalid.has(a.anchor)) { // the runbook has fallen behind the system const key = `${b.service}:${i}`; if (seen.has(key)) { minute += LATER_PENALTY; errorStep++; continue; } seen.add(key); minute += FIRST_PENALTY; firstDelay += FIRST_PENALTY; errorStep++; continue; } if (auto) continue; // a machine-run step never errs if (r() < ERROR[mode === "none" ? "none" : "written"]) { minute += FIX; errorStep++; } if (a.type === "decide" && r() < WRONG_BRANCH) { minute += BRANCH_PENALTY; errorStep++; } } return { minute, errorStep, firstDelay, skipped }; } // Staleness: CHANGE anchors change every period. Every step that depends on a changed anchor is invalid. export function age(seed, period) { const r = generator(seed), g = new Set(); for (let p = 0; p < period; p++) for (let i = 0; i < CHANGE; i++) g.add(pick(r, ANCHOR)); return g; } ``` ```js // runbook/measure.mjs — the runbook itself: steps, decision points, automation, and staleness. import { runbooks, run, age, generator, TYPE, ANCHOR, CHANGE } from "./runbook.mjs"; const SEED = 20261107; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const f1 = (x) => x.toFixed(1); const count = (a, f) => a.filter(f).length; const R = runbooks(SEED); const allSteps = R.flatMap((b) => b.step); const RUNS = 40; // the same runbook set is run over 40 incidents // Runs the same incident set in one mode. Every run starts from the same seed, so the // difference between modes comes from the mode, not from the draws. const attempt = (mode, invalid) => { const r = generator(SEED + 7), seen = new Set(), record = []; for (let i = 0; i < RUNS; i++) record.push(run(R[i % R.length], mode, r, invalid, seen)); const t = (f) => record.reduce((a, x) => a + f(x), 0); return { minute: t((x) => x.minute) / RUNS, error: t((x) => x.errorStep), first: t((x) => x.firstDelay), skipped: t((x) => x.skipped) }; }; const E = [-19, 7, 12, 15, 10, 11]; console.log(`seed ${SEED}: ${R.length} runbooks, ${allSteps.length} steps, ${RUNS} runs`); console.log("\n1. runbook inventory"); write(E, "runbook", "steps", "dec. pts.", "auto steps", "minutes", "anchors"); for (const b of R) write(E, b.service, b.step.length, count(b.step, (a) => a.type === "decide"), count(b.step, (a) => a.auto), b.step.reduce((a, x) => a + x.minute, 0), new Set(b.step.map((a) => a.anchor)).size); write(E, "total", allSteps.length, count(allSteps, (a) => a.type === "decide"), count(allSteps, (a) => a.auto), allSteps.reduce((a, x) => a + x.minute, 0), new Set(allSteps.map((a) => a.anchor)).size); console.log("step type: " + TYPE.map((t) => `${t} ${count(allSteps, (a) => a.type === t)}`).join(", ")); const g2 = age(SEED + 3, 2); // a runbook aged two periods const M = [-22, 17, 14, 16, 16]; console.log("\n2. the same 40 incidents in three run modes (fresh runbook / aged two periods)"); write(M, "mode", "avg close min", "error steps", "auto steps", "vs. no runbook"); const fresh = {}, aged = {}; for (const [name, mode] of [["no runbook", "none"], ["written runbook", "written"], ["automated", "auto"]]) { fresh[mode] = attempt(mode, new Set()); aged[mode] = mode === "none" ? fresh[mode] : attempt(mode, g2); const share = (x) => `${((x.minute / fresh.none.minute) * 100).toFixed(0)}%`; write(M, name, `${f1(fresh[mode].minute)} / ${f1(aged[mode].minute)}`, `${fresh[mode].error} / ${aged[mode].error}`, `${fresh[mode].skipped} / ${aged[mode].skipped}`, `${share(fresh[mode])} / ${share(aged[mode])}`); } const D = [-8, 15, 15, 19, 16, 13]; console.log(`\n3. the runbook's staleness (written runbook, ${CHANGE} changes per period)`); write(D, "period", "invalid anch.", "invalid steps", "1st run delay", "avg close min", "error steps"); for (const period of [0, 1, 2, 3, 4]) { const g = age(SEED + 3, period); const r = attempt("written", g); write(D, period, `${g.size}/${ANCHOR.length}`, `${count(allSteps, (a) => g.has(a.anchor))}/${allSteps.length}`, `${r.first} min`, f1(r.minute), r.error); } const Q = [-47, 20, 8]; console.log("\n4. the question set the runbook answers"); write(Q, "question", "data", "answer"); const QUESTION = [ ["has this incident happened before", "runbook index", true], ["which step does it close with", "runbook text", true], ["how many steps, how many decision points", "runbook text", true], ["which step can go to a machine", "step type", true], ["when was the runbook last run", "run record", true], ["is this step still valid", "-", false], ["which branch is correct at the decision point", "-", false], ["how many steps are now invalid", "-", false]]; for (const [q, d, y] of QUESTION) write(Q, q, d, y ? "yes" : "no"); console.log(`${count(QUESTION, (x) => x[2])} of ${QUESTION.length} questions are answered from ` + `the runbook, ${count(QUESTION, (x) => !x[2])} cannot be answered without running it`); ``` ``` seed 20261107: 4 runbooks, 33 steps, 40 runs 1. runbook inventory runbook steps dec. pts. auto steps minutes anchors reading-collector 6 2 2 47 3 verifier 9 1 5 64 5 billing 10 0 5 86 6 work-order 8 3 2 56 5 total 33 6 14 253 8 step type: observe 7, verify 8, intervene 12, decide 6 2. the same 40 incidents in three run modes (fresh runbook / aged two periods) mode avg close min error steps auto steps vs. no runbook no runbook 203.1 / 203.1 113 / 113 0 / 0 100% / 100% written runbook 72.8 / 83.3 40 / 161 0 / 0 36% / 41% automated 37.5 / 49.8 25 / 154 140 / 140 18% / 24% 3. the runbook's staleness (written runbook, 2 changes per period) period invalid anch. invalid steps 1st run delay avg close min error steps 0 0/8 0/33 0 min 72.8 40 1 2/8 10/33 160 min 80.6 126 2 3/8 14/33 224 min 83.3 161 3 5/8 20/33 320 min 88.1 214 4 6/8 24/33 384 min 91.6 251 4. the question set the runbook answers question data answer has this incident happened before runbook index yes which step does it close with runbook text yes how many steps, how many decision points runbook text yes which step can go to a machine step type yes when was the runbook last run run record yes is this step still valid - no which branch is correct at the decision point - no how many steps are now invalid - no 5 of 8 questions are answered from the runbook, 3 cannot be answered without running it ``` The numbers are of the **measurement** class; their inputs are the IM13–IM18 assumptions. Four runbooks carry 33 steps, 6 decision points, and a nominal 253 minutes. The inventory alone says two things. `billing` is the longest runbook (10 steps, 86 minutes) but has no decision point at all — it runs straight through. `work-order` is only 8 steps, but carries 3 decision points: a short runbook is not an easy runbook. ## With No Runbook, With a Runbook, With a Machine The left half of the second table closes the same 40 incidents in three modes. In the no-runbook run, average close time is 203.1 minutes with 113 error steps; finding the step multiplies the time, and every step found can go wrong. The written runbook drops close time to 72.8 minutes, that is, **36% of no runbook**, and cuts error steps from 113 to 40. The gain has two sources, and they separate: dropping the discovery multiplier cuts the time, and the error probability falling from 0.34 to 0.08 cuts the mistakes. Automation takes third place. 14 of the 33 steps can go to a machine (42%), and across 40 runs a total of 140 steps run on a machine; close time drops to 37.5 minutes, **18% of no runbook**, and error steps drop to 25. This ceiling is no coincidence: 6 of the remaining 19 steps are **decision points**, and by assumption they cannot be automated. **A runbook's unautomatable core is a countable quantity** — here, six decisions. ## The Runbook's Staleness The third table leaves the runbook where it is and changes the system instead. In one period, two anchors change and **10** of the 33 steps go invalid; after four periods, six of eight anchors have changed and invalid steps reach 24. The invalid step's cost on the first run reads directly: 160 minutes in the first period, 384 in the fourth. The real finding is that the two columns degrade **at different rates**. Average close time rises from 72.8 to 91.6 minutes — only 26% over four periods. Error steps, on the other hand, rise from 40 to 251, a sixfold increase. The reason sits inside the assumption: an invalid step costs 16 minutes on the first run, is worked around in 3 minutes on later runs, but stays an error step every single time. **A stale runbook does not slow down, it becomes unreliable**; a measure that watches only the clock cannot see this. The right half of the second table sharpens this. A runbook aged two periods pushes close time from only 72.8 to 83.3 minutes and is still far faster than no runbook (41% versus 100%). But its error steps are 161 — **more than the no-runbook run's 113 error steps**. A stale runbook speeds the runner up while walking them into the wrong step more often. Automation does not stop this decay. A step handed to a machine still depends on the same anchor: after two periods, the automated mode's error steps rise from 25 to 154, landing almost exactly where the written runbook's 161 sits. Close time rises from 37.5 to 49.8 minutes. **Automation speeds up the step; it does not verify the step's anchor.** ## What the Runbook Does Not Know The fourth table shows that five of eight questions are answered from the runbook: whether the incident has happened before, which step it closes with, how many steps and decision points there are, which step can go to a machine, and when the runbook was last run. The three left unanswered reduce to a single gap — **the runbook does not know its own validity**. "Is this step still valid", "which branch is correct at the decision point", and "how many steps are now invalid" can only be answered once the runbook is run, that is, in the middle of an incident. The delay between 160 and 384 minutes in the third table is the bill for these three questions. ## Summary - Four runbooks carry 33 steps, 6 decision points, and a nominal 253 minutes; 14 of the steps (42%) can go to a machine. The longest runbook has no decision point at all, and the 8-step runbook has three. - The same 40 incidents close in 203.1 minutes with 113 error steps with no runbook, and in 72.8 minutes (36%) with 40 error steps with a written runbook. - Automation drops close time to 37.5 minutes (18%) and error steps to 25; what sets the ceiling is the 6 decision points that cannot be automated. - When two anchors change in one period, 10 of the 33 steps go invalid, and after four periods 24 do; the delay invalid steps produce on the first run rises from 160 to 384 minutes. - Staleness degrades time and trust at different rates: close time rises from 72.8 to 91.6 minutes (26%) while error steps rise from 40 to 251 (sixfold). - A runbook aged two periods is still more than twice as fast as no runbook (83.3 versus 203.1 minutes) but errs far more than no runbook (161 versus 113 error steps); automation does not change this (154). ## Next Step A runbook writes **how** an incident closes, not **why** it opened. The third table's invalid steps also do not correct themselves; someone has to see them and update the runbook. The next lesson looks at what happens after the incident closes: a blameless review of the incident, tracking the action items that come out of the review, and an item left open reproducing the same incident.