Post-Incident Review

The previous lesson counted the runbook itself: step count, decision points, and stale steps. The runbook was written, the incident closed, the line reopened. One question was left afterward, never asked: will the same incident happen again, what was done to keep it from happening, and was that work finished.

The post-incident analysis itself was measured in M16/K07. What is measured here is not the review’s output, but the tracking of that output. The measure is this: how many action items were opened in a period, how many closed, how long closing took, and how many times an unclosed item reproduced the same incident. The blameless review also has a measurable counterpart: how many systemic fixes come out when the root cause is written as a person, versus when it is written as a process.

The Question Set and Record Layers

The example is once more the fictional regional measurement network; the meter-reading collector, verifier, billing, and work-order services. Both the incident history and the review log are fictional.

IM19: Eighteen periods are run, each period is 30 days; each of five cause classes has a fixed daily incident probability and a fixed outage in minutes. IM20: Every incident is reviewed three days later, and the review takes five person-hours. IM21: Every review opens three action items; the root cause is written as either person or process, and the writeup determines the item types’ shares — for person: reminder 0.55, documentation 0.30, systemic 0.15; for process: systemic 0.55, documentation 0.25, reminder 0.20. IM22: The item type determines the closing probability, the closing time, and the person-hours per closed item; the follow-up multiplier scales the closing probability, capped at 0.98. IM23: A closed systemic item multiplies its class’s daily incident probability by 0.70; documentation and reminders never change any probability. IM24: Repeat means the class has produced an incident before; recurring incident means the class produced a new incident while one of its systemic items that will never close sits open. The seed is 20260812, and all dice are rolled up front, so the five runs share the same sequence.

Incidents, reviews, and items are a model internal to the course; there is no real incident record.

// review/model.mjs — the post-incident review model: there is no real incident record;
// days, incidents, and action items are data structures. The generator is our own, the seed is visible.

export const PERIOD = 30, PERIOD_COUNT = 18, DAY = PERIOD * PERIOD_COUNT;
export const SEED = 20260812, ITEM = 3, DELAY = 3, DROP = 0.70;
// cause class: name, daily incident probability, outage minutes
export const CLASS = [["configuration drift", 0.030, 25], ["capacity", 0.022, 40],
  ["dependency timeout", 0.026, 18], ["data quality", 0.020, 55],
  ["batch job window", 0.016, 70]];
// item type: closing probability, closing days, person-hours per closed item
export const TYPE = { systemic: [0.55, 25, 12], documentation: [0.80, 8, 3], reminder: [0.95, 2, 0.5] };
// the shares of item types opened when the root cause is written up as person or process
export const MIX = { person: [["reminder", 0.55], ["documentation", 0.30], ["systemic", 0.15]],
  process: [["systemic", 0.55], ["documentation", 0.25], ["reminder", 0.20]] };
export const REVIEW_HOURS = 5;                    // review person-hours per incident

export function generator(seed) {                    // linear congruential generator
  let x = seed >>> 0;
  return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32);
}

// all dice are rolled up front: runs share the same incident and item sequence
const Z = (() => { const z = generator(SEED);
  return Array.from({ length: DAY * CLASS.length }, () => ({ incident: z(),
    item: Array.from({ length: ITEM }, () => ({ type: z(), close: z(), duration: z() })) })); })();
const pick = (mix, p) => { let e = 0; for (const [name, y] of mix) { e += y; if (p < e) return name; } };

export function run({ review = true, writeup = "process", followUp = 1.0 } = {}) {
  const multiplier = CLASS.map(() => 1), openSystemic = CLASS.map(() => 0);   // per-class state
  const stuckSystemic = CLASS.map(() => 0), seen = CLASS.map(() => 0), incidents = [], items = [];
  for (let g = 0; g < DAY; g++) {
    for (const m of items)                      // an item whose closing day has arrived, closes
      if (m.status === "open" && m.closeDay === g) {
        m.status = "closed";
        if (m.type === "systemic") { multiplier[m.class] *= DROP; openSystemic[m.class]--; }
      }
    for (let s = 0; s < CLASS.length; s++) {
      const z = Z[g * CLASS.length + s];
      if (z.incident >= CLASS[s][1] * multiplier[s]) continue;
      // repeat: the class has produced an incident before. recurring: the class produced a new
      // incident while one of its systemic items that will never close sits open
      incidents.push({ g, class: s, outage: CLASS[s][2],
        repeat: seen[s]++ > 0, recurring: stuckSystemic[s] > 0 });
      if (!review) continue;
      for (const d of z.item) {
        const type = pick(MIX[writeup], d.type), [closeProb, closeDays] = TYPE[type];
        const closes = d.close < Math.min(0.98, closeProb * followUp);
        const elapsed = Math.round(closeDays * (0.6 + 0.8 * d.duration));
        items.push({ class: s, type, opened: g + DELAY, status: "open",
          closeDay: closes ? g + DELAY + elapsed : null, elapsed: closes ? elapsed : null });
        if (type === "systemic") { openSystemic[s]++; if (!closes) stuckSystemic[s]++; }
      }
    }
  }
  return { incidents, items };
}

const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-");
const pct = (a, b) => (b ? (100 * a / b).toFixed(0) + "%" : "-");

export function measure(settings) {
  const { incidents, items } = run(settings), M = items.length;
  const closed = items.filter((m) => m.status === "closed");
  const recurring = incidents.filter((o) => o.recurring), repeat = incidents.filter((o) => o.repeat);
  const minutes = (a) => a.reduce((s, o) => s + o.outage, 0);
  const systemic = items.filter((m) => m.type === "systemic");
  return { incident: incidents.length, outage: minutes(incidents), opened: M, closed: closed.length,
    repeat: repeat.length, repeatRate: pct(repeat.length, incidents.length), repeatOutage: minutes(repeat),
    recurring: recurring.length, recurringRate: pct(recurring.length, incidents.length), recurringOutage: minutes(recurring),
    itemsPerPeriod: (M / PERIOD_COUNT).toFixed(1), closeRate: pct(closed.length, M),
    closeTime: avg(closed.map((m) => m.elapsed)), systemic: systemic.length, systemicShare: pct(systemic.length, M),
    systemicClosed: systemic.filter((m) => m.status === "closed").length,
    hours: incidents.length * (settings.review === false ? 0 : REVIEW_HOURS)
      + closed.reduce((s, m) => s + TYPE[m.type][2], 0) };
}

The measurement file sets up an eight-question set and writes the record fields each question’s answer needs. The layers are added on top of each other.

// review/measure.mjs — which record layer answers which question, what changes when the
// action item is tracked, how many incidents an unclosed item reproduces.
import { PERIOD, PERIOD_COUNT, CLASS, SEED, ITEM, measure } from "./model.mjs";

const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const t0 = process.hrtime.bigint();

// question set: the record fields needed to answer each question
const QUESTION = [
  ["when did the incident happen, how long did it last", ["incident"]],
  ["what was written as the root cause", ["note"]],
  ["was the root cause written as person or process", ["writeup"]],
  ["how many items did the review open", ["log"]],
  ["how many of the items closed, in what time", ["log", "closing"]],
  ["how many systemic fixes did the person writeup produce", ["writeup", "log"]],
  ["which incident did an unclosed item reproduce", ["log", "link"]],
  ["what outage did closing the item prevent", ["counterfactual"]],
];
// record layer: name, the fields it opens, unit bytes, the object the unit is counted on
const LAYER = [["incident record", ["incident"], 240, "incident"], ["+ review note", ["note"], 1800, "incident"],
  ["+ writeup field", ["writeup"], 24, "incident"], ["+ item log", ["log", "closing"], 320, "item"],
  ["+ item-class link", ["link"], 16, "item"]];

const K = { "no review": { review: false }, "person writeup": { writeup: "person" },
  "process writeup": { writeup: "process" }, "process + no follow-up": { writeup: "process", followUp: 0.5 },
  "process + weekly follow-up": { writeup: "process", followUp: 1.35 } };
const R = Object.fromEntries(Object.entries(K).map(([a, y]) => [a, measure(y)]));
const T = R["process writeup"];

console.log(`${PERIOD_COUNT} periods x ${PERIOD} days, ${CLASS.length} cause classes, ` +
  `${ITEM} items per review; seed ${SEED}`);

console.log("\n1. which record layer answers how many questions (process writeup run)");
const A = [-21, 10, 14, 10, 16];
write(A, "layer", "unit", "cumul. bytes", "answer/8", "bytes/question");
let field = [], bytes = 0;
for (const [name, opens, b, unit] of LAYER) {
  field = field.concat(opens);
  bytes += b * (unit === "incident" ? T.incident : T.opened);
  const y = QUESTION.filter(([, g]) => g.every((f) => field.includes(f))).length;
  write(A, name, unit, bytes, `${y}/8`, Math.round(bytes / y));
}
console.log("  unanswered: " + QUESTION.filter(([, g]) => g.includes("counterfactual")).map(([s]) => s)
  .join("") + " — counterfactual, in no layer");

console.log("\n2. action item tracking and item type");
const B = [-28, 10, 8, 12, 9, 9, 8, 11, 12];
write(B, "run", "incident", "opened", "per period", "closed", "closed", "days", "systemic", "sys. share");
for (const [name, r] of Object.entries(R))
  write(B, name, r.incident, r.opened, r.itemsPerPeriod, r.closed, r.closeRate, r.closeTime,
    `${r.systemic} (${(r.systemic / r.incident).toFixed(2)})`, r.systemicShare);   // systemic: count (per incident)

console.log("\n3. unclosed items and recurring incidents");
const C = [-28, 9, 8, 11, 8, 15, 17];
write(C, "run", "repeat", "rate", "recurring", "rate", "recur. outage", "unclosed sys.");
for (const [name, r] of Object.entries(R))
  write(C, name, r.repeat, r.repeatRate, r.recurring, r.recurringRate, r.recurringOutage + " min",
    `${r.systemic - r.systemicClosed}/${r.systemic}`);

console.log("\n4. the cost of the review and the answer to the unanswered question");
const D = [-28, 12, 14, 11, 14];
write(D, "run", "person-hrs", "hrs/incident", "outage", "repeat min");
for (const [name, r] of Object.entries(R))
  write(D, name, r.hours.toFixed(0), (r.hours / r.incident).toFixed(1), r.outage + " min",
    `${r.repeatOutage} (${(100 * r.repeatOutage / r.outage).toFixed(0)}%)`);

const ms = Number(process.hrtime.bigint() - t0) / 1e6;   // the raw measurement is not printed, only the band is checked
console.log(`\nfive runs and the layer sweep within the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`);

18 periods x 30 days, 5 cause classes, 3 items per review; seed 20260812

1. which record layer answers how many questions (process writeup run)
layer                      unit  cumul. bytes  answer/8  bytes/question
incident record        incident          6000       1/8            6000
+ review note          incident         51000       2/8           25500
+ writeup field        incident         51600       3/8           17200
+ item log                 item         75600       6/8           12600
+ item-class link          item         76800       7/8           10971
  unanswered: what outage did closing the item prevent — counterfactual, in no layer

2. action item tracking and item type
run                           incident  opened  per period   closed   closed    days   systemic  sys. share
no review                           48       0         0.0        0        -       -   0 (0.00)           -
person writeup                      32      96         5.3       81      84%     6.9  15 (0.47)         16%
process writeup                     25      75         4.2       52      69%    15.5  42 (1.68)         56%
process + no follow-up              38     114         6.3       38      33%    14.0  61 (1.61)         54%
process + weekly follow-up          23      69         3.8       60      87%    16.5  40 (1.74)         58%

3. unclosed items and recurring incidents
run                            repeat    rate  recurring    rate  recur. outage    unclosed sys.
no review                          43     90%          0      0%          0 min              0/0
person writeup                     27     84%          8     25%        335 min             5/15
process writeup                    20     80%         14     56%        593 min            16/42
process + no follow-up             33     87%         32     84%       1105 min            46/61
process + weekly follow-up         18     78%          8     35%        433 min             8/40

4. the cost of the review and the answer to the unanswered question
run                           person-hrs  hrs/incident     outage    repeat min
no review                              0           0.0   1791 min    1583 (88%)
person writeup                       376          11.7   1173 min     965 (82%)
process writeup                      480          19.2    966 min     758 (78%)
process + no follow-up               404          10.6   1353 min    1145 (85%)
process + weekly follow-up           556          24.2    878 min     670 (76%)

five runs and the layer sweep within the 2000 ms band: yes

The numbers are of the measurement class; their inputs are the assumptions above.

Seven of the Eight Questions

The first table turns record layers into questions. With just the incident record, one of eight questions is answered in 6,000 bytes. The review note takes it to 51,000 bytes and the answer count only goes to two — a note is not a field but a text, it is expensive, and it cannot be counted on its own. The writeup field opens the third question at 600 bytes, the item log opens three questions at once at 24,000 bytes; bytes per question fall from 17,200 to 12,600. The last layer is a 16-byte field linking the item to its cause class: at 1,200 bytes, it opens the which incident did an unclosed item reproduce question. The cheapest layer is the last one added, and it answers the most expensive question.

The eighth question is in no layer: what outage did closing this item prevent. Its answer is an unobserved state, and it cannot be answered by widening the record. The closest thing the log gives is “the unclosed item produced this many incidents” — what is counted is what was not prevented, not what was.

What the Close Rate Does Not Measure

At first glance, the second table makes the person writeup look good: 96 items open, 81 close (84%), and closing takes 6.9 days — the fastest of the five runs. In the process writeup, 52 of 75 items close (69%), and closing takes 15.5 days. The last two columns tell the difference. Of the person writeup’s 96 items, only 15 are systemic fixes (16%); the rest are reminders and documentation. Of the process writeup’s 75 items, 42 are systemic (56%); 0.47 per incident versus 1.68. A reminder closes in two days and changes no probability; a systemic fix closes in twenty-five days and multiplies its class’s incident probability by 0.70. The close rate does not measure what was closed.

The third and fourth tables give the result. In the person writeup, 32 incidents and 1,173 minutes of outage are measured; the repeat rate is 84% — six points below the no-review run’s 90%. In the process writeup, incidents drop to 25, outage to 966 minutes, repeats to 80%. This is the measurable counterpart of blame: when the writeup is a person, items open, close fast, and the system stays the same.

The Item Opened and the Item Tracked

The third run drops follow-up: the root cause is still written as process, but the closing probability is halved. Opened items do not fall — the opposite, they rise from 75 to 114, because more incidents happen and more reviews take place. Closed items stay at 38 (33%). The critical number is in the systemic items: 61 fixes open, and 46 never close. The recurring incident rate rises to 84%, recurrence outage rises to 1,105 minutes, and total outage climbs from 966 to 1,353 minutes.

Weekly follow-up does the opposite: 60 of 69 items close (87%), 32 of 40 systemic items close, and only 8 stay open. Recurring incidents fall from 14 to 8, total outage falls to 878 minutes. Its cost is in the fourth table: person-hours rise from 480 to 556, to 24.2 person-hours per incident — the most expensive of the five runs. What is being paid for is not the review, but the tracking of the review.

In the same table, the no-review run looks free: zero person-hours. Its counterpart is 1,791 minutes, and 1,583 minutes of that (88%) comes from repeat incidents.

Where the Difference Hides

The difference hides between the item opened and the systemic item closed: in the no-follow-up run, the log shows 114 rows, and 6.3 items per period are reported — the highest output of the five runs. In the same run, 46 of 61 systemic items never close, and 84% of incidents happen while an unclosed item sits open. The reported part is the count of items opened; the unreported part is the item’s type and status.

In the person writeup, only 5 systemic items go unclosed, and the recurring incident rate looks low at 25%. This is not good news — because only 15 systemic items were opened at all, an incident recurring because of an unclosed item cannot rack up much either. An item never opened does not show up as an unclosed item either. The real number there is the repeat rate: 84%.

Summary

• Seven of eight questions are answered by five record layers and 76,800 bytes (10,971 bytes per question); the last field, 16 bytes linking the item to its cause class, opens the most expensive question at 1,200 bytes.
• The question left unanswered is “what outage did closing this item prevent”; the counterfactual is in no layer.
• When the root cause is written as a person, 84% of 96 items close, and fastest of all (6.9 days), but only 15 are systemic fixes (0.47 per incident) and the repeat rate stays at 84% — close to the no-review run’s 90%. When written as a process, 42 of 75 items are systemic (1.68) and incidents drop to 25.
• When follow-up is dropped, 114 items open but only 38 close; 46 of 61 systemic items never close, the recurring incident rate rises to 84%, and recurrence outage rises to 1,105 minutes.
• Weekly follow-up drops outage to 878 minutes at a cost of 556 person-hours; the no-review run’s cost is zero person-hours and 1,791 minutes.

Next Step

Once an item closed, the incident probability dropped, but all of it ran after the incident had already happened: a systemic fix closes in twenty-five days, and the same class kept producing incidents in the meantime. No rule was ever put in place to regulate the pace of change before the incident. The next lesson turns the error budget into exactly such a rule: when the budget runs out, new deployment stops — what happens to deployment frequency, what piles up in the queue, how long it takes to recover the budget, and how the rule is loosened when a single major incident stops the line for months.