Error Budget Management

The previous lesson counted action items and showed that an unclosed one reproduces the same incident. Every fix ran after the incident had already happened; no rule was ever put in place to regulate the pace of change before the incident.

The error budget itself was measured in M22/K01: derived from the target, with how much a single major incident takes out of it counted. Here the budget becomes a rule — when it runs out, new deployment stops — and the rule itself is measured: deploy frequency, the piled-up work in the queue, the budget’s recovery time, and its triggering by a single incident. The rule is then loosened, and each loosening’s effect on deploy frequency and incident count is counted.

Setting Up the Rule

The example is once more the fictional regional measurement network; both the deployment pipeline and the incident sequence are fictional.

IM25: 360 days are run, and the budget operates on a rolling window: each day, the last 90 days’ outage is deducted from it. The window budget is 129.6 minutes, derived in M22/K01 and taken here as an input. IM26: One change enters the pipeline daily; the batch floor is five, prep one day, the batch cap forty. IM27: A quarter of changes are tagged risky; defect probability is 0.09 for risky ones, 0.014 for the rest. IM28: A defective change that ships causes an incident that same day; classes are minor 8 minutes (share 0.58), moderate 35 (0.30), major 150 (0.12). IM29: Every change above the batch floor grows the defect probability by a coefficient of 0.04. IM30: While the rule is active, full freeze ships no batch at all, partial freeze extends prep to ten days, only risky leaves risky-tagged changes queued, gradual reopen drops the batch cap to ten. The seed is 20260819, and every run shares the same change sequence.

Days, changes, and incidents are a model internal to the course; there is no real deployment pipeline.

// budget/rule.mjs — the error budget rule model: there is no real deployment pipeline; days,
// changes, and incidents are data structures. The generator is our own, the seed is visible.

export const DAY = 360, WINDOW = 90, DAY_MIN = 1440, TARGET = 0.999, SEED = 20260819;
export const BUDGET = WINDOW * DAY_MIN * (1 - TARGET);   // derived in M22/K01, an input here
export const BATCH = 5, BATCH_CAP = 40, PREP = 1, PARTIAL_PREP = 10;
export const COUPLING = 0.04;   // each extra change in a batch grows the defect probability by this much
export const RISK_SHARE = 0.25, DEFECT_RISKY = 0.09, DEFECT_PLAIN = 0.014;
export const CLASS = [["minor", 8, 0.58], ["moderate", 35, 0.30], ["major", 150, 0.12]];

export function generator(seed) {                    // linear congruential generator
  let x = seed >>> 0;
  return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32);
}
// dice are rolled up front: every run shares the same change sequence
const Z = (() => { const z = generator(SEED);
  return Array.from({ length: DAY }, () => ({ risk: z(), defect: z(), class: z() })); })();

export function run({ rule = "none", cap = BATCH_CAP } = {}) {
  const D = Z.map((z, i) => {                      // one change enters the pipeline every day
    const risky = z.risk < RISK_SHARE;
    let e = 0, minute = 0;
    for (const [, outage, share] of CLASS) { e += share; if (minute === 0 && z.class < e) minute = outage; }
    return { i, risky, die: z.defect, base: risky ? DEFECT_RISKY : DEFECT_PLAIN,
      outage: minute, status: "queued", releaseDay: null };
  });
  const incidents = [], episodes = [];
  const S = { release: 0, frozenDays: 0, queueTotal: 0, maxQueue: 0,
    batchTotal: 0, maxBatch: 0 };
  let active = null, start = null, single = false;

  for (let t = 0; t < DAY; t++) {
    if (active && active.done === t) {               // the batch ships
      S.release++; S.batchTotal += active.group.length;
      S.maxBatch = Math.max(S.maxBatch, active.group.length);
      // the bigger the batch, the more the coupling coefficient grows the defect probability
      const factor = 1 + COUPLING * (active.group.length - BATCH);
      for (const x of active.group) {
        x.status = "done"; x.releaseDay = t;
        if (x.die < x.base * factor) incidents.push({ day: t, outage: x.outage });
      }
      active = null;
    }
    // rolling window: the outage from the last WINDOW days is deducted from the budget
    const win = incidents.filter((o) => o.day > t - WINDOW);
    const left = BUDGET - win.reduce((s, o) => s + o.outage, 0);
    const triggered = rule !== "none" && left < 0;   // has the budget run out?
    if (triggered) {
      if (start === null) { start = t; single = win.some((o) => o.outage >= BUDGET); }
      S.frozenDays++;
    } else if (start !== null) { episodes.push({ length: t - start, single }); start = null; }

    const queue = D.filter((x) => x.status === "queued" && x.i <= t);
    S.queueTotal += queue.length;
    S.maxQueue = Math.max(S.maxQueue, queue.length);
    if (active) continue;
    let batch = BATCH, prep = PREP, candidate = queue;
    if (triggered) {                                  // the ways the rule can be loosened
      if (rule === "full") candidate = [];
      else if (rule === "partial") prep = PARTIAL_PREP;
      else if (rule === "risky") candidate = queue.filter((x) => !x.risky);
    }
    if (candidate.length < batch) continue;
    const group = candidate.slice(0, Math.min(candidate.length, cap));
    for (const x of group) x.status = "releasing";
    active = { group, done: t + prep };
  }
  if (start !== null) episodes.push({ length: DAY - start, single });
  return { D, incidents, episodes, S };
}

const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-");

export function measure(settings) {
  const { D, incidents, episodes, S } = run(settings);
  const L = episodes.map((e) => e.length);
  return { release: S.release, deploys: (S.release / (DAY / 30)).toFixed(1),
    incident: incidents.length, outage: incidents.reduce((s, o) => s + o.outage, 0),
    major: incidents.filter((o) => o.outage === CLASS[2][1]).length,
    episode: episodes.length, frozenDays: S.frozenDays,
    longest: L.length ? Math.max(...L) : 0, recovery: avg(L),
    singleIncident: episodes.filter((e) => e.single).length,
    avgQueue: (S.queueTotal / DAY).toFixed(1), maxQueue: S.maxQueue,
    avgBatch: (S.batchTotal / S.release).toFixed(1), maxBatch: S.maxBatch,
    stuck: D.filter((x) => x.releaseDay === null).length };
}

The measurement file first sets up a six-question set — questions genuinely asked in operation while the rule is active — and writes which record answers each question.

// budget/measure.mjs — the budget is turned into a rule and run over a period: deploy frequency,
// piled-up work, the budget's recovery time, and the rule's early trigger are counted.
import { DAY, WINDOW, BUDGET, TARGET, CLASS, SEED, COUPLING, BATCH, measure } from "./rule.mjs";

const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const t0 = process.hrtime.bigint();

const K = { "no rule": { rule: "none" }, "full freeze": { rule: "full" },
  "partial freeze": { rule: "partial" }, "only risky halts": { rule: "risky" },
  "full + gradual reopen": { rule: "full", cap: 10 } };
const R = Object.fromEntries(Object.entries(K).map(([a, y]) => [a, measure(y)]));
const T = R["full freeze"];

// question set: the record needed to answer each question, unit bytes, and unit count
const QUESTION = [
  ["how much budget is left right now", "daily outage record", 48, DAY],
  ["why did the line freeze", "freeze log", 96, T.episode],
  ["when will the line reopen", "window drop-off calendar", 32, T.incident],
  ["how many changes did the freeze pile up", "queue record", 24, DAY],
  ["which change was marked risky", "risk tag", 8, DAY],
  ["which incident did the freeze prevent", "counterfactual", 0, 0],
];

console.log(`${DAY} days, rolling window ${WINDOW} days, target ${(100 * TARGET).toFixed(1)}%, ` +
  `window budget ${BUDGET.toFixed(1)} min; seed ${SEED}`);
console.log("incident classes: " + CLASS.map(([a, d, p]) => `${a} ${d} min (share ${p})`).join(", "));
console.log(`major incident / budget = ${(100 * CLASS[2][1] / BUDGET).toFixed(0)}% (a single ` +
  `incident exhausts the budget); batch floor ${BATCH}, coupling coefficient ${COUPLING}`);

console.log("\n1. which question is answered by which record (full freeze run)");
const A = [-41, -26, 10, 8];
write(A, "question", "record needed", "bytes", "answer");
let bytes = 0, answer = 0;
for (const [s, k, b, n] of QUESTION) {
  const needed = b > 0;
  if (needed) { bytes += b * n; answer++; }
  write(A, s, k, needed ? b * n : "-", needed ? "yes" : "no");
}
console.log(`  total ${bytes} bytes, ${answer}/6 questions, ` +
  `${Math.round(bytes / answer)} bytes per question`);

console.log("\n2. how the rule regulates the pace of change");
const B = [-23, 10, 13, 11, 7, 11];
write(B, "run", "releases", "deploys/30d", "incidents", "major", "outage");
for (const [name, r] of Object.entries(R))
  write(B, name, r.release, r.deploys, r.incident, r.major, r.outage + " min");

console.log("\n3. freeze episodes and recovering the budget");
const C = [-23, 10, 13, 11, 15, 17];
write(C, "run", "episodes", "frozen days", "longest", "recovery", "single-incident");
for (const [name, r] of Object.entries(R))
  write(C, name, r.episode, r.frozenDays, r.longest + " d", r.recovery + " d",
    `${r.singleIncident}/${r.episode}`);

console.log("\n4. piled-up work");
const D = [-23, 13, 15, 12, 16, 11];
write(D, "run", "avg queue", "max queue", "avg batch", "max batch", "stuck");
for (const [name, r] of Object.entries(R))
  write(D, name, r.avgQueue, r.maxQueue, r.avgBatch, r.maxBatch, r.stuck);

const ms = Number(process.hrtime.bigint() - t0) / 1e6;   // the raw measurement is not printed, only the band is checked
console.log(`\nfive runs and the question sweep within the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`);

360 days, rolling window 90 days, target 99.9%, window budget 129.6 min; seed 20260819
incident classes: minor 8 min (share 0.58), moderate 35 min (share 0.3), major 150 min (share 0.12)
major incident / budget = 116% (a single incident exhausts the budget); batch floor 5, coupling coefficient 0.04

1. which question is answered by which record (full freeze run)
question                                 record needed                  bytes  answer
how much budget is left right now        daily outage record            17280     yes
why did the line freeze                  freeze log                       384     yes
when will the line reopen                window drop-off calendar         576     yes
how many changes did the freeze pile up  queue record                    8640     yes
which change was marked risky            risk tag                        2880     yes
which incident did the freeze prevent    counterfactual                     -      no
  total 29760 bytes, 5/6 questions, 5952 bytes per question

2. how the rule regulates the pace of change
run                      releases  deploys/30d  incidents  major     outage
no rule                        71          5.9          8      2    402 min
full freeze                    22          1.8         18      4    901 min
partial freeze                 58          4.8          9      2    437 min
only risky halts               62          5.2         11      2    480 min
full + gradual reopen          51          4.3          9      2    437 min

3. freeze episodes and recovering the budget
run                      episodes  frozen days    longest       recovery  single-incident
no rule                         0            0        0 d            - d              0/0
full freeze                     4          274       90 d         68.5 d              4/4
partial freeze                  1          134      134 d        134.0 d              1/1
only risky halts                2          180       90 d         90.0 d              2/2
full + gradual reopen           2          180       90 d         90.0 d              2/2

4. piled-up work
run                        avg queue      max queue   avg batch       max batch      stuck
no rule                          3.0              5         5.0               5          5
full freeze                     65.6            154        14.5              40         40
partial freeze                   3.9             10         6.1              10          5
only risky halts                 9.6             31         5.8              31          3
full + gradual reopen           44.4            154         7.0              10          5

five runs and the question sweep within the 2000 ms band: yes

The numbers are of the measurement class; their inputs are the assumptions above.

The Questions the Rule Asks

The first table ties the six questions asked while the rule is active to a record. Five are answered, totaling 29,760 bytes; 5,952 bytes per question. The two expensive ones are kept per day: the daily outage record (17,280 bytes) and the queue record (8,640 bytes). The cheap one is when will the line reopen: 576 bytes. The rolling window turns this question into an exact date, because each incident’s drop-out day is known. Without the calendar, the answer is “we do not know”, asked again for every change in the queue.

The sixth is the one left unanswered: which incident did the freeze prevent. Here too, no record answers it. While the rule is active, outage is measured; prevented outage is not.

The Budget Runs Out on a Single Incident

The third table tells how the rule gets triggered. A major incident is 150 minutes, the window budget is 129.6 minutes: a single major incident takes out 116% of the budget. The last column gives the result: in the full freeze, all four freeze episodes are triggered by a single incident; the rule responds to a single incident, not to accumulated outage.

The recovery time falls out from this. In a rolling window, the budget returns only once the incident drops out of it; a single major incident alone exhausts the budget, so the line stays closed for ninety days — the longest freeze is exactly the window’s length. In the full freeze, the average episode is 68.5 days, and frozen days are 274: three quarters of the 360 days. In the partial freeze, the single episode lasts 134 days, because the slowed-down line keeps adding new incidents to the window.

The Rule Increases Outage

The row worth reading in the second table is the full freeze: deploy frequency over 30 days drops from 5.9 to 1.8, exactly as intended. But incidents rise from 8 to 18, outage from 402 to 901 minutes, major incidents from 2 to 4.

The reason is in the fourth table. With no rule, the average queue is 3.0 and the average batch 5.0. In the full freeze, the queue averages 65.6, peaks at 154, and when the line reopens it drains in batches of 40 — an average batch of 14.5. At a coupling coefficient of 0.04, a batch of 40 carries 2.4 times the floor defect probability. A freeze does not eliminate change, it piles it up; the piled-up work comes out as a large batch, and a large batch produces more incidents. New incidents trigger new freezes: episodes climb to 4, and forty changes never ship.

Three Ways to Loosen the Rule

The partial freeze does not stop the line, it slows it down: prep rises to ten days. Deploy frequency stays at 4.8, the average queue is 3.9, and outage stops at 437 minutes — close to the no-rule run’s 402 minutes. Its cost is a single 134-day slowdown episode.

Freezing only risky changes sits in the middle: deploy frequency 5.2, queue 9.6, outage 480 minutes. It is worse than the partial freeze, because the waiting risky changes eventually ship together in a batch of 31. The tag itself costs 2,880 bytes; what it buys is the queue staying at 31 instead of 154.

The third loosening touches not the rule but its reopening: the full freeze is kept, but the batch cap is pulled from 40 to 10 as the line reopens. The same rule, the same threshold, the same 90-day longest freeze — but frozen days drop from 274 to 180, episodes from 4 to 2, and the average batch drops from 14.5 to 7.0, incidents from 18 to 9, and outage from 901 to 437 minutes. Deploy frequency rises from 1.8 to 4.3, and changes that never ship drop from 40 to 5. The rule’s cost is set not by how long the freeze lasts, but by the shape of the exit from it.

Where the Difference Hides

The difference hides between how long the freeze lasts and how big the post-freeze batch is: the full freeze and the gradual reopen share the same rule, the same threshold, and the same 90-day longest freeze, and the max queue is 154 in both. The reported metric is the longest freeze, identical between the two; the unreported part is the batch cap, which alone splits outage between 901 and 437 minutes.

The major incident’s share is only 0.12, but it takes out 116% of the budget on its own; whether the rule triggers or not depends almost entirely on that single class. The rule reads not accumulated outage, but the size of one incident.

Summary

• In the rolling window, the budget is 129.6 minutes and a major incident is 150: a single incident takes out 116% of the budget; in the full freeze, all four episodes are triggered by a single incident, and the longest freeze is 90 days — exactly the window’s length.
• The full freeze drops deploy frequency from 5.9 to 1.8, but because the queue climbs to 154 and drains in batches of 40, incidents rise from 8 to 18, outage from 402 to 901 minutes, and 40 changes never ship at all.
• The partial freeze (prep 1 → 10 days) holds deploy frequency at 4.8, the queue at 3.9, and outage at 437 minutes; its cost is a single 134-day slowdown episode.
• Freezing only risky changes holds deploy frequency at 5.2, but outage is 480 minutes, because the waiting risky changes ship together in a single batch of 31.
• When the rule is kept but the batch cap is pulled from 40 to 10 at reopening, frozen days drop from 274 to 180, outage from 901 to 437 minutes, and incidents from 18 to 9; deploy frequency rises to 4.3.
• Five of six questions are answered with 29,760 bytes (5,952 bytes per question); “which incident did the freeze prevent” is answered by no record.

Next Step

The rule was put in place, the line froze, the queue piled up, and the shape of the exit either doubled the outage or cut it in half. All these numbers are in minutes and changes. The frozen line, the piled-up queue, and the all-night batch job also have a resource counterpart. The next lesson treats capacity as an operating cost — how resource usage turns into a price, which question that price answers, and how much the collected monitoring data costs on its own.