--- title: 'Error Budget Management' source: 'https://academia.sh/en/courses/devops-observability/error-budget-management' course: 'Observability and Operations' language: en updated: '2026-09-13T05:08:21+00:00' license: 'CC BY-SA 4.0' objectives: - 'Turns the error budget into a freeze rule that operates on a rolling window' - "Counts the rule's cost: deploy frequency, piled-up work, the budget's recovery time, and a freeze triggered by a single incident" - 'Compares the ways the rule can be loosened, by deploy frequency and incident count' --- # Error Budget Management Turning the error budget into a rule that regulates the pace of change: over 360 days, in a 90-day rolling window, a 129.6-minute budget is exhausted by a single major incident (150 minutes, 116% of the budget), and all four freeze episodes are triggered by a single incident; a full freeze drops deploy frequency from 5.9 to 1.8 while draining a queue of 154 in batches of 40, raising outage from 402 to 901 minutes; a partial freeze holds at 437 minutes and a queue of 3.9; freezing only risky changes gives 480 minutes; and a gradual reopen keeps the same rule while dropping outage to 437 minutes. The previous lesson counted action items and showed that an unclosed one reproduces the same incident. Every fix ran after the incident had already happened; no rule was ever put in place to regulate the pace of change before the incident. The error budget itself was measured in M22/K01: derived from the target, with how much a single major incident takes out of it counted. Here the budget becomes a **rule** — when it runs out, new deployment stops — and the rule itself is measured: deploy frequency, the **piled-up work** in the queue, the budget's **recovery time**, and its **triggering by a single incident**. The rule is then loosened, and each loosening's effect on deploy frequency and incident count is counted. ## Setting Up the Rule The example is once more the fictional **regional measurement network**; both the deployment pipeline and the incident sequence are fictional. **IM25:** 360 days are run, and the budget operates on a **rolling window**: each day, the last 90 days' outage is deducted from it. The window budget is 129.6 minutes, derived in M22/K01 and taken here as an input. **IM26:** One change enters the pipeline daily; the batch floor is five, prep one day, the batch cap forty. **IM27:** A quarter of changes are tagged risky; defect probability is 0.09 for risky ones, 0.014 for the rest. **IM28:** A defective change that ships causes an incident that same day; classes are minor 8 minutes (share 0.58), moderate 35 (0.30), major 150 (0.12). **IM29:** Every change above the batch floor grows the defect probability by a coefficient of 0.04. **IM30:** While the rule is active, *full freeze* ships no batch at all, *partial freeze* extends prep to ten days, *only risky* leaves risky-tagged changes queued, *gradual reopen* drops the batch cap to ten. The seed is 20260819, and every run shares the same change sequence. Days, changes, and incidents are a **model** internal to the course; there is no real deployment pipeline. ```js // budget/rule.mjs — the error budget rule model: there is no real deployment pipeline; days, // changes, and incidents are data structures. The generator is our own, the seed is visible. export const DAY = 360, WINDOW = 90, DAY_MIN = 1440, TARGET = 0.999, SEED = 20260819; export const BUDGET = WINDOW * DAY_MIN * (1 - TARGET); // derived in M22/K01, an input here export const BATCH = 5, BATCH_CAP = 40, PREP = 1, PARTIAL_PREP = 10; export const COUPLING = 0.04; // each extra change in a batch grows the defect probability by this much export const RISK_SHARE = 0.25, DEFECT_RISKY = 0.09, DEFECT_PLAIN = 0.014; export const CLASS = [["minor", 8, 0.58], ["moderate", 35, 0.30], ["major", 150, 0.12]]; export function generator(seed) { // linear congruential generator let x = seed >>> 0; return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32); } // dice are rolled up front: every run shares the same change sequence const Z = (() => { const z = generator(SEED); return Array.from({ length: DAY }, () => ({ risk: z(), defect: z(), class: z() })); })(); export function run({ rule = "none", cap = BATCH_CAP } = {}) { const D = Z.map((z, i) => { // one change enters the pipeline every day const risky = z.risk < RISK_SHARE; let e = 0, minute = 0; for (const [, outage, share] of CLASS) { e += share; if (minute === 0 && z.class < e) minute = outage; } return { i, risky, die: z.defect, base: risky ? DEFECT_RISKY : DEFECT_PLAIN, outage: minute, status: "queued", releaseDay: null }; }); const incidents = [], episodes = []; const S = { release: 0, frozenDays: 0, queueTotal: 0, maxQueue: 0, batchTotal: 0, maxBatch: 0 }; let active = null, start = null, single = false; for (let t = 0; t < DAY; t++) { if (active && active.done === t) { // the batch ships S.release++; S.batchTotal += active.group.length; S.maxBatch = Math.max(S.maxBatch, active.group.length); // the bigger the batch, the more the coupling coefficient grows the defect probability const factor = 1 + COUPLING * (active.group.length - BATCH); for (const x of active.group) { x.status = "done"; x.releaseDay = t; if (x.die < x.base * factor) incidents.push({ day: t, outage: x.outage }); } active = null; } // rolling window: the outage from the last WINDOW days is deducted from the budget const win = incidents.filter((o) => o.day > t - WINDOW); const left = BUDGET - win.reduce((s, o) => s + o.outage, 0); const triggered = rule !== "none" && left < 0; // has the budget run out? if (triggered) { if (start === null) { start = t; single = win.some((o) => o.outage >= BUDGET); } S.frozenDays++; } else if (start !== null) { episodes.push({ length: t - start, single }); start = null; } const queue = D.filter((x) => x.status === "queued" && x.i <= t); S.queueTotal += queue.length; S.maxQueue = Math.max(S.maxQueue, queue.length); if (active) continue; let batch = BATCH, prep = PREP, candidate = queue; if (triggered) { // the ways the rule can be loosened if (rule === "full") candidate = []; else if (rule === "partial") prep = PARTIAL_PREP; else if (rule === "risky") candidate = queue.filter((x) => !x.risky); } if (candidate.length < batch) continue; const group = candidate.slice(0, Math.min(candidate.length, cap)); for (const x of group) x.status = "releasing"; active = { group, done: t + prep }; } if (start !== null) episodes.push({ length: DAY - start, single }); return { D, incidents, episodes, S }; } const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-"); export function measure(settings) { const { D, incidents, episodes, S } = run(settings); const L = episodes.map((e) => e.length); return { release: S.release, deploys: (S.release / (DAY / 30)).toFixed(1), incident: incidents.length, outage: incidents.reduce((s, o) => s + o.outage, 0), major: incidents.filter((o) => o.outage === CLASS[2][1]).length, episode: episodes.length, frozenDays: S.frozenDays, longest: L.length ? Math.max(...L) : 0, recovery: avg(L), singleIncident: episodes.filter((e) => e.single).length, avgQueue: (S.queueTotal / DAY).toFixed(1), maxQueue: S.maxQueue, avgBatch: (S.batchTotal / S.release).toFixed(1), maxBatch: S.maxBatch, stuck: D.filter((x) => x.releaseDay === null).length }; } ``` The measurement file first sets up a six-question set — questions genuinely asked in operation while the rule is active — and writes which record answers each question. ```js // budget/measure.mjs — the budget is turned into a rule and run over a period: deploy frequency, // piled-up work, the budget's recovery time, and the rule's early trigger are counted. import { DAY, WINDOW, BUDGET, TARGET, CLASS, SEED, COUPLING, BATCH, measure } from "./rule.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const t0 = process.hrtime.bigint(); const K = { "no rule": { rule: "none" }, "full freeze": { rule: "full" }, "partial freeze": { rule: "partial" }, "only risky halts": { rule: "risky" }, "full + gradual reopen": { rule: "full", cap: 10 } }; const R = Object.fromEntries(Object.entries(K).map(([a, y]) => [a, measure(y)])); const T = R["full freeze"]; // question set: the record needed to answer each question, unit bytes, and unit count const QUESTION = [ ["how much budget is left right now", "daily outage record", 48, DAY], ["why did the line freeze", "freeze log", 96, T.episode], ["when will the line reopen", "window drop-off calendar", 32, T.incident], ["how many changes did the freeze pile up", "queue record", 24, DAY], ["which change was marked risky", "risk tag", 8, DAY], ["which incident did the freeze prevent", "counterfactual", 0, 0], ]; console.log(`${DAY} days, rolling window ${WINDOW} days, target ${(100 * TARGET).toFixed(1)}%, ` + `window budget ${BUDGET.toFixed(1)} min; seed ${SEED}`); console.log("incident classes: " + CLASS.map(([a, d, p]) => `${a} ${d} min (share ${p})`).join(", ")); console.log(`major incident / budget = ${(100 * CLASS[2][1] / BUDGET).toFixed(0)}% (a single ` + `incident exhausts the budget); batch floor ${BATCH}, coupling coefficient ${COUPLING}`); console.log("\n1. which question is answered by which record (full freeze run)"); const A = [-41, -26, 10, 8]; write(A, "question", "record needed", "bytes", "answer"); let bytes = 0, answer = 0; for (const [s, k, b, n] of QUESTION) { const needed = b > 0; if (needed) { bytes += b * n; answer++; } write(A, s, k, needed ? b * n : "-", needed ? "yes" : "no"); } console.log(` total ${bytes} bytes, ${answer}/6 questions, ` + `${Math.round(bytes / answer)} bytes per question`); console.log("\n2. how the rule regulates the pace of change"); const B = [-23, 10, 13, 11, 7, 11]; write(B, "run", "releases", "deploys/30d", "incidents", "major", "outage"); for (const [name, r] of Object.entries(R)) write(B, name, r.release, r.deploys, r.incident, r.major, r.outage + " min"); console.log("\n3. freeze episodes and recovering the budget"); const C = [-23, 10, 13, 11, 15, 17]; write(C, "run", "episodes", "frozen days", "longest", "recovery", "single-incident"); for (const [name, r] of Object.entries(R)) write(C, name, r.episode, r.frozenDays, r.longest + " d", r.recovery + " d", `${r.singleIncident}/${r.episode}`); console.log("\n4. piled-up work"); const D = [-23, 13, 15, 12, 16, 11]; write(D, "run", "avg queue", "max queue", "avg batch", "max batch", "stuck"); for (const [name, r] of Object.entries(R)) write(D, name, r.avgQueue, r.maxQueue, r.avgBatch, r.maxBatch, r.stuck); const ms = Number(process.hrtime.bigint() - t0) / 1e6; // the raw measurement is not printed, only the band is checked console.log(`\nfive runs and the question sweep within the 2000 ms band: ${ms < 2000 ? "yes" : "no"}`); ``` ``` 360 days, rolling window 90 days, target 99.9%, window budget 129.6 min; seed 20260819 incident classes: minor 8 min (share 0.58), moderate 35 min (share 0.3), major 150 min (share 0.12) major incident / budget = 116% (a single incident exhausts the budget); batch floor 5, coupling coefficient 0.04 1. which question is answered by which record (full freeze run) question record needed bytes answer how much budget is left right now daily outage record 17280 yes why did the line freeze freeze log 384 yes when will the line reopen window drop-off calendar 576 yes how many changes did the freeze pile up queue record 8640 yes which change was marked risky risk tag 2880 yes which incident did the freeze prevent counterfactual - no total 29760 bytes, 5/6 questions, 5952 bytes per question 2. how the rule regulates the pace of change run releases deploys/30d incidents major outage no rule 71 5.9 8 2 402 min full freeze 22 1.8 18 4 901 min partial freeze 58 4.8 9 2 437 min only risky halts 62 5.2 11 2 480 min full + gradual reopen 51 4.3 9 2 437 min 3. freeze episodes and recovering the budget run episodes frozen days longest recovery single-incident no rule 0 0 0 d - d 0/0 full freeze 4 274 90 d 68.5 d 4/4 partial freeze 1 134 134 d 134.0 d 1/1 only risky halts 2 180 90 d 90.0 d 2/2 full + gradual reopen 2 180 90 d 90.0 d 2/2 4. piled-up work run avg queue max queue avg batch max batch stuck no rule 3.0 5 5.0 5 5 full freeze 65.6 154 14.5 40 40 partial freeze 3.9 10 6.1 10 5 only risky halts 9.6 31 5.8 31 3 full + gradual reopen 44.4 154 7.0 10 5 five runs and the question sweep within the 2000 ms band: yes ``` The numbers are of the **measurement** class; their inputs are the assumptions above. ## The Questions the Rule Asks The first table ties the six questions asked while the rule is active to a record. Five are answered, totaling 29,760 bytes; 5,952 bytes per question. The two expensive ones are kept per day: the daily outage record (17,280 bytes) and the queue record (8,640 bytes). The cheap one is **when will the line reopen**: 576 bytes. The rolling window turns this question into an exact date, because each incident's drop-out day is known. Without the calendar, the answer is "we do not know", asked again for every change in the queue. The sixth is the one left unanswered: which incident did the freeze prevent. Here too, no record answers it. While the rule is active, **outage is measured; prevented outage is not**. ## The Budget Runs Out on a Single Incident The third table tells how the rule gets triggered. A major incident is 150 minutes, the window budget is 129.6 minutes: **a single major incident takes out 116% of the budget**. The last column gives the result: in the full freeze, **all four** freeze episodes are triggered by a single incident; the rule responds to a single incident, not to accumulated outage. The recovery time falls out from this. In a rolling window, the budget returns only once the incident drops out of it; a single major incident alone exhausts the budget, so the line stays closed for **ninety days** — the longest freeze is exactly the window's length. In the full freeze, the average episode is 68.5 days, and frozen days are 274: three quarters of the 360 days. In the partial freeze, the single episode lasts 134 days, because the slowed-down line keeps adding new incidents to the window. ## The Rule Increases Outage The row worth reading in the second table is the full freeze: deploy frequency over 30 days drops from 5.9 to 1.8, exactly as intended. But incidents rise from 8 to **18**, outage from 402 to **901 minutes**, major incidents from 2 to 4. The reason is in the fourth table. With no rule, the average queue is 3.0 and the average batch 5.0. In the full freeze, the queue averages 65.6, peaks at 154, and when the line reopens it drains in **batches of 40** — an average batch of 14.5. At a coupling coefficient of 0.04, a batch of 40 carries 2.4 times the floor defect probability. **A freeze does not eliminate change, it piles it up; the piled-up work comes out as a large batch, and a large batch produces more incidents.** New incidents trigger new freezes: episodes climb to 4, and forty changes never ship. ## Three Ways to Loosen the Rule The partial freeze does not stop the line, it slows it down: prep rises to ten days. Deploy frequency stays at 4.8, the average queue is 3.9, and outage stops at 437 minutes — close to the no-rule run's 402 minutes. Its cost is a single 134-day slowdown episode. Freezing only risky changes sits in the middle: deploy frequency 5.2, queue 9.6, outage 480 minutes. It is worse than the partial freeze, because the waiting risky changes eventually ship together in a batch of 31. The tag itself costs 2,880 bytes; what it buys is the queue staying at 31 instead of 154. The third loosening touches not the rule but its **reopening**: the full freeze is kept, but the batch cap is pulled from 40 to 10 as the line reopens. The same rule, the same threshold, the same 90-day longest freeze — but frozen days drop from 274 to 180, episodes from 4 to 2, and the average batch drops from 14.5 to 7.0, incidents from 18 to 9, and outage from 901 to **437 minutes**. Deploy frequency rises from 1.8 to 4.3, and changes that never ship drop from 40 to 5. **The rule's cost is set not by how long the freeze lasts, but by the shape of the exit from it.** ## Where the Difference Hides The difference hides **between how long the freeze lasts and how big the post-freeze batch is**: the full freeze and the gradual reopen share the same rule, the same threshold, and the same 90-day longest freeze, and the max queue is 154 in both. The reported metric is the longest freeze, identical between the two; the unreported part is the batch cap, which alone splits outage between 901 and 437 minutes. The major incident's share is only 0.12, but it takes out 116% of the budget on its own; whether the rule triggers or not depends almost entirely on that single class. The rule reads not accumulated outage, but **the size of one incident**. ## Summary - In the rolling window, the budget is 129.6 minutes and a major incident is 150: a single incident takes out 116% of the budget; in the full freeze, all four episodes are triggered by a single incident, and the longest freeze is 90 days — exactly the window's length. - The full freeze drops deploy frequency from 5.9 to 1.8, but because the queue climbs to 154 and drains in batches of 40, incidents rise from 8 to 18, outage from 402 to 901 minutes, and 40 changes never ship at all. - The partial freeze (prep 1 → 10 days) holds deploy frequency at 4.8, the queue at 3.9, and outage at 437 minutes; its cost is a single 134-day slowdown episode. - Freezing only risky changes holds deploy frequency at 5.2, but outage is 480 minutes, because the waiting risky changes ship together in a single batch of 31. - When the rule is kept but the batch cap is pulled from 40 to 10 at reopening, frozen days drop from 274 to 180, outage from 901 to 437 minutes, and incidents from 18 to 9; deploy frequency rises to 4.3. - Five of six questions are answered with 29,760 bytes (5,952 bytes per question); "which incident did the freeze prevent" is answered by no record. ## Next Step The rule was put in place, the line froze, the queue piled up, and the shape of the exit either doubled the outage or cut it in half. All these numbers are in minutes and changes. The frozen line, the piled-up queue, and the all-night batch job also have a resource counterpart. The next lesson treats capacity as an operating cost — how resource usage turns into a price, which question that price answers, and how much the collected monitoring data costs on its own.