Site Reliability Engineering

The previous lesson counted the bill of three decisions measured across four metrics, but every decision was made against an implicit threshold: how much error is acceptable, and what stops when it becomes unacceptable. That threshold was never written down anywhere.

Site reliability engineering is the discipline that treats operational responsibility as an engineering problem and ties the reliability target to a number first. Where it overlaps with DevOps is shared responsibility — both approaches want development and operations held accountable for the same outcome. Where they diverge is the metric: the target is written down first, then an error budget is derived from that target, and delivery decisions are tied to that budget. This lesson ties the budget to a minute count and counts the rule’s effect measured across four metrics. Recovery mechanics were measured in the Resilience and Reliability course; here, time to restore is taken as an input.

Building the Budget

The example is again the fictional regional measurement network; the network and the periods are both fictional. CF26: a period is 30 days, that is 43200 minutes, and twelve periods are run. CF27: twenty changes enter the flow at equal intervals in every period; the batch is four and prep takes one day. CF28: two unconditional dice are rolled for every change — whether it is defective (rate 0.10) and its incident class (light 10 minutes, share 0.60; moderate 30 minutes, share 0.30; heavy 90 minutes, share 0.10); the seed is 20260805. CF29: if a defective change ships, it becomes an incident in production, and the incident’s outage minutes are deducted from the period’s budget. The budget is derived at the start of the period from the target availability: period minutes times one minus the target. CF30: if the budget rule is on, the moment the remaining budget drops below zero, no new release ships until the period ends; queued changes wait, and the budget only renews in the new period. CF31: lead time is the days from a change entering the flow to being released, time to restore is the incident’s outage minutes, availability is one minus the ratio of total outage to the total period.

The periods, the incidents, and the budget rule are a process-internal model; there is no real outage record.

// budget/period.mjs — the period model of the error budget (model): there is no real outage record,
// days and minutes are a data structure. Randomness is written with a generator, the seed is visible.

export const PERIOD_DAYS = 30, PERIODS = 12, DAY_MIN = 1440, CHANGES = 20, BATCH = 4, PREP = 1;
export const DEFECT = 0.10, SEED = 20260805;
// incident class: outage minutes and its share
export const CLASS = [["light", 10, 0.60], ["moderate", 30, 0.30], ["heavy", 90, 0.10]];

export function rng(seed) {                        // linear congruential generator
  let x = seed >>> 0;
  return () => ((x = (x * 1664525 + 1013904223) >>> 0) / 2 ** 32);
}

export const budgetMin = (target) => PERIOD_DAYS * DAY_MIN * (1 - target);

export function run({ target = 0.999, rule = true, outageMultiplier = 1 } = {}) {
  const roll = rng(SEED), budget = budgetMin(target);
  const D = Array.from({ length: PERIODS * CHANGES }, (_, i) => {
    const p = Math.floor(i / CHANGES), k = i % CHANGES;
    const defective = roll() < DEFECT, r = roll();
    let share = 0, min = 0;
    for (const [, m, sh] of CLASS) { share += sh; if (min === 0 && r < share) min = m; }
    return { i, arrival: p * PERIOD_DAYS + Math.floor(k * PERIOD_DAYS / CHANGES), defective,
      outage: min * outageMultiplier, status: "in-transit", releaseDay: null };
  });
  const S = { releases: 0, brokenReleases: 0, outage: 0, incidents: 0, frozenDays: 0, exhausted: 0,
    remainders: [], outages: [], periodReleases: Array(PERIODS).fill(0) };
  let active = null, remaining = budget, period = -1, frozen = false;

  for (let t = 0; t < PERIODS * PERIOD_DAYS; t++) {
    const p = Math.floor(t / PERIOD_DAYS);
    if (p !== period) {                            // the budget renews at the start of the period
      if (period >= 0) S.remainders.push(remaining);
      period = p; remaining = budget; frozen = false;
    }
    for (const x of D) if (x.status === "in-transit" && x.arrival <= t) x.status = "queued";
    if (active && active.finishAt === t) {          // a release reaches production
      S.releases++; S.periodReleases[p]++;
      const broken = active.group.filter((x) => x.defective);
      if (broken.length) {
        S.brokenReleases++;
        for (const x of broken) {
          S.incidents++; S.outage += x.outage; S.outages.push(x.outage);
          remaining -= x.outage;
        }
        if (rule && remaining < 0 && !frozen) { frozen = true; S.exhausted++; }
      }
      for (const x of active.group) { x.status = "done"; x.releaseDay = t; }
      active = null;
    }
    if (frozen) { S.frozenDays++; continue; }       // budget exhausted: no new release ships
    if (active) continue;
    const K = D.filter((x) => x.status === "queued");
    if (K.length < BATCH) continue;
    const group = K.slice(0, BATCH);
    for (const x of group) x.status = "releasing";
    active = { group, finishAt: t + PREP };
  }
  S.remainders.push(remaining);
  return { D, S, budget };
}

export const avg = (a) => (a.length ? (a.reduce((s, v) => s + v, 0) / a.length).toFixed(1) : "-");

export function measure(config) {
  const { D, S, budget } = run(config);
  const shipped = D.filter((x) => x.releaseDay !== null);
  const totalMin = PERIODS * PERIOD_DAYS * DAY_MIN;
  return { D, S, budget,
    deployFrequency: (S.releases / PERIODS).toFixed(1),
    leadTime: avg(shipped.map((x) => x.releaseDay - x.arrival)),
    changeFailureRate: (100 * S.brokenReleases / S.releases).toFixed(0) + "%",
    recovery: avg(S.outages),
    availability: (100 * (1 - S.outage / totalMin)).toFixed(3) + "%",
    outage: S.outage, incidents: S.incidents, releases: S.releases,
    exhausted: S.exhausted, frozenDays: S.frozenDays,
    avgRemaining: avg(S.remainders), unshipped: D.length - shipped.length };
}

// budget/measure.mjs — tying the error budget to a number; the budget rule's effect across four metrics.
import { PERIODS, PERIOD_DAYS, DAY_MIN, CLASS, SEED, DEFECT, budgetMin, measure } from "./period.mjs";

const print = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));

console.log(`${PERIODS} periods x ${PERIOD_DAYS} days = ${PERIODS * PERIOD_DAYS * DAY_MIN} minutes; ` +
  `defect rate ${DEFECT}, seed ${SEED}`);

const H = [0.995, 0.999, 0.9995];
const A = [-14, 16, 16, 18, 15];
console.log("\n1. how many minutes of budget a target availability leaves in a period");
print(A, "target", "budget (minutes)", "light incident", "moderate incident", "heavy incident");
for (const h of H) {
  const b = budgetMin(h);
  print(A, (100 * h).toFixed(2) + "%", b.toFixed(1),
    ...CLASS.map(([, dk]) => `${(100 * dk / b).toFixed(0)}%`));
}
console.log("  (last three columns: the share a single incident takes from the budget)");
console.log("  incident classes: " +
  CLASS.map(([ad, dk, p]) => `${ad} ${dk} min (share ${p})`).join(", "));

const K = {
  "99.9% no rule": { target: 0.999, rule: false },
  "99.9% rule on": { target: 0.999, rule: true },
  "99.5% rule on": { target: 0.995, rule: true },
  "99.95% rule on": { target: 0.9995, rule: true },
  "99.9% outage halved": { target: 0.999, rule: true, outageMultiplier: 0.5 },
};
const R = Object.fromEntries(Object.entries(K).map(([ad, a]) => [ad, measure(a)]));

const B = [-22, 15, 12, 17, 12];
console.log("\n2. the budget rule's effect measured across four delivery metrics");
print(B, "run", "deploy", "lead time", "change failure", "recovery");
print(B, "", "(period/release)", "(days)", "(release)", "(minutes)");
for (const [ad, r] of Object.entries(R))
  print(B, ad, r.deployFrequency, r.leadTime, r.changeFailureRate, r.recovery);

const C = [-22, 10, 11, 17, 20, 14];
console.log("\n3. the state of the budget in the same runs");
print(C, "run", "outage", "incidents", "availability", "periods exhausted", "frozen days");
for (const [ad, r] of Object.entries(R))
  print(C, ad, r.outage, r.incidents, r.availability, `${r.exhausted}/${PERIODS}`, r.frozenDays);

const E = [-22, 8, 24, 20, 13];
console.log("\n4. releases and the queue");
print(E, "run", "releases", "avg. remaining budget", "unshipped changes", "target held");
for (const [ad, r] of Object.entries(R)) {
  const h = K[ad].target;
  print(E, ad, r.releases, r.avgRemaining, r.unshipped,
    +r.availability.replace("%", "") >= 100 * h ? "yes" : "no");
}

12 periods x 30 days = 518400 minutes; defect rate 0.1, seed 20260805

1. how many minutes of budget a target availability leaves in a period
target        budget (minutes)  light incident moderate incident heavy incident
99.50%                   216.0              5%               14%            42%
99.90%                    43.2             23%               69%           208%
99.95%                    21.6             46%              139%           417%
  (last three columns: the share a single incident takes from the budget)
  incident classes: light 10 min (share 0.6), moderate 30 min (share 0.3), heavy 90 min (share 0.1)

2. the budget rule's effect measured across four delivery metrics
run                            deploy   lead time   change failure    recovery
                      (period/release)      (days)        (release)   (minutes)
99.9% no rule                     5.0         3.0              35%        23.3
99.9% rule on                     4.8        12.6              34%        23.8
99.5% rule on                     5.0         3.0              35%        23.3
99.95% rule on                    3.8        49.9              29%        27.5
99.9% outage halved               5.0         5.4              35%        11.7

3. the state of the budget in the same runs
run                       outage  incidents     availability   periods exhausted   frozen days
99.9% no rule                630         27          99.878%                0/12             0
99.9% rule on                620         26          99.880%                8/12           166
99.5% rule on                630         27          99.878%                0/12             0
99.95% rule on               440         16          99.915%                9/12           236
99.9% outage halved          315         27          99.939%                4/12            65

4. releases and the queue
run                   releases   avg. remaining budget   unshipped changes  target held
99.9% no rule               60                    -9.3                   0           no
99.9% rule on               58                    -8.5                   8           no
99.5% rule on               60                   163.5                   0          yes
99.95% rule on              45                   -15.1                  60           no
99.9% outage halved         60                    17.0                   0          yes

The numbers are of the measurement kind; their inputs are the assumptions above.

How Much of the Budget One Incident Takes

The first table converts the target directly into minutes. In a thirty-day period, a 99.5% target leaves 216.0 minutes, a 99.9% target leaves 43.2 minutes, a 99.95% target leaves 21.6 minutes. The three columns on the right are the share a single incident takes from this budget, and the decision starts here: at the 99.5% target a heavy incident takes 42% of the budget, at 99.9% 208%, at 99.95% 417%. A target reads by how many heavy incidents it can withstand in a period: the 99.5% target withstands two, the 99.9% target does not even withstand one.

The same relationship holds in reverse too. The measured average outage is 23.3 minutes; the 99.9% budget is 43.2 minutes. So the budget runs out after two incidents, and an average of 2.3 incidents occur per period. The error budget is a ceiling set on the period total of time to restore.

What the Rule Stops

The second and third tables compare the same sequence of incidents with the rule off and on. With the rule off, twelve periods measure 27 incidents, 630 minutes of outage, and 99.878% availability — below the target. Once the rule is on, the budget runs out in eight of the twelve periods and 166 days of freezing are applied. In exchange: outage falls from 630 to 620 minutes, incidents from 27 to 26, and availability rises from 99.878% to 99.880%. The target still does not hold.

Why so little? The fourth table tells us: the number of releases falls from 60 to only 58. Freezing does not eliminate a change, it postpones it; once the frozen period ends, the queue empties and the same defects ship in the same order. The cost is lead time: it climbs from 3.0 days to 12.6 days, deployment frequency falls from 5.0 to 4.8, and eight changes never ship at all within the twelve periods. The budget rule is not a prevention tool, it is a stopping tool; it runs after the incident, not before it.

The timing confirms this too. The budget runs out after an average of two incidents; by then, most of the period’s five releases have already shipped. The releases the freeze blocks belong to the last days of the period, and they ship in the first days of the next period instead. Across the twelve periods combined, the number of blocked releases is two — against 166 days of freezing. The change failure rate falling from 35% to 34% has the same cause: the rule does not touch a release’s content, only its timing. The same holds for time to restore; its rise from 23.3 to 23.8 minutes is nothing more than a small shift in the class distribution of the delayed releases.

How Tight the Target Is and Where the Budget Comes From

At the 99.5% target the rule never fires: the budget never runs out in any of the twelve periods, the average remaining at period end is 163.5 minutes, and every measured number looks as if the rule does not exist. A rule tied to a loose target cannot be measured; writing it down does not change behavior.

At the 99.95% target the opposite happens. The budget runs out in nine of the twelve periods, 236 days freeze, releases fall from 60 to 45, and sixty changes never ship at all. Lead time jumps from 3.0 to 49.9 days. Outage falls from 630 to 440 minutes and availability comes to 99.915% — still below the target. A tight target stops the flow and still does not hit the target.

The fifth run shows where the budget really comes from. Leaving the target at 99.9% and halving outage duration, the budget runs out in only four periods, freezing falls to 65 days, releases stay at 60, and lead time stays at 5.4 days; availability rises to 99.939% and the target holds. Time to restore has fallen from 23.3 to 11.7 minutes. It is not the freeze that determines the budget, it is time to restore.

Where the Difference Hides

In this lesson, the difference hides in the relationship between the target and the outage distribution, and its number is this: the same 27 incidents produce three different budget draws across three targets — never exhausted at 216.0 minutes, exhausted in eight periods at 43.2 minutes, exhausted in nine periods at 21.6 minutes. The declared part is the target itself; the undeclared part is the distribution of incident classes. The heavy incident’s share is only 0.10, but it carries most of the total 630 minutes, and it is what exhausts the budget.

How many steps it takes the signal to reach whom also changes with this rule. Without the budget rule, the remaining budget is never read at all; with the rule, every incident is deducted from the budget the same day, and the moment of exhaustion stops the release decision on that same day. The effect measured across the four metrics also reads together: once the rule is on, deployment frequency goes from 5.0 to 4.8, lead time from 3.0 to 12.6 days; the change failure rate goes from 35% to 34%, time to restore from 23.3 to 23.8 minutes — that is, the last two barely move at all.

Summary

• The error budget is derived from the target: in a 30-day period, a 99.5% target leaves 216.0, a 99.9% target 43.2, a 99.95% target 21.6 minutes; a single heavy incident (90 minutes) takes 42%, 208%, and 417% of these respectively.
• With the rule off, 12 periods measure 27 incidents, 630 minutes of outage, and 99.878% availability.
• Once the rule is on, the budget runs out in 8 periods and 166 days freeze, but since releases fall from 60 to only 58, outage falls only to 620 minutes; the cost is lead time (3.0 → 12.6 days), and 8 changes never ship at all.
• If the target is loosened to 99.5%, the rule never fires (0/12 exhausted, average remaining 163.5 minutes); if it is tightened to 99.95%, 9/12 periods exhaust, 236 days freeze, 60 changes never ship, and lead time climbs to 49.9 days — the target still does not hold.
• Once outage duration is halved, the budget runs out in only 4 periods, lead time stays at 5.4 days, and availability hits the target at 99.939%: it is time to restore that determines the budget.

Next Step

The budget rule saw the incident only after it happened, and stopped things only after that; that is why the measured effect was small. How late the signal arrives has come up somewhere in every lesson, but it was never counted directly. The next lesson takes on the loop itself: how many separate feedback loops there are, how long each one is, which error class it covers, and what moving an error class to an earlier loop costs and what it misses.