Error Budget

The previous lesson computed four signals at minute resolution; it is now possible to tell when something drops below an objective. What happens once it drops below is still not written down. The loan system might get two releases a week, or two a day; every release leaves a measurable trace in the records, and every minute spent below the objective is paid for from somewhere. This lesson keeps the ledger for that payment.

The Budget Comes from Telemetry

If the objective is 99.9%, one in a thousand requests is allowed to be bad. Once that ratio is converted into a count, it becomes a spendable quantity: whatever number of requests arrived over thirty days, one-thousandth of that is the month’s error budget. A budget is a number whose remainder can be read, whose daily consumption can be plotted, and whose depletion day is known.

The Introduction to System Design course set up the same quantity as monthly downtime minutes; the conversion between percentage and minutes was done there and is not repeated here. The difference here is where the denominator comes from: when the budget is computed from the request log, it moves with traffic. If traffic doubles, the allowed number of bad requests doubles too, because the same ratio is applied to a larger base. A budget written in minutes has no such give.

How Release Rate Eats the Budget

The run below steps through thirty days minute by minute. Releases happen at fixed intervals, each release leaves a degradation behind with some probability, and the degradation’s severity and duration depend on the release’s size. The measure of release size is the number of days between two releases: the longer the interval, the more change a release carries.

// budget/run.mjs — the error budget derived from telemetry: if the objective is 99.9%, the
// allowed bad-request count is 1 per thousand of total requests. Rates are in per-ten-thousand.
export const MIN = 30 * 1440, SEED = 20260731, OBJECTIVE = 0.999, BASE = 2;
const makeRand = (t) => (n) => Math.floor(((t = (t * 1664525 + 1013904223) >>> 0) / 2 ** 32) * n);
const hour = (t) => Math.floor(t / 60) % 24;
export const reqPerMin = (t) => (hour(t) >= 8 && hour(t) < 22 ? 100 : 30);
export const TOTAL = [...Array(MIN)].reduce((a, _, t) => a + reqPerMin(t), 0);
export const BUDGET = Math.round(TOTAL * (1 - OBJECTIVE));
export const gate = (remaining) => remaining > 0;        // release gate: no release once the budget is out

// SD1 (assumption): as the interval between releases grows, the probability, severity, and
// recovery time of a degradation grow with it: probability 15% + 15%*days, severity 200 + 400*days
// per ten-thousand, recovery 10 + 20*days min.
export function run(intervalDays, gateOpen) {
  const rRel = makeRand(SEED), rReq = makeRand(SEED), period = Math.round(intervalDays * 1440);
  // intervalDays 0: no release is ever made, what is measured is the base error eating the budget alone
  const p = Math.min(900, 150 + 150 * intervalDays), sev = Math.min(2500, 200 + 400 * intervalDays);
  const dur = Math.min(120, Math.round(10 + 20 * intervalDays));
  let remaining = BUDGET, releases = 0, degraded = 0, blocked = 0, depletedDay = 0, degradedUntil = -1;
  const daily = [];
  for (let t = 0; t < MIN; t += 1) {
    if (period > 0 && t >= 600 && (t - 600) % period === 0) {
      if (gateOpen && !gate(remaining)) blocked += 1;
      else { releases += 1; if (rRel(1000) < p) { degraded += 1; degradedUntil = t + dur; } }
    }
    const rate = BASE + (t < degradedUntil ? sev : 0);
    for (let i = 0, n = reqPerMin(t); i < n; i += 1) if (rReq(10000) < rate) remaining -= 1;
    if (remaining <= 0 && depletedDay === 0) depletedDay = Math.floor(t / 1440) + 1;
    if ((t + 1) % 1440 === 0) daily.push(remaining / BUDGET);
  }
  return { releases, degraded, blocked, depletedDay, spent: BUDGET - remaining, daily };
}

export const RATE = [["no releases", 0], ["2/day", 0.5], ["1/day", 1],
  ["every 3 days", 3], ["every 5 days", 5]];

if (import.meta.filename === process.argv[1]) {
  console.log(`${TOTAL} requests in 30 days, objective %99.9 -> budget ${BUDGET} bad requests; base error ` +
    `at ${BASE} per ten-thousand alone eats ${Math.round((TOTAL * BASE) / 10000)} requests`);
  console.log(`\n${"release interval".padEnd(18)}${"releases".padStart(9)}${"degraded".padStart(9)}` +
    `${"bad minutes".padStart(14)}${"spent".padStart(10)}${"of budget".padStart(10)}` +
    `${"ran out".padStart(11)}`);
  for (const [name, g] of RATE) {
    const r = run(g, false), min = r.degraded * Math.min(120, Math.round(10 + 20 * g));
    console.log(`${name.padEnd(18)}${String(r.releases).padStart(9)}${String(r.degraded).padStart(9)}` +
      `${(min + " min").padStart(14)}${String(r.spent).padStart(10)}` +
      `${`%${(100 * r.spent / BUDGET).toFixed(1)}`.padStart(10)}` +
      `${(r.depletedDay ? `${r.depletedDay}.` : "-").padStart(11)}`);
  }
}

3060000 requests in 30 days, objective %99.9 -> budget 3060 bad requests; base error at 2 per ten-thousand alone eats 612 requests

release interval   releases degraded   bad minutes     spent of budget    ran out
no releases               0        0         0 min       631     %20.6          -
2/day                    60       13       260 min      1486     %48.6          -
1/day                    30        6       180 min      1717     %56.1          -
every 3 days             10        5       350 min      5523    %180.5        16.
every 5 days              6        4       440 min     10248    %334.9         6.

Not a Count, a Magnitude

The first row says that part of the budget has nothing to do with change at all: even with no releases at all, the base error alone eats 20.6% of the budget over thirty days. Even a system left untouched for the whole month has a budget, and not all of it can be set aside for change.

The real contrast is between the second and fourth rows. The team releasing twice a day shipped 60 releases for the month and spent 48.6% of the budget; the team releasing once every three days shipped 10 releases and spent 180.5% of the budget, and the budget ran out on day 16. Six times as many releases, roughly a third of the consumption.

The difference shows up in the cost per release. Subtracting the base consumption (631), the twice-a-day rate has 855 bad requests attributable to releases: 14 per release. The every-three- days rate has 4,892 bad requests attributable to releases: 489 per release. Thirty-five times as much. The bad-minutes column shows why: the fast rate had 13 degradations but they lasted 260 minutes total; the slow rate had 5 degradations and they lasted 350 minutes. Degradation count is higher on the fast team, degradation duration and severity are higher on the slow team, and it is the latter that eats the budget.

This result is baked into SD1 and should be argued there: the model assumes that a degradation’s probability, severity, and recovery time all grow with release size. If the assumption changes, the table changes with it. The way to turn the assumption into a measurement is written in the telemetry: every release is timestamped, the degradation window following it is extracted from the records, and the three parameters — probability, severity, duration — are plotted against release size. Once those three numbers are measured, the release-rate debate stops being a matter of preference.

Once the Budget Runs Out

A budget is a permission slip: while the remainder is positive, change is free to happen; once it is spent, priority shifts back to stability. The counterpart of this in code is a one-line gate — when gate(remaining) returns false, no release happens. The run below executes the same four rates twice, once with the gate closed and once with it open.

// budget/gate.mjs — what changes when the release gate is open: once the budget runs out,
// releases are blocked. The same four release intervals are run twice, gate closed and open.
import { run, RATE, BUDGET } from "./run.mjs";
const pct = (x) => `%${(100 * x).toFixed(1)}`;

console.log(`budget ${BUDGET} bad requests\n`);
console.log(`${"release interval".padEnd(18)}${"gate".padStart(8)}${"releases".padStart(9)}` +
  `${"blocked".padStart(12)}${"spent".padStart(10)}${"of budget".padStart(10)}` +
  `${"overage".padStart(9)}`);
for (const [name, g] of RATE)
  for (const open of [false, true]) {
    const r = run(g, open);
    console.log(`${name.padEnd(18)}${(open ? "open" : "closed").padStart(8)}` +
      `${String(r.releases).padStart(9)}${String(r.blocked).padStart(12)}` +
      `${String(r.spent).padStart(10)}${pct(r.spent / BUDGET).padStart(10)}` +
      `${(r.spent > BUDGET ? String(r.spent - BUDGET) : "-").padStart(9)}`);
  }

const DAY = [3, 6, 9, 12, 15, 18, 21, 24, 27, 30];
console.log(`\nremaining budget (gate closed)\n${"day".padEnd(18)}${DAY.map((g) => String(g).padStart(8)).join("")}`);
for (const [name, g] of RATE) {
  const r = run(g, false);
  console.log(`${name.padEnd(18)}${DAY.map((d) => pct(r.daily[d - 1]).padStart(8)).join("")}`);
}

budget 3060 bad requests

release interval      gate releases     blocked     spent of budget  overage
no releases         closed        0           0       631     %20.6        -
no releases           open        0           0       631     %20.6        -
2/day               closed       60           0      1486     %48.6        -
2/day                 open       60           0      1486     %48.6        -
1/day               closed       30           0      1717     %56.1        -
1/day                 open       30           0      1717     %56.1        -
every 3 days        closed       10           0      5523    %180.5     2463
every 3 days          open        6           4      3531    %115.4      471
every 5 days        closed        6           0     10248    %334.9     7188
every 5 days          open        2           4      5416    %177.0     2356

remaining budget (gate closed)
day                      3       6       9      12      15      18      21      24      27      30
no releases          %98.0   %96.2   %93.5   %91.2   %89.2   %87.2   %85.3   %82.9   %81.2   %79.4
2/day                %98.0   %92.0   %86.7   %80.7   %76.2   %68.8   %63.9   %59.2   %56.8   %51.4
1/day                %98.0   %96.2   %93.5   %80.4   %78.3   %70.2   %62.1   %53.8   %45.7   %43.9
every 3 days         %67.0   %65.2   %62.5   %27.8   %25.8   %-7.6   %-9.5  %-43.9  %-45.7  %-80.5
every 5 days         %19.9  %-60.2  %-62.8  %-65.2  %-67.2 %-148.2 %-150.1 %-152.5 %-233.1 %-234.9

The gate says three things at once. It never stops the fast team: in the twice-a-day and once-a-day rows, the number of blocked releases is zero, because the budget never runs out. The gate is not a release-rate limit, it is a budget limit; a team releasing fast never sees it.

On the slow team, four releases are blocked and the overage drops from 2,463 to 471: 81% of the consumption beyond the budget is prevented. Six of the ten releases go out, four are left for the end of the month.

The gate is late. Even with the gate open, there is still an overage of 471 requests, because the gate engages after the budget runs out, not before. The remaining-budget table shows this day by day: for the team releasing every three days, the remainder had dropped to 27.8% by day 12 and crossed into negative by day 18. Six days pass between the day the remainder reads 27.8% and the day the overage begins, and two more releases went out in those days. This is why the gate alone is not enough: a gate that closes after a threshold is crossed does not prevent the overage itself; what is needed is something that gives warning before the budget runs out.

Summary

• An error budget is the objective’s complement applied to the request log: 3,060,000 requests over thirty days and a 99.9% objective give a budget of 3,060 bad requests, and this budget flexes with traffic.
• Part of the budget has nothing to do with change: with no releases at all, the base error alone ate 20.6% of the budget.
• Six times as many releases does not mean six times the consumption: 60 releases spent 48.6% of the budget, 10 releases spent 180.5%, and in the latter case the budget ran out on day 16.
• Cost per release: 14 bad requests versus 489 bad requests; degradation count is higher on the fast team (13 to 5), degradation duration is higher on the slow team (260 minutes to 350), and it is duration that eats the budget.
• The release gate is a one-line condition and never stops the fast team; on the slow team, it blocked four releases and brought the overage down from 2,463 to 471.
• Because the gate closes after the budget runs out, it cannot prevent the entire overage; a mechanism is needed that gives warning before the budget is depleted.

Next Step

Every row in the remaining-budget table can be read at the end of the month, yet the decision needed to be made on day 12. If no one is looking at the table that day, the number has no value — someone has to be paged. Which rule the page comes from is still an open question: does the rule watch the notification queue’s depth, the loan service’s memory usage, or the error rate the member sees directly? That choice determines how many alerts fire, how many times the same rule fires for the same incident, and how many incidents never fire an alert at all. The next lesson runs two separate rule sets across the same incidents and compares them on these three numbers.