Failure Modes

Across four courses, the system never broke, not once. A request entered at the edge, passed through the cache, was load balanced, landed on a service, reached a store, and returned; every computation assumed the parts worked. Once that assumption is removed, the first question is not “what to do” but “what happened.” When a dependency does not answer, the caller has only its own observation, and very different breakdowns can hide behind it; the right response depends on which one it is.

This lesson builds no pattern; the course’s first pattern is next. What it builds is a vocabulary: the forms of breakdown are named, how they look from the caller’s eye is measured, and how far each spreads through the system is counted.

Vocabulary: Five Forms of Breakdown

A failure mode is a component’s form of breakdown: not what it does but how it does it wrong. The system’s own structure yields five scenarios, each an instance of a mode.

• Down. A shard node goes down; the call errors immediately.
• Slowdown. The carrier notification endpoint slows down; the response arrives, but later than the caller can wait.
• Partial failure. One of eight shards goes down; some calls error, the rest respond.
• Stale content. The log consumer feeding the projection falls behind; the response is fast and correctly shaped, but stale.
• Network partition. The link between two regions breaks; both sides stay up, neither can see the other.

Here, network partition counts only as a failure mode; the choice between consistency and service availability under a partition was set in the Introduction to System Design course. The Relational Database Administration course’s partitioning and the Scaling the Data Layer course’s sharding are separate from network partition: the first two are decisions to split data, the split here an unwanted, unintended break.

A sixth is added to the five, marking the vocabulary’s limit: the end-of-day billing job stalls halfway through. No call errors, none slows down; the measurement below cannot see this mode at all. Seeing a mode needs an observation aimed at it.

The Signature the Caller Sees

The run below is a model. A round is an abstract step; no real network, cluster, or container is set up, the dependency’s behavior is a parameter, and randomness comes from a custom-written, seeded generator. The assumptions are this course’s own, not added to K01’s table.

AY1 — round-to-second cycle: 1 round = 10 ms. Rationale: the Application Layer and Service Interaction course counted a step call’s usual duration as 8 ms; a two-round call comes to 20 ms.

AY2 — 32 concurrent slots per component, timeout 20 rounds. Rationale: at AY1, 20 rounds is exactly 200 ms, that is, K01’s read threshold itself; 32 slots give a peak utilization of 0.78 on a failure-free day. Their sensitivities are printed in the second run.

AY4 — on a failure-free day, 0.025 of calls time out. Rationale: brief delays happen even without a failure. Its sensitivity is given below at 0.050.

// modes/signature.mjs — the CALLER's-eye signature of the six failure modes. MODEL: a round is an
// abstract step, the dependency's behavior is a parameter, randomness comes from a seeded generator.
const N = 400, USUAL = 2, TIMEOUT_ROUNDS = 20;   // call count; usual call 2 rounds; AY2: timeout 20 rounds
const SHARDS = 8, DOWN_SHARDS = 1;              // partial failure: one of eight shards is down (M19/K04)
const RARE = 0.025;                             // AY4: 1/40 of calls time out on a failure-free day
let s = 20260730 % 2147483647;
const random = () => (s = (s * 48271) % 2147483647) / 2147483647;
const RESPONSE = { kind: "response", rounds: USUAL, fresh: true };
const ERROR = { kind: "error", rounds: 1, fresh: null };
const TIMEOUT = { kind: "timeout", rounds: TIMEOUT_ROUNDS, fresh: null };

function call(mode) {                 // one call's outcome: kind, rounds held, content freshness
  if (mode === "down") return ERROR;
  if (mode === "slowdown" || mode === "partition") return TIMEOUT;
  if (mode === "stale") return { ...RESPONSE, fresh: false };
  if (mode === "partial") return random() < DOWN_SHARDS / SHARDS ? ERROR : RESPONSE;
  return random() < RARE ? TIMEOUT : RESPONSE;
}

function signature(mode) {            // signature vector of N calls
  const c = Array.from({ length: N }, () => call(mode));
  const y = c.filter((x) => x.kind === "response");
  const ratio = (t) => c.filter((x) => x.kind === t).length / N;
  return { response: y.length / N, error: ratio("error"), timeout: ratio("timeout"),
    slotRounds: c.reduce((t, x) => t + x.rounds, 0) / N,
    fresh: y.length === 0 ? null : y.filter((x) => x.fresh).length / y.length };
}

const MODES = ["healthy", "down", "slowdown", "partial", "stale", "partition"];
const b = (x) => (x === null ? "-" : x.toFixed(3));
const signatures = Object.fromEntries(MODES.map((k) => [k, signature(k)]));

console.log(`model: ${N} calls per mode, usual call ${USUAL} rounds, timeout ${TIMEOUT_ROUNDS} rounds, seed 20260730`);
console.log(`\n${"mode".padEnd(11)}${"response".padStart(9)}${"error".padStart(7)}${"timeout".padStart(13)}` +
  `${"slot-rounds/call".padStart(17)}${"fresh content".padStart(14)}`);
for (const k of MODES) {
  const i = signatures[k];
  console.log(`${k.padEnd(11)}${b(i.response).padStart(9)}${b(i.error).padStart(7)}${b(i.timeout).padStart(13)}` +
    `${i.slotRounds.toFixed(2).padStart(17)}${b(i.fresh).padStart(14)}`);
}

const equal = (a, c, fields) => fields.every((k) => Math.abs((a[k] ?? -1) - (c[k] ?? -1)) < 0.02);
console.log("\nmode pairs the class ratios cannot separate:");
for (let i = 0; i < MODES.length; i += 1)
  for (let j = i + 1; j < MODES.length; j += 1)
    if (equal(signatures[MODES[i]], signatures[MODES[j]], ["response", "error", "timeout"]))
      console.log(`  ${MODES[i]} = ${MODES[j]} -> content check ` +
        `${equal(signatures[MODES[i]], signatures[MODES[j]], ["fresh"]) ? "does not separate them either" : "separates them"}`);

const WINDOW = 20, STAMP = 8, EVENT = 97.22, EDGE = 513.89, CYCLE = 100;  // K01 rates; AY1
console.log(`\nfailure-free day's cost: an ${STAMP}-byte freshness stamp per response = ` +
  `${(STAMP * EDGE).toFixed(2)} bytes/s (K01's edge rate of ${EDGE} req/s); naming needs a window of ${WINDOW} calls = ` +
  `${(WINDOW / (EVENT / CYCLE)).toFixed(1)} rounds (${EVENT} events/s) = ` +
  `${((WINDOW / (EVENT / CYCLE) * 1000) / CYCLE).toFixed(0)} ms`);
console.log(`sensitivity: if AY4 were 0.050, slot-rounds/call would be ${(USUAL + 0.05 * (TIMEOUT_ROUNDS - USUAL)).toFixed(2)} instead of ` +
  `${signatures.healthy.slotRounds.toFixed(2)}`);

model: 400 calls per mode, usual call 2 rounds, timeout 20 rounds, seed 20260730

mode        response  error      timeout slot-rounds/call fresh content
healthy        0.988  0.000        0.013             2.23         1.000
down           0.000  1.000        0.000             1.00             -
slowdown       0.000  0.000        1.000            20.00             -
partial        0.880  0.120        0.000             1.88         1.000
stale          1.000  0.000        0.000             2.00         0.000
partition      0.000  0.000        1.000            20.00             -

mode pairs the class ratios cannot separate:
  healthy = stale -> content check separates them
  slowdown = partition -> content check does not separate them either

failure-free day's cost: an 8-byte freshness stamp per response = 4111.12 bytes/s (K01's edge rate of 513.89 req/s); naming needs a window of 20 calls = 20.6 rounds (97.22 events/s) = 206 ms
sensitivity: if AY4 were 0.050, slot-rounds/call would be 2.90 instead of 2.23

These numbers belong to the measurement class: they come from a model run on this machine and are deterministic.

The table’s most important column is slot-rounds per call: how many rounds the caller holds its slot for. Down is 1.00, slowdown 20.00. A down dependency frees the caller’s resource; a slowing one holds it — twenty times the difference for the same call. Partial failure sits between: its error ratio is 0.120, one in eight, so a retrying caller may still land on a healthy shard — a hope down offers none of. Stale content, by contrast, is indistinguishable from healthy in the class ratios — only the freshness stamp sets it apart.

The last row is the lesson’s line of distinction. Slowdown and network partition look identical to the caller, and the content check does not separate them either: neither returns a response, and both hold the slot until timeout. The difference is not in the call — under a partition the other side is up and may have processed the request. A retried write is only delayed under slowdown; under a partition it can apply twice.

The vocabulary’s failure-free-day cost sits in the last two lines. Making stale content visible costs an 8-byte freshness stamp on every response (4111.12 bytes/s at K01’s edge rate of 513.89 req/s), and naming a mode needs a window of twenty calls — 20.6 rounds at the carrier’s event rate, 206 ms.

Blast Radius

A failure’s cost sits not in the broken component but in where it spreads. Blast radius measures the components that cannot serve during a failure and the request rate affected. The second run models the system as a call tree: slots held in a component equal flow rate times hold duration; once the total exceeds the pool, the component becomes saturated and looks slow to its own caller.

// modes/blast-radius.mjs — a failure mode's blast radius. MODEL: spread comes out of slot occupancy.
// Rates come from K01's peak computation, structure comes from K02-K04.
const CYCLE = 100, SLOTS = 32, TIMEOUT_ROUNDS = 20, USUAL = 2;   // AY1: 1 round = 10 ms; AY2: slots and timeout
const d = (name, ...children) => ({ name, children });
const FLOWS = [                                        // K01 peak: 416.67 + 97.22 = 513.89 req/s
  { name: "tracking query", rate: 416.67, root: d("gateway", d("tracking-service", d("projection-store"))) },
  { name: "state event", rate: 97.22,
    root: d("gateway", d("event-endpoint", d("carrier-validation"), d("state-store"))) },
  { name: "end-of-day billing", rate: 833.33, root: d("end-of-day-billing", d("state-store")) },
  { name: "log consumption", rate: 97.22, root: d("log-consumer", d("state-store"), d("projection-store")) },
];

function hold(n, broken, mode) {        // rounds held; no further calls are made after the first failure
  if (n.name === broken) return mode === "down" ? { rounds: 1, ok: false } : { rounds: TIMEOUT_ROUNDS, ok: false };
  if (n.children.length === 0) return { rounds: USUAL, ok: true };
  let rounds = 1, ok = true;
  for (const c of n.children) { const r = hold(c, broken, mode); rounds += r.rounds; if (!r.ok) { ok = false; break; } }
  return { rounds, ok };
}

function run(broken, mode) {            // occupancy per component = flow rate x hold duration
  const occupied = {}, through = {};
  const walk = (n, a) => {
    occupied[n.name] = (occupied[n.name] ?? 0) + (a.rate / CYCLE) * hold(n, broken, mode).rounds;
    (through[n.name] ??= new Set()).add(a.name);
    n.children.forEach((c) => walk(c, a));
  };
  FLOWS.forEach((a) => walk(a.root, a));
  const saturated = Object.keys(occupied).filter((k) => occupied[k] >= SLOTS);
  const hit = new Set(broken ? [...saturated, broken] : saturated);
  const flows = new Set([...hit].flatMap((v) => [...through[v]]));
  const rate = [...flows].reduce((t, a) => t + FLOWS.find((x) => x.name === a).rate, 0);
  return { occupied, saturated, hit: hit.size, flows: flows.size, rate };
}

const t = run(null, null);
console.log(`model: ${Object.keys(t.occupied).length} components, ${FLOWS.length} flows, ${SLOTS} ` +
  `slots per component, timeout ${TIMEOUT_ROUNDS} rounds, 1 round = ${1000 / CYCLE} ms\n`);
console.log(`failure-free day: ` + Object.entries(t.occupied)
  .map(([k, v]) => `${k} ${(v / SLOTS).toFixed(3)}`).join(", "));

console.log(`\n${"broken component".padEnd(20)}${"mode".padEnd(11)}${"hit".padStart(8)}` +
  `${"saturated".padStart(42)}${"flows".padStart(6)}${"affected req/s".padStart(19)}`);
for (const component of ["carrier-validation", "projection-store", "state-store"])
  for (const mode of ["down", "slowdown"]) {
    const r = run(component, mode);
    console.log(`${component.padEnd(20)}${mode.padEnd(11)}${String(r.hit).padStart(8)}` +
      `${(r.saturated.join(",") || "-").padStart(42)}${`${r.flows}/${FLOWS.length}`.padStart(6)}` +
      `${r.rate.toFixed(2).padStart(19)}`);
  }

const down = run("carrier-validation", "down"), slow = run("carrier-validation", "slowdown");
const g = (r) => (r.occupied["gateway"] / SLOTS).toFixed(3);
console.log(`\ncarrier validation: down ${down.rate.toFixed(2)} req/s and ${down.hit} component(s), ` +
  `slowdown ${slow.rate.toFixed(2)} and ${slow.hit} -> x${(slow.rate / down.rate).toFixed(2)} requests`);
console.log(`gateway utilization: failure-free ${g(t)} -> down ${g(down)} -> slowdown ${g(slow)}`);
console.log(`sensitivity: with 48 slots, utilization under slowdown would be ${(slow.occupied["gateway"] / 48).toFixed(3)} ` +
  `(no overflow); with a 20 ms round it would be ${(2 * slow.occupied["gateway"] / SLOTS).toFixed(3)}`);

const COUNT = 3, DURATION = 10, SHARE = 28.2;  // AY3; K01 fundamental-properties/04: monthly failure share (minutes)
console.log(`\nAY3: ${COUNT} x ${DURATION} min a month = ${COUNT * DURATION} min, ${((100 * COUNT * DURATION) / SHARE).toFixed(1)}% ` +
  `of K01's ${SHARE}-min failure share (at a 5 min duration it would be ${((100 * COUNT * 5) / SHARE).toFixed(1)}%)`);
console.log(`the same ${COUNT * DURATION} minutes: down ${(COUNT * DURATION * 60 * down.rate).toFixed(0)} requests, ` +
  `slowdown ${(COUNT * DURATION * 60 * slow.rate).toFixed(0)} requests, difference ${(COUNT * DURATION * 60 * (slow.rate - down.rate)).toFixed(0)}`);

model: 8 components, 4 flows, 32 slots per component, timeout 20 rounds, 1 round = 10 ms

failure-free day: gateway 0.703, tracking-service 0.391, projection-store 0.321, event-endpoint 0.152, carrier-validation 0.061, state-store 0.642, end-of-day-billing 0.781, log-consumer 0.152

broken component    mode            hit                                 saturated flows     affected req/s
carrier-validation  down              1                                         -   1/4              97.22
carrier-validation  slowdown          2                                   gateway   2/4             513.89
projection-store    down              1                                         -   2/4             513.89
projection-store    slowdown          3 gateway,tracking-service,projection-store   3/4             611.11
state-store         down              1                                         -   3/4            1027.77
state-store         slowdown          3    gateway,state-store,end-of-day-billing   4/4            1444.44

carrier validation: down 97.22 req/s and 1 component(s), slowdown 513.89 and 2 -> x5.29 requests
gateway utilization: failure-free 0.703 -> down 0.612 -> slowdown 1.189
sensitivity: with 48 slots, utilization under slowdown would be 0.793 (no overflow); with a 20 ms round it would be 2.378

AY3: 3 x 10 min a month = 30 min, 106.4% of K01's 28.2-min failure share (at a 5 min duration it would be 53.2%)
the same 30 minutes: down 174996 requests, slowdown 925002 requests, difference 750006

The end-of-day billing job runs in its own window; showing it alongside the online flows gives the worst case.

Same Component, Two Modes, a Fivefold Difference

The reading starts from one pair of rows. When carrier validation goes down, the event endpoint errors in one round, releases its slot, and rejects the request. Only the state event is affected: 97.22 req/s, one component. The tracking query is untouched.

When the same dependency slows down, the event endpoint holds its slot for twenty rounds on every call, and that hold propagates upward: the gateway’s leg calling the event endpoint climbs from 6 to 22 rounds, utilization goes from 0.703 to 1.189, and the gateway becomes saturated. The tracking query now cannot be answered either, though it never touches the broken dependency. Affected rate: 513.89 req/s, components hit: 2 — 5.29 times the requests.

This is the course’s first law and the reason for the next three lessons: slowdown costs more than being down. A down dependency kills its own flow and stops there; a slowing one consumes a shared resource and drags unrelated flows with it. Under the down mode, gateway utilization falls from 0.703 to 0.612 — a failure relieves the gateway, since rejected requests hold no resources.

The remaining rows confirm the pattern. When the projection store goes down, 513.89 req/s is affected; slowing down saturates the whole chain, and the rate becomes 611.11. When the state store goes down, 1027.77 req/s is affected but nothing saturates; slowing down hits all four flows at once, 1444.44 req/s.

The sensitivity row says what saturation depends on. With a 48-slot pool, slowdown utilization would stay at 0.793, the gateway would not overflow, and blast radius would fall from two components to one; at a 20 ms round, utilization would be 2.378. Blast radius comes not from the failure itself but from the product of pool size and timeout.

The last two lines tie the two days to the outage budget. AY3 — slowdown happens three times a month, ten minutes each. Rationale: K01 assumed a ten-minute recovery time and fit 2.82 failures into a monthly failure share of 28.2 minutes; three failures eat 106.4 percent of it. Sensitivity: at five minutes the share drops to 53.2 percent. The same 30 minutes hits 174,996 requests under down and 925,002 under slowdown — a difference of 750,006. This is the vocabulary’s gain on a failing day: it cuts no failure, but makes confusing the two modes countable.

Summary

• A failure mode is a component’s form of breakdown; the scenarios map to five modes — down, slowdown, partial failure, stale content, network partition. A job stalling halfway through is invisible to call observation.
• The signature partly tells the modes apart: down is 1.00 slot-rounds and 1.000 error; slowdown is 20.00 slot-rounds and 1.000 timeout; partial failure is 0.120 error; stale content is set apart from healthy only by the freshness stamp.
• Slowdown and network partition look identical to the caller, and the content check does not separate them either; under a partition the other side stays up and may have processed the request.
• A failure-free day costs the 8-byte stamp on every response (4111.12 bytes/s) and the twenty-call window naming requires (20.6 rounds at 97.22 events/s, 206 ms).
• Blast radius depends on the mode: carrier validation going down affects 97.22 req/s and 1 component; slowing down affects 513.89 req/s and 2 — 5.29 times. Gateway utilization falls from 0.703 to 0.612 under down and climbs to 1.189 under slowdown.
• In AY3’s 30 minutes, the two modes differ by 750,006 requests — 106.4 percent of K01’s monthly failure share.

Next Step

The measurement points to a direction. Slowdown is expensive not because the dependency is broken but because the caller keeps calling it: every call holds a slot for twenty rounds that do nothing useful. The same table shows the fix — under down, utilization falls to 0.612, because a fast failure holds no resources. Stopping calls to a slowing dependency turns the slowdown into a down, cutting blast radius from 513.89 to 97.22 req/s. The next lesson designs the component that makes that call: at what threshold it decides to cut, how many calls are never made while cut off, and what wrongly cutting off a healthy dependency costs.