--- title: 'Failure Modes' source: 'https://academia.sh/en/courses/resilience-patterns/failure-modes' course: 'Resilience and Reliability' language: en updated: '2026-08-23T07:01:30+00:00' license: 'CC BY-SA 4.0' --- # Failure Modes Removing the assumption that the parts work: naming partial failure, slowdown, stale content, and network partition as separate failure modes, telling a mode apart by its signature from the caller's eye, showing where two modes give the same signature, and comparing a component going down against slowing down in terms of blast radius. Across four courses, the system never broke, not once. A request entered at the edge, passed through the cache, was load balanced, landed on a service, reached a store, and returned; every computation assumed the parts **worked**. Once that assumption is removed, the first question is not "what to do" but "what happened." When a dependency does not answer, the caller has only its own observation, and very different breakdowns can hide behind it; the right response depends on which one it is. This lesson builds no pattern; the course's first pattern is next. What it builds is a **vocabulary**: the forms of breakdown are named, how they look from the caller's eye is measured, and how far each spreads through the system is counted. ## Vocabulary: Five Forms of Breakdown A **failure mode** is a component's form of breakdown: not what it does but **how** it does it wrong. The system's own structure yields five scenarios, each an instance of a mode. - **Down.** A shard node goes down; the call errors immediately. - **Slowdown.** The carrier notification endpoint slows down; the response arrives, but later than the caller can wait. - **Partial failure.** One of eight shards goes down; some calls error, the rest respond. - **Stale content.** The log consumer feeding the projection falls behind; the response is fast and correctly shaped, but stale. - **Network partition.** The link between two regions breaks; both sides stay up, neither can see the other. Here, network partition counts only as a failure mode; the choice between consistency and service availability under a partition was set in the Introduction to System Design course. The Relational Database Administration course's **partitioning** and the Scaling the Data Layer course's **sharding** are separate from **network partition**: the first two are decisions to split data, the split here an unwanted, unintended break. A sixth is added to the five, marking the vocabulary's limit: **the end-of-day billing job stalls halfway through.** No call errors, none slows down; the measurement below **cannot see this mode at all**. Seeing a mode needs an observation aimed at it. ## The Signature the Caller Sees The run below is a **model**. A round is an abstract step; no real network, cluster, or container is set up, the dependency's behavior is a parameter, and randomness comes from a custom-written, seeded generator. The assumptions are this course's own, not added to K01's table. **AY1 — round-to-second cycle: 1 round = 10 ms.** Rationale: the Application Layer and Service Interaction course counted a step call's usual duration as 8 ms; a two-round call comes to 20 ms. **AY2 — 32 concurrent slots per component, timeout 20 rounds.** Rationale: at AY1, 20 rounds is exactly 200 ms, that is, K01's read threshold itself; 32 slots give a peak utilization of 0.78 on a failure-free day. Their sensitivities are printed in the second run. **AY4 — on a failure-free day, 0.025 of calls time out.** Rationale: brief delays happen even without a failure. Its sensitivity is given below at 0.050. ```js // modes/signature.mjs — the CALLER's-eye signature of the six failure modes. MODEL: a round is an // abstract step, the dependency's behavior is a parameter, randomness comes from a seeded generator. const N = 400, USUAL = 2, TIMEOUT_ROUNDS = 20; // call count; usual call 2 rounds; AY2: timeout 20 rounds const SHARDS = 8, DOWN_SHARDS = 1; // partial failure: one of eight shards is down (M19/K04) const RARE = 0.025; // AY4: 1/40 of calls time out on a failure-free day let s = 20260730 % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const RESPONSE = { kind: "response", rounds: USUAL, fresh: true }; const ERROR = { kind: "error", rounds: 1, fresh: null }; const TIMEOUT = { kind: "timeout", rounds: TIMEOUT_ROUNDS, fresh: null }; function call(mode) { // one call's outcome: kind, rounds held, content freshness if (mode === "down") return ERROR; if (mode === "slowdown" || mode === "partition") return TIMEOUT; if (mode === "stale") return { ...RESPONSE, fresh: false }; if (mode === "partial") return random() < DOWN_SHARDS / SHARDS ? ERROR : RESPONSE; return random() < RARE ? TIMEOUT : RESPONSE; } function signature(mode) { // signature vector of N calls const c = Array.from({ length: N }, () => call(mode)); const y = c.filter((x) => x.kind === "response"); const ratio = (t) => c.filter((x) => x.kind === t).length / N; return { response: y.length / N, error: ratio("error"), timeout: ratio("timeout"), slotRounds: c.reduce((t, x) => t + x.rounds, 0) / N, fresh: y.length === 0 ? null : y.filter((x) => x.fresh).length / y.length }; } const MODES = ["healthy", "down", "slowdown", "partial", "stale", "partition"]; const b = (x) => (x === null ? "-" : x.toFixed(3)); const signatures = Object.fromEntries(MODES.map((k) => [k, signature(k)])); console.log(`model: ${N} calls per mode, usual call ${USUAL} rounds, timeout ${TIMEOUT_ROUNDS} rounds, seed 20260730`); console.log(`\n${"mode".padEnd(11)}${"response".padStart(9)}${"error".padStart(7)}${"timeout".padStart(13)}` + `${"slot-rounds/call".padStart(17)}${"fresh content".padStart(14)}`); for (const k of MODES) { const i = signatures[k]; console.log(`${k.padEnd(11)}${b(i.response).padStart(9)}${b(i.error).padStart(7)}${b(i.timeout).padStart(13)}` + `${i.slotRounds.toFixed(2).padStart(17)}${b(i.fresh).padStart(14)}`); } const equal = (a, c, fields) => fields.every((k) => Math.abs((a[k] ?? -1) - (c[k] ?? -1)) < 0.02); console.log("\nmode pairs the class ratios cannot separate:"); for (let i = 0; i < MODES.length; i += 1) for (let j = i + 1; j < MODES.length; j += 1) if (equal(signatures[MODES[i]], signatures[MODES[j]], ["response", "error", "timeout"])) console.log(` ${MODES[i]} = ${MODES[j]} -> content check ` + `${equal(signatures[MODES[i]], signatures[MODES[j]], ["fresh"]) ? "does not separate them either" : "separates them"}`); const WINDOW = 20, STAMP = 8, EVENT = 97.22, EDGE = 513.89, CYCLE = 100; // K01 rates; AY1 console.log(`\nfailure-free day's cost: an ${STAMP}-byte freshness stamp per response = ` + `${(STAMP * EDGE).toFixed(2)} bytes/s (K01's edge rate of ${EDGE} req/s); naming needs a window of ${WINDOW} calls = ` + `${(WINDOW / (EVENT / CYCLE)).toFixed(1)} rounds (${EVENT} events/s) = ` + `${((WINDOW / (EVENT / CYCLE) * 1000) / CYCLE).toFixed(0)} ms`); console.log(`sensitivity: if AY4 were 0.050, slot-rounds/call would be ${(USUAL + 0.05 * (TIMEOUT_ROUNDS - USUAL)).toFixed(2)} instead of ` + `${signatures.healthy.slotRounds.toFixed(2)}`); ``` ``` model: 400 calls per mode, usual call 2 rounds, timeout 20 rounds, seed 20260730 mode response error timeout slot-rounds/call fresh content healthy 0.988 0.000 0.013 2.23 1.000 down 0.000 1.000 0.000 1.00 - slowdown 0.000 0.000 1.000 20.00 - partial 0.880 0.120 0.000 1.88 1.000 stale 1.000 0.000 0.000 2.00 0.000 partition 0.000 0.000 1.000 20.00 - mode pairs the class ratios cannot separate: healthy = stale -> content check separates them slowdown = partition -> content check does not separate them either failure-free day's cost: an 8-byte freshness stamp per response = 4111.12 bytes/s (K01's edge rate of 513.89 req/s); naming needs a window of 20 calls = 20.6 rounds (97.22 events/s) = 206 ms sensitivity: if AY4 were 0.050, slot-rounds/call would be 2.90 instead of 2.23 ``` These numbers belong to the measurement class: they come from a model run on this machine and are deterministic. The table's most important column is **slot-rounds per call**: how many rounds the caller holds its slot for. Down is 1.00, slowdown 20.00. **A down dependency frees the caller's resource; a slowing one holds it** — twenty times the difference for the same call. Partial failure sits between: its error ratio is 0.120, one in eight, so a retrying caller may still land on a healthy shard — a hope down offers none of. Stale content, by contrast, is **indistinguishable from healthy** in the class ratios — only the freshness stamp sets it apart. The last row is the lesson's line of distinction. **Slowdown and network partition look identical to the caller**, and the content check does not separate them either: neither returns a response, and both hold the slot until timeout. The difference is not in the call — under a partition the other side **is up and may have processed the request**. A retried write is only delayed under slowdown; under a partition it can apply twice. The vocabulary's failure-free-day cost sits in the last two lines. Making stale content visible costs an 8-byte freshness stamp on every response (4111.12 bytes/s at K01's edge rate of 513.89 req/s), and naming a mode needs a window of twenty calls — 20.6 rounds at the carrier's event rate, 206 ms. ## Blast Radius A failure's cost sits not in the broken component but in **where it spreads**. **Blast radius** measures the components that cannot serve during a failure and the request rate affected. The second run models the system as a call tree: slots held in a component equal **flow rate times hold duration**; once the total exceeds the pool, the component becomes **saturated** and looks slow to its own caller. ```js // modes/blast-radius.mjs — a failure mode's blast radius. MODEL: spread comes out of slot occupancy. // Rates come from K01's peak computation, structure comes from K02-K04. const CYCLE = 100, SLOTS = 32, TIMEOUT_ROUNDS = 20, USUAL = 2; // AY1: 1 round = 10 ms; AY2: slots and timeout const d = (name, ...children) => ({ name, children }); const FLOWS = [ // K01 peak: 416.67 + 97.22 = 513.89 req/s { name: "tracking query", rate: 416.67, root: d("gateway", d("tracking-service", d("projection-store"))) }, { name: "state event", rate: 97.22, root: d("gateway", d("event-endpoint", d("carrier-validation"), d("state-store"))) }, { name: "end-of-day billing", rate: 833.33, root: d("end-of-day-billing", d("state-store")) }, { name: "log consumption", rate: 97.22, root: d("log-consumer", d("state-store"), d("projection-store")) }, ]; function hold(n, broken, mode) { // rounds held; no further calls are made after the first failure if (n.name === broken) return mode === "down" ? { rounds: 1, ok: false } : { rounds: TIMEOUT_ROUNDS, ok: false }; if (n.children.length === 0) return { rounds: USUAL, ok: true }; let rounds = 1, ok = true; for (const c of n.children) { const r = hold(c, broken, mode); rounds += r.rounds; if (!r.ok) { ok = false; break; } } return { rounds, ok }; } function run(broken, mode) { // occupancy per component = flow rate x hold duration const occupied = {}, through = {}; const walk = (n, a) => { occupied[n.name] = (occupied[n.name] ?? 0) + (a.rate / CYCLE) * hold(n, broken, mode).rounds; (through[n.name] ??= new Set()).add(a.name); n.children.forEach((c) => walk(c, a)); }; FLOWS.forEach((a) => walk(a.root, a)); const saturated = Object.keys(occupied).filter((k) => occupied[k] >= SLOTS); const hit = new Set(broken ? [...saturated, broken] : saturated); const flows = new Set([...hit].flatMap((v) => [...through[v]])); const rate = [...flows].reduce((t, a) => t + FLOWS.find((x) => x.name === a).rate, 0); return { occupied, saturated, hit: hit.size, flows: flows.size, rate }; } const t = run(null, null); console.log(`model: ${Object.keys(t.occupied).length} components, ${FLOWS.length} flows, ${SLOTS} ` + `slots per component, timeout ${TIMEOUT_ROUNDS} rounds, 1 round = ${1000 / CYCLE} ms\n`); console.log(`failure-free day: ` + Object.entries(t.occupied) .map(([k, v]) => `${k} ${(v / SLOTS).toFixed(3)}`).join(", ")); console.log(`\n${"broken component".padEnd(20)}${"mode".padEnd(11)}${"hit".padStart(8)}` + `${"saturated".padStart(42)}${"flows".padStart(6)}${"affected req/s".padStart(19)}`); for (const component of ["carrier-validation", "projection-store", "state-store"]) for (const mode of ["down", "slowdown"]) { const r = run(component, mode); console.log(`${component.padEnd(20)}${mode.padEnd(11)}${String(r.hit).padStart(8)}` + `${(r.saturated.join(",") || "-").padStart(42)}${`${r.flows}/${FLOWS.length}`.padStart(6)}` + `${r.rate.toFixed(2).padStart(19)}`); } const down = run("carrier-validation", "down"), slow = run("carrier-validation", "slowdown"); const g = (r) => (r.occupied["gateway"] / SLOTS).toFixed(3); console.log(`\ncarrier validation: down ${down.rate.toFixed(2)} req/s and ${down.hit} component(s), ` + `slowdown ${slow.rate.toFixed(2)} and ${slow.hit} -> x${(slow.rate / down.rate).toFixed(2)} requests`); console.log(`gateway utilization: failure-free ${g(t)} -> down ${g(down)} -> slowdown ${g(slow)}`); console.log(`sensitivity: with 48 slots, utilization under slowdown would be ${(slow.occupied["gateway"] / 48).toFixed(3)} ` + `(no overflow); with a 20 ms round it would be ${(2 * slow.occupied["gateway"] / SLOTS).toFixed(3)}`); const COUNT = 3, DURATION = 10, SHARE = 28.2; // AY3; K01 fundamental-properties/04: monthly failure share (minutes) console.log(`\nAY3: ${COUNT} x ${DURATION} min a month = ${COUNT * DURATION} min, ${((100 * COUNT * DURATION) / SHARE).toFixed(1)}% ` + `of K01's ${SHARE}-min failure share (at a 5 min duration it would be ${((100 * COUNT * 5) / SHARE).toFixed(1)}%)`); console.log(`the same ${COUNT * DURATION} minutes: down ${(COUNT * DURATION * 60 * down.rate).toFixed(0)} requests, ` + `slowdown ${(COUNT * DURATION * 60 * slow.rate).toFixed(0)} requests, difference ${(COUNT * DURATION * 60 * (slow.rate - down.rate)).toFixed(0)}`); ``` ``` model: 8 components, 4 flows, 32 slots per component, timeout 20 rounds, 1 round = 10 ms failure-free day: gateway 0.703, tracking-service 0.391, projection-store 0.321, event-endpoint 0.152, carrier-validation 0.061, state-store 0.642, end-of-day-billing 0.781, log-consumer 0.152 broken component mode hit saturated flows affected req/s carrier-validation down 1 - 1/4 97.22 carrier-validation slowdown 2 gateway 2/4 513.89 projection-store down 1 - 2/4 513.89 projection-store slowdown 3 gateway,tracking-service,projection-store 3/4 611.11 state-store down 1 - 3/4 1027.77 state-store slowdown 3 gateway,state-store,end-of-day-billing 4/4 1444.44 carrier validation: down 97.22 req/s and 1 component(s), slowdown 513.89 and 2 -> x5.29 requests gateway utilization: failure-free 0.703 -> down 0.612 -> slowdown 1.189 sensitivity: with 48 slots, utilization under slowdown would be 0.793 (no overflow); with a 20 ms round it would be 2.378 AY3: 3 x 10 min a month = 30 min, 106.4% of K01's 28.2-min failure share (at a 5 min duration it would be 53.2%) the same 30 minutes: down 174996 requests, slowdown 925002 requests, difference 750006 ``` The end-of-day billing job runs in its own window; showing it alongside the online flows gives the worst case. ## Same Component, Two Modes, a Fivefold Difference The reading starts from one pair of rows. When carrier validation **goes down**, the event endpoint errors in one round, releases its slot, and rejects the request. Only the state event is affected: 97.22 req/s, one component. The tracking query is untouched. When the same dependency **slows down**, the event endpoint holds its slot for twenty rounds on every call, and that hold propagates upward: the gateway's leg calling the event endpoint climbs from 6 to 22 rounds, utilization goes from 0.703 to 1.189, and the gateway becomes **saturated**. The tracking query now cannot be answered either, though it never touches the broken dependency. Affected rate: 513.89 req/s, components hit: 2 — **5.29 times the requests.** This is the course's first law and the reason for the next three lessons: **slowdown costs more than being down.** A down dependency kills its own flow and stops there; a slowing one consumes a shared resource and drags unrelated flows with it. Under the down mode, gateway utilization **falls** from 0.703 to 0.612 — a failure relieves the gateway, since rejected requests hold no resources. The remaining rows confirm the pattern. When the projection store goes down, 513.89 req/s is affected; slowing down saturates the whole chain, and the rate becomes 611.11. When the state store goes down, 1027.77 req/s is affected but nothing saturates; slowing down hits all four flows at once, 1444.44 req/s. The sensitivity row says what saturation depends on. With a 48-slot pool, slowdown utilization would stay at 0.793, the gateway would not overflow, and blast radius would fall from two components to one; at a 20 ms round, utilization would be 2.378. **Blast radius comes not from the failure itself but from the product of pool size and timeout.** The last two lines tie the two days to the outage budget. AY3 — **slowdown happens three times a month, ten minutes each.** Rationale: K01 assumed a ten-minute recovery time and fit 2.82 failures into a monthly failure share of 28.2 minutes; three failures eat 106.4 percent of it. Sensitivity: at five minutes the share drops to 53.2 percent. The same 30 minutes hits 174,996 requests under down and 925,002 under slowdown — **a difference of 750,006.** This is the vocabulary's gain on a failing day: it cuts no failure, but makes confusing the two modes countable. ## Summary - A failure mode is a component's form of breakdown; the scenarios map to five modes — down, slowdown, partial failure, stale content, network partition. A job stalling halfway through is invisible to call observation. - The signature partly tells the modes apart: down is 1.00 slot-rounds and 1.000 error; slowdown is 20.00 slot-rounds and 1.000 timeout; partial failure is 0.120 error; stale content is set apart from healthy only by the freshness stamp. - Slowdown and network partition look identical to the caller, and the content check does not separate them either; under a partition the other side stays up and may have processed the request. - A failure-free day costs the 8-byte stamp on every response (4111.12 bytes/s) and the twenty-call window naming requires (20.6 rounds at 97.22 events/s, 206 ms). - Blast radius depends on the mode: carrier validation going down affects 97.22 req/s and 1 component; slowing down affects 513.89 req/s and 2 — 5.29 times. Gateway utilization falls from 0.703 to 0.612 under down and climbs to 1.189 under slowdown. - In AY3's 30 minutes, the two modes differ by 750,006 requests — 106.4 percent of K01's monthly failure share. ## Next Step The measurement points to a direction. Slowdown is expensive not because the dependency is broken but because the caller **keeps calling it**: every call holds a slot for twenty rounds that do nothing useful. The same table shows the fix — under down, utilization **falls** to 0.612, because a fast failure holds no resources. Stopping calls to a slowing dependency turns the slowdown into a down, cutting blast radius from 513.89 to 97.22 req/s. The next lesson designs the component that makes that call: at what threshold it decides to cut, how many calls are never made while cut off, and what wrongly cutting off a healthy dependency costs.