--- title: 'Resilience and Reliability' source: 'https://academia.sh/en/courses/resilience-patterns' language: en updated: '2026-08-23T07:01:29+00:00' license: 'CC BY-SA 4.0' lessons: 20 outcomes: - 'Identifies failure modes in advance and applies isolation patterns' - 'Defines failover and recovery objectives numerically' - 'Builds compensation mechanisms in distributed workflows' --- # Resilience and Reliability Removing the assumption that the parts work: naming partial failure, slowdown, and network partition as failure modes, cutting the spread of a failure with the circuit breaker, the bulkhead pattern, retries, timeout budgets, backpressure, throttling, and graceful degradation, tying failover and recovery objectives to the downtime budget, and building idempotence, compensating transactions, and staged migration arrangements in distributed workflows; each pattern's cost on a failure-free day and its gain on a failing day are counted separately. ## Fault Isolation Patterns that cut a component's failure from spreading through the whole system: recognizing partial failure, slowdown, and network partition as separate failure modes, taking a broken dependency out of the circuit with a circuit breaker, splitting resource pools into bulkheads, the risk of retrying turning into a storm, budgeting the timeout across the chain, reporting excess load back to the source, controlled refusal of service, and degrading while the core function is preserved. ### Failure Modes Source: https://academia.sh/en/courses/resilience-patterns/failure-modes Removing the assumption that the parts work: naming partial failure, slowdown, stale content, and network partition as separate failure modes, telling a mode apart by its signature from the caller's eye, showing where two modes give the same signature, and comparing a component going down against slowing down in terms of blast radius. Across four courses, the system never broke, not once. A request entered at the edge, passed through the cache, was load balanced, landed on a service, reached a store, and returned; every computation assumed the parts **worked**. Once that assumption is removed, the first question is not "what to do" but "what happened." When a dependency does not answer, the caller has only its own observation, and very different breakdowns can hide behind it; the right response depends on which one it is. This lesson builds no pattern; the course's first pattern is next. What it builds is a **vocabulary**: the forms of breakdown are named, how they look from the caller's eye is measured, and how far each spreads through the system is counted. #### Vocabulary: Five Forms of Breakdown A **failure mode** is a component's form of breakdown: not what it does but **how** it does it wrong. The system's own structure yields five scenarios, each an instance of a mode. - **Down.** A shard node goes down; the call errors immediately. - **Slowdown.** The carrier notification endpoint slows down; the response arrives, but later than the caller can wait. - **Partial failure.** One of eight shards goes down; some calls error, the rest respond. - **Stale content.** The log consumer feeding the projection falls behind; the response is fast and correctly shaped, but stale. - **Network partition.** The link between two regions breaks; both sides stay up, neither can see the other. Here, network partition counts only as a failure mode; the choice between consistency and service availability under a partition was set in the Introduction to System Design course. The Relational Database Administration course's **partitioning** and the Scaling the Data Layer course's **sharding** are separate from **network partition**: the first two are decisions to split data, the split here an unwanted, unintended break. A sixth is added to the five, marking the vocabulary's limit: **the end-of-day billing job stalls halfway through.** No call errors, none slows down; the measurement below **cannot see this mode at all**. Seeing a mode needs an observation aimed at it. #### The Signature the Caller Sees The run below is a **model**. A round is an abstract step; no real network, cluster, or container is set up, the dependency's behavior is a parameter, and randomness comes from a custom-written, seeded generator. The assumptions are this course's own, not added to K01's table. **AY1 — round-to-second cycle: 1 round = 10 ms.** Rationale: the Application Layer and Service Interaction course counted a step call's usual duration as 8 ms; a two-round call comes to 20 ms. **AY2 — 32 concurrent slots per component, timeout 20 rounds.** Rationale: at AY1, 20 rounds is exactly 200 ms, that is, K01's read threshold itself; 32 slots give a peak utilization of 0.78 on a failure-free day. Their sensitivities are printed in the second run. **AY4 — on a failure-free day, 0.025 of calls time out.** Rationale: brief delays happen even without a failure. Its sensitivity is given below at 0.050. ```js // modes/signature.mjs — the CALLER's-eye signature of the six failure modes. MODEL: a round is an // abstract step, the dependency's behavior is a parameter, randomness comes from a seeded generator. const N = 400, USUAL = 2, TIMEOUT_ROUNDS = 20; // call count; usual call 2 rounds; AY2: timeout 20 rounds const SHARDS = 8, DOWN_SHARDS = 1; // partial failure: one of eight shards is down (M19/K04) const RARE = 0.025; // AY4: 1/40 of calls time out on a failure-free day let s = 20260730 % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const RESPONSE = { kind: "response", rounds: USUAL, fresh: true }; const ERROR = { kind: "error", rounds: 1, fresh: null }; const TIMEOUT = { kind: "timeout", rounds: TIMEOUT_ROUNDS, fresh: null }; function call(mode) { // one call's outcome: kind, rounds held, content freshness if (mode === "down") return ERROR; if (mode === "slowdown" || mode === "partition") return TIMEOUT; if (mode === "stale") return { ...RESPONSE, fresh: false }; if (mode === "partial") return random() < DOWN_SHARDS / SHARDS ? ERROR : RESPONSE; return random() < RARE ? TIMEOUT : RESPONSE; } function signature(mode) { // signature vector of N calls const c = Array.from({ length: N }, () => call(mode)); const y = c.filter((x) => x.kind === "response"); const ratio = (t) => c.filter((x) => x.kind === t).length / N; return { response: y.length / N, error: ratio("error"), timeout: ratio("timeout"), slotRounds: c.reduce((t, x) => t + x.rounds, 0) / N, fresh: y.length === 0 ? null : y.filter((x) => x.fresh).length / y.length }; } const MODES = ["healthy", "down", "slowdown", "partial", "stale", "partition"]; const b = (x) => (x === null ? "-" : x.toFixed(3)); const signatures = Object.fromEntries(MODES.map((k) => [k, signature(k)])); console.log(`model: ${N} calls per mode, usual call ${USUAL} rounds, timeout ${TIMEOUT_ROUNDS} rounds, seed 20260730`); console.log(`\n${"mode".padEnd(11)}${"response".padStart(9)}${"error".padStart(7)}${"timeout".padStart(13)}` + `${"slot-rounds/call".padStart(17)}${"fresh content".padStart(14)}`); for (const k of MODES) { const i = signatures[k]; console.log(`${k.padEnd(11)}${b(i.response).padStart(9)}${b(i.error).padStart(7)}${b(i.timeout).padStart(13)}` + `${i.slotRounds.toFixed(2).padStart(17)}${b(i.fresh).padStart(14)}`); } const equal = (a, c, fields) => fields.every((k) => Math.abs((a[k] ?? -1) - (c[k] ?? -1)) < 0.02); console.log("\nmode pairs the class ratios cannot separate:"); for (let i = 0; i < MODES.length; i += 1) for (let j = i + 1; j < MODES.length; j += 1) if (equal(signatures[MODES[i]], signatures[MODES[j]], ["response", "error", "timeout"])) console.log(` ${MODES[i]} = ${MODES[j]} -> content check ` + `${equal(signatures[MODES[i]], signatures[MODES[j]], ["fresh"]) ? "does not separate them either" : "separates them"}`); const WINDOW = 20, STAMP = 8, EVENT = 97.22, EDGE = 513.89, CYCLE = 100; // K01 rates; AY1 console.log(`\nfailure-free day's cost: an ${STAMP}-byte freshness stamp per response = ` + `${(STAMP * EDGE).toFixed(2)} bytes/s (K01's edge rate of ${EDGE} req/s); naming needs a window of ${WINDOW} calls = ` + `${(WINDOW / (EVENT / CYCLE)).toFixed(1)} rounds (${EVENT} events/s) = ` + `${((WINDOW / (EVENT / CYCLE) * 1000) / CYCLE).toFixed(0)} ms`); console.log(`sensitivity: if AY4 were 0.050, slot-rounds/call would be ${(USUAL + 0.05 * (TIMEOUT_ROUNDS - USUAL)).toFixed(2)} instead of ` + `${signatures.healthy.slotRounds.toFixed(2)}`); ``` ``` model: 400 calls per mode, usual call 2 rounds, timeout 20 rounds, seed 20260730 mode response error timeout slot-rounds/call fresh content healthy 0.988 0.000 0.013 2.23 1.000 down 0.000 1.000 0.000 1.00 - slowdown 0.000 0.000 1.000 20.00 - partial 0.880 0.120 0.000 1.88 1.000 stale 1.000 0.000 0.000 2.00 0.000 partition 0.000 0.000 1.000 20.00 - mode pairs the class ratios cannot separate: healthy = stale -> content check separates them slowdown = partition -> content check does not separate them either failure-free day's cost: an 8-byte freshness stamp per response = 4111.12 bytes/s (K01's edge rate of 513.89 req/s); naming needs a window of 20 calls = 20.6 rounds (97.22 events/s) = 206 ms sensitivity: if AY4 were 0.050, slot-rounds/call would be 2.90 instead of 2.23 ``` These numbers belong to the measurement class: they come from a model run on this machine and are deterministic. The table's most important column is **slot-rounds per call**: how many rounds the caller holds its slot for. Down is 1.00, slowdown 20.00. **A down dependency frees the caller's resource; a slowing one holds it** — twenty times the difference for the same call. Partial failure sits between: its error ratio is 0.120, one in eight, so a retrying caller may still land on a healthy shard — a hope down offers none of. Stale content, by contrast, is **indistinguishable from healthy** in the class ratios — only the freshness stamp sets it apart. The last row is the lesson's line of distinction. **Slowdown and network partition look identical to the caller**, and the content check does not separate them either: neither returns a response, and both hold the slot until timeout. The difference is not in the call — under a partition the other side **is up and may have processed the request**. A retried write is only delayed under slowdown; under a partition it can apply twice. The vocabulary's failure-free-day cost sits in the last two lines. Making stale content visible costs an 8-byte freshness stamp on every response (4111.12 bytes/s at K01's edge rate of 513.89 req/s), and naming a mode needs a window of twenty calls — 20.6 rounds at the carrier's event rate, 206 ms. #### Blast Radius A failure's cost sits not in the broken component but in **where it spreads**. **Blast radius** measures the components that cannot serve during a failure and the request rate affected. The second run models the system as a call tree: slots held in a component equal **flow rate times hold duration**; once the total exceeds the pool, the component becomes **saturated** and looks slow to its own caller. ```js // modes/blast-radius.mjs — a failure mode's blast radius. MODEL: spread comes out of slot occupancy. // Rates come from K01's peak computation, structure comes from K02-K04. const CYCLE = 100, SLOTS = 32, TIMEOUT_ROUNDS = 20, USUAL = 2; // AY1: 1 round = 10 ms; AY2: slots and timeout const d = (name, ...children) => ({ name, children }); const FLOWS = [ // K01 peak: 416.67 + 97.22 = 513.89 req/s { name: "tracking query", rate: 416.67, root: d("gateway", d("tracking-service", d("projection-store"))) }, { name: "state event", rate: 97.22, root: d("gateway", d("event-endpoint", d("carrier-validation"), d("state-store"))) }, { name: "end-of-day billing", rate: 833.33, root: d("end-of-day-billing", d("state-store")) }, { name: "log consumption", rate: 97.22, root: d("log-consumer", d("state-store"), d("projection-store")) }, ]; function hold(n, broken, mode) { // rounds held; no further calls are made after the first failure if (n.name === broken) return mode === "down" ? { rounds: 1, ok: false } : { rounds: TIMEOUT_ROUNDS, ok: false }; if (n.children.length === 0) return { rounds: USUAL, ok: true }; let rounds = 1, ok = true; for (const c of n.children) { const r = hold(c, broken, mode); rounds += r.rounds; if (!r.ok) { ok = false; break; } } return { rounds, ok }; } function run(broken, mode) { // occupancy per component = flow rate x hold duration const occupied = {}, through = {}; const walk = (n, a) => { occupied[n.name] = (occupied[n.name] ?? 0) + (a.rate / CYCLE) * hold(n, broken, mode).rounds; (through[n.name] ??= new Set()).add(a.name); n.children.forEach((c) => walk(c, a)); }; FLOWS.forEach((a) => walk(a.root, a)); const saturated = Object.keys(occupied).filter((k) => occupied[k] >= SLOTS); const hit = new Set(broken ? [...saturated, broken] : saturated); const flows = new Set([...hit].flatMap((v) => [...through[v]])); const rate = [...flows].reduce((t, a) => t + FLOWS.find((x) => x.name === a).rate, 0); return { occupied, saturated, hit: hit.size, flows: flows.size, rate }; } const t = run(null, null); console.log(`model: ${Object.keys(t.occupied).length} components, ${FLOWS.length} flows, ${SLOTS} ` + `slots per component, timeout ${TIMEOUT_ROUNDS} rounds, 1 round = ${1000 / CYCLE} ms\n`); console.log(`failure-free day: ` + Object.entries(t.occupied) .map(([k, v]) => `${k} ${(v / SLOTS).toFixed(3)}`).join(", ")); console.log(`\n${"broken component".padEnd(20)}${"mode".padEnd(11)}${"hit".padStart(8)}` + `${"saturated".padStart(42)}${"flows".padStart(6)}${"affected req/s".padStart(19)}`); for (const component of ["carrier-validation", "projection-store", "state-store"]) for (const mode of ["down", "slowdown"]) { const r = run(component, mode); console.log(`${component.padEnd(20)}${mode.padEnd(11)}${String(r.hit).padStart(8)}` + `${(r.saturated.join(",") || "-").padStart(42)}${`${r.flows}/${FLOWS.length}`.padStart(6)}` + `${r.rate.toFixed(2).padStart(19)}`); } const down = run("carrier-validation", "down"), slow = run("carrier-validation", "slowdown"); const g = (r) => (r.occupied["gateway"] / SLOTS).toFixed(3); console.log(`\ncarrier validation: down ${down.rate.toFixed(2)} req/s and ${down.hit} component(s), ` + `slowdown ${slow.rate.toFixed(2)} and ${slow.hit} -> x${(slow.rate / down.rate).toFixed(2)} requests`); console.log(`gateway utilization: failure-free ${g(t)} -> down ${g(down)} -> slowdown ${g(slow)}`); console.log(`sensitivity: with 48 slots, utilization under slowdown would be ${(slow.occupied["gateway"] / 48).toFixed(3)} ` + `(no overflow); with a 20 ms round it would be ${(2 * slow.occupied["gateway"] / SLOTS).toFixed(3)}`); const COUNT = 3, DURATION = 10, SHARE = 28.2; // AY3; K01 fundamental-properties/04: monthly failure share (minutes) console.log(`\nAY3: ${COUNT} x ${DURATION} min a month = ${COUNT * DURATION} min, ${((100 * COUNT * DURATION) / SHARE).toFixed(1)}% ` + `of K01's ${SHARE}-min failure share (at a 5 min duration it would be ${((100 * COUNT * 5) / SHARE).toFixed(1)}%)`); console.log(`the same ${COUNT * DURATION} minutes: down ${(COUNT * DURATION * 60 * down.rate).toFixed(0)} requests, ` + `slowdown ${(COUNT * DURATION * 60 * slow.rate).toFixed(0)} requests, difference ${(COUNT * DURATION * 60 * (slow.rate - down.rate)).toFixed(0)}`); ``` ``` model: 8 components, 4 flows, 32 slots per component, timeout 20 rounds, 1 round = 10 ms failure-free day: gateway 0.703, tracking-service 0.391, projection-store 0.321, event-endpoint 0.152, carrier-validation 0.061, state-store 0.642, end-of-day-billing 0.781, log-consumer 0.152 broken component mode hit saturated flows affected req/s carrier-validation down 1 - 1/4 97.22 carrier-validation slowdown 2 gateway 2/4 513.89 projection-store down 1 - 2/4 513.89 projection-store slowdown 3 gateway,tracking-service,projection-store 3/4 611.11 state-store down 1 - 3/4 1027.77 state-store slowdown 3 gateway,state-store,end-of-day-billing 4/4 1444.44 carrier validation: down 97.22 req/s and 1 component(s), slowdown 513.89 and 2 -> x5.29 requests gateway utilization: failure-free 0.703 -> down 0.612 -> slowdown 1.189 sensitivity: with 48 slots, utilization under slowdown would be 0.793 (no overflow); with a 20 ms round it would be 2.378 AY3: 3 x 10 min a month = 30 min, 106.4% of K01's 28.2-min failure share (at a 5 min duration it would be 53.2%) the same 30 minutes: down 174996 requests, slowdown 925002 requests, difference 750006 ``` The end-of-day billing job runs in its own window; showing it alongside the online flows gives the worst case. #### Same Component, Two Modes, a Fivefold Difference The reading starts from one pair of rows. When carrier validation **goes down**, the event endpoint errors in one round, releases its slot, and rejects the request. Only the state event is affected: 97.22 req/s, one component. The tracking query is untouched. When the same dependency **slows down**, the event endpoint holds its slot for twenty rounds on every call, and that hold propagates upward: the gateway's leg calling the event endpoint climbs from 6 to 22 rounds, utilization goes from 0.703 to 1.189, and the gateway becomes **saturated**. The tracking query now cannot be answered either, though it never touches the broken dependency. Affected rate: 513.89 req/s, components hit: 2 — **5.29 times the requests.** This is the course's first law and the reason for the next three lessons: **slowdown costs more than being down.** A down dependency kills its own flow and stops there; a slowing one consumes a shared resource and drags unrelated flows with it. Under the down mode, gateway utilization **falls** from 0.703 to 0.612 — a failure relieves the gateway, since rejected requests hold no resources. The remaining rows confirm the pattern. When the projection store goes down, 513.89 req/s is affected; slowing down saturates the whole chain, and the rate becomes 611.11. When the state store goes down, 1027.77 req/s is affected but nothing saturates; slowing down hits all four flows at once, 1444.44 req/s. The sensitivity row says what saturation depends on. With a 48-slot pool, slowdown utilization would stay at 0.793, the gateway would not overflow, and blast radius would fall from two components to one; at a 20 ms round, utilization would be 2.378. **Blast radius comes not from the failure itself but from the product of pool size and timeout.** The last two lines tie the two days to the outage budget. AY3 — **slowdown happens three times a month, ten minutes each.** Rationale: K01 assumed a ten-minute recovery time and fit 2.82 failures into a monthly failure share of 28.2 minutes; three failures eat 106.4 percent of it. Sensitivity: at five minutes the share drops to 53.2 percent. The same 30 minutes hits 174,996 requests under down and 925,002 under slowdown — **a difference of 750,006.** This is the vocabulary's gain on a failing day: it cuts no failure, but makes confusing the two modes countable. #### Summary - A failure mode is a component's form of breakdown; the scenarios map to five modes — down, slowdown, partial failure, stale content, network partition. A job stalling halfway through is invisible to call observation. - The signature partly tells the modes apart: down is 1.00 slot-rounds and 1.000 error; slowdown is 20.00 slot-rounds and 1.000 timeout; partial failure is 0.120 error; stale content is set apart from healthy only by the freshness stamp. - Slowdown and network partition look identical to the caller, and the content check does not separate them either; under a partition the other side stays up and may have processed the request. - A failure-free day costs the 8-byte stamp on every response (4111.12 bytes/s) and the twenty-call window naming requires (20.6 rounds at 97.22 events/s, 206 ms). - Blast radius depends on the mode: carrier validation going down affects 97.22 req/s and 1 component; slowing down affects 513.89 req/s and 2 — 5.29 times. Gateway utilization falls from 0.703 to 0.612 under down and climbs to 1.189 under slowdown. - In AY3's 30 minutes, the two modes differ by 750,006 requests — 106.4 percent of K01's monthly failure share. #### Next Step The measurement points to a direction. Slowdown is expensive not because the dependency is broken but because the caller **keeps calling it**: every call holds a slot for twenty rounds that do nothing useful. The same table shows the fix — under down, utilization **falls** to 0.612, because a fast failure holds no resources. Stopping calls to a slowing dependency turns the slowdown into a down, cutting blast radius from 513.89 to 97.22 req/s. The next lesson designs the component that makes that call: at what threshold it decides to cut, how many calls are never made while cut off, and what wrongly cutting off a healthy dependency costs. ### Circuit Breaker Source: https://academia.sh/en/courses/resilience-patterns/circuit-breaker Taking a broken dependency out of the circuit: the breaker protecting the caller's slot pool shared between services, the threshold being chosen by failure mode, comparing the consecutive-count and window-ratio rules across four modes, counting the calls saved and the neighboring requests rescued while open, and the cost of a false open that cuts off a healthy dependency, measured in rejected valid requests. The previous lesson left a direction: stopping calls to a dependency that is slowing down rounds the slowdown into a down, cutting the blast radius from 513.89 to 97.22 req/s. This lesson's job is to round that sentence into a component and measure, in the same failure scenario, whether it actually holds. A **circuit breaker** is a component that looks at the outcome of a dependency's recent calls and stops making calls to it for a while. It has three states, established in the Application Architecture course: **closed** lets the call through; **open** makes no call and fails the caller immediately; **half-open** lets a single probe call through and closes again or reopens by its result. The transition mechanics are not retold here. Three things change across services. First, the resource the breaker protects is the caller's **slot pool**, shared with other flows — the breaker protects not the broken dependency but the flows sharing its pool. Second, the threshold is not a preference but **a choice bound to the failure mode**: the same threshold is cheap under down, expensive under slowdown, wrong under partial failure. Third, requests rejected in the open state are real requests that have to go somewhere. #### The Same Scenario Without and With the Breaker The run below expands lesson 1's model into rounds. The slot pool sits in the gateway and is shared between the tracking query and the state event; the dependency's mode is a parameter. It is again a **model** — no real cluster, container, or chaos tool is set up, randomness comes from a seeded generator. **AY5 — the breaker's rule: open if 0.50 of the last 20 calls fail, stay open for 100 rounds (1 second).** Rationale: 20 calls is roughly a 20-round window at the carrier's event rate, that is, K01's read threshold; the 0.50 threshold sits between partial failure's 0.125 failure ratio and down/slowdown's 1.000. Its sensitivity is printed below at an open duration of 200 rounds. ```js // breaker/run.mjs — the same failure scenario without and with the breaker. MODEL: a round is an // abstract step, the gateway's slot pool is shared between two flows, the dependency's mode is a parameter. const SLOTS = 32, CYCLE = 100, T = 1200, START = 200; // AY2, AY1; 12 s run, failure at second 2 const TRACKING = 416.67, EVENT = 97.22, TRACKING_HOLD = 4; // K01 peak rates; lesson 01: tracking leg 4 rounds const MODES = { // p: the call's failure probability healthy: { p: 0.025, ok: 6, fail: 22 }, // AY4: rare timeout, holds the slot 22 rounds partial: { p: 0.125, ok: 6, fail: 3 }, // one of eight shards: fast error down: { p: 1, ok: 6, fail: 3 }, slowdown: { p: 1, ok: 6, fail: 22 }, }; function run({ mode, rule }) { let s = 20260731 % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const free = new Array(SLOTS).fill(0); const c = { trackingResponses: 0, trackingNoSlot: 0, eventResponses: 0, eventRejected: 0, eventErrors: 0, eventNoSlot: 0, calls: 0, eventSlotRounds: 0, opens: 0, detectRound: null, detectSlotRounds: null }; let trackingAccum = 0, eventAccum = 0, state = "closed", consecutive = 0, openUntil = 0, window = [], startSlotRounds = 0; const acquireSlot = (t, duration) => { const i = free.findIndex((x) => x <= t); if (i < 0) return false; free[i] = t + duration; return true; }; const opened = (t) => { // when the breaker moves from closed to open c.opens += 1; state = "open"; openUntil = t + rule.openFor; consecutive = 0; window = []; if (c.detectRound === null && t >= START && mode !== "healthy") { c.detectRound = t - START; c.detectSlotRounds = c.eventSlotRounds - startSlotRounds; } }; for (let t = 0; t < T; t += 1) { if (t === START) startSlotRounds = c.eventSlotRounds; if (state === "open" && t >= openUntil) state = "halfOpen"; trackingAccum += TRACKING / CYCLE; eventAccum += EVENT / CYCLE; const arrivals = []; // arrivals within the round are shuffled (seeded) for (let n = Math.floor(trackingAccum); n > 0; n -= 1, trackingAccum -= 1) arrivals.push("tracking"); for (let n = Math.floor(eventAccum); n > 0; n -= 1, eventAccum -= 1) arrivals.push("event"); for (let i = arrivals.length - 1; i > 0; i -= 1) { const j = Math.floor(random() * (i + 1)); [arrivals[i], arrivals[j]] = [arrivals[j], arrivals[i]]; } for (const flow of arrivals) { if (flow === "tracking") { if (acquireSlot(t, TRACKING_HOLD)) c.trackingResponses += 1; else c.trackingNoSlot += 1; continue; } if (state === "open") { c.eventRejected += 1; acquireSlot(t, 1); continue; } // open: the call is never made const cfg = MODES[mode === "healthy" || t < START ? "healthy" : mode]; const failed = random() < cfg.p; const duration = failed ? cfg.fail : cfg.ok; if (acquireSlot(t, duration) === false) { c.eventNoSlot += 1; continue; } c.calls += 1; c.eventSlotRounds += duration; if (failed) c.eventErrors += 1; else c.eventResponses += 1; if (rule === null) continue; // ruleless run: state is not tracked if (state === "halfOpen") { if (failed) opened(t); else { state = "closed"; consecutive = 0; window = []; } continue; } if (rule.kind === "consecutive") { consecutive = failed ? consecutive + 1 : 0; if (consecutive >= rule.threshold) opened(t); } else { window.push(failed ? 1 : 0); if (window.length > rule.windowSize) window.shift(); if (window.length === rule.windowSize && window.reduce((a, x) => a + x, 0) / rule.windowSize >= rule.threshold) opened(t); } } } return c; } const RATIO_RULE = { kind: "ratio", windowSize: 20, threshold: 0.5, openFor: 100 }; // AY5 const b = (x) => x.toFixed(2); const METRICS = ["trackingResponses", "trackingNoSlot", "eventResponses", "eventRejected", "eventErrors", "calls", "eventSlotRounds"]; const HEADERS = ["tracking responses", "tracking no slot", "event responses", "event rejected", "event errors", "calls made", "event slot-rounds"]; console.log(`model: ${T} rounds (${T / CYCLE} s), failure at round ${START}, ${SLOTS} slots; chosen rule: open if ` + `${RATIO_RULE.threshold} of the last ${RATIO_RULE.windowSize} calls fail, stay open ${RATIO_RULE.openFor} rounds`); console.log(`\n${"run".padEnd(24)}` + HEADERS.map((x) => x.padStart(20)).join("")); const r = {}; for (const mode of ["healthy", "partial", "slowdown", "down"]) for (const [suffix, rule] of [["no breaker", null], ["with breaker", RATIO_RULE]]) { const name = `${mode}, ${suffix}`; r[name] = run({ mode, rule }); console.log(`${name.padEnd(24)}` + METRICS.map((k) => String(r[name][k]).padStart(20)).join("")); } const h1 = r["healthy, with breaker"]; const [p0, p1] = [r["partial, no breaker"], r["partial, with breaker"]]; const [w0, w1] = [r["slowdown, no breaker"], r["slowdown, with breaker"]]; const [d0, d1] = [r["down, no breaker"], r["down, with breaker"]]; const scale = (600 * CYCLE) / (T - START); // AY3: scaling to a 10-minute failure console.log(`\nfailure-free day's cost: the breaker is queried on all ${h1.calls} calls and a ` + `${RATIO_RULE.windowSize}-record window is kept; ${h1.opens} false opens, ${h1.eventRejected} rejected valid events; ` + `partial failure opens ${p1.opens} times and event responses ${p0.eventResponses} -> ${p1.eventResponses} ` + `(${p0.eventResponses - p1.eventResponses} events that could have succeeded were rejected)`); console.log(`failing day's gain (slowdown): tracking no slot ${w0.trackingNoSlot} -> ${w1.trackingNoSlot}, ` + `event slot-rounds ${w0.eventSlotRounds} -> ${w1.eventSlotRounds} (${b((100 * (w0.eventSlotRounds - w1.eventSlotRounds)) / w0.eventSlotRounds)}% less), ` + `calls to the broken dependency ${w0.calls} -> ${w1.calls}`); console.log(`the same gain under down: tracking no slot ${d0.trackingNoSlot} -> ${d1.trackingNoSlot}, ` + `event slot-rounds ${d0.eventSlotRounds} -> ${d1.eventSlotRounds}, calls ${d0.calls} -> ${d1.calls}`); console.log(`scaled to AY3's 10 minutes: calls saved under slowdown ` + `${((w0.calls - w1.calls) * scale).toFixed(0)}, tracking requests rescued ` + `${((w0.trackingNoSlot - w1.trackingNoSlot) * scale).toFixed(0)}`); console.log(`\n${"rule".padEnd(18)}${"mode".padEnd(11)}${"detection round".padStart(15)}` + `${"slot-rounds to detection".padStart(26)}${"opens".padStart(8)}${"rejected valid events".padStart(25)}`); const RULES = [ ["consecutive 5", { kind: "consecutive", threshold: 5, openFor: 100 }], ["ratio 0.50 / 20", { kind: "ratio", windowSize: 20, threshold: 0.5, openFor: 100 }], ["ratio 0.10 / 20", { kind: "ratio", windowSize: 20, threshold: 0.1, openFor: 100 }], ]; for (const [name, rule] of RULES) for (const mode of ["healthy", "partial", "slowdown", "down"]) { const k = run({ mode, rule }); const lost = mode === "partial" ? p0.eventResponses - k.eventResponses : k.eventRejected; console.log(`${name.padEnd(18)}${mode.padEnd(11)}${String(k.detectRound ?? "-").padStart(15)}` + `${String(k.detectSlotRounds ?? "-").padStart(26)}${String(k.opens).padStart(8)}` + `${String(mode === "healthy" || mode === "partial" ? lost : "-").padStart(25)}`); } const longer = { ...RATIO_RULE, openFor: 200 }, narrower = { kind: "ratio", windowSize: 20, threshold: 0.1, openFor: 200 }; console.log(`sensitivity: if open duration were ${longer.openFor} rounds instead of ${RATIO_RULE.openFor}, calls made under slowdown ` + `would go ${w1.calls} -> ${run({ mode: "slowdown", rule: longer }).calls}; under the ratio 0.10 rule, rejected valid events ` + `on a failure-free day would go ${run({ mode: "healthy", rule: RULES[2][1] }).eventRejected} -> ` + `${run({ mode: "healthy", rule: narrower }).eventRejected}`); ``` ``` model: 1200 rounds (12 s), failure at round 200, 32 slots; chosen rule: open if 0.5 of the last 20 calls fail, stay open 100 rounds run tracking responses tracking no slot event responses event rejected event errors calls made event slot-rounds healthy, no breaker 5000 0 1135 0 31 1166 7492 healthy, with breaker 5000 0 1135 0 31 1166 7492 partial, no breaker 5000 0 1021 0 145 1166 6637 partial, with breaker 5000 0 1021 0 145 1166 6637 slowdown, no breaker 4329 671 190 0 826 1016 19312 slowdown, with breaker 5000 0 190 953 23 213 1646 down, no breaker 5000 0 190 0 976 1166 4144 down, with breaker 5000 0 190 953 23 213 1285 failure-free day's cost: the breaker is queried on all 1166 calls and a 20-record window is kept; 0 false opens, 0 rejected valid events; partial failure opens 0 times and event responses 1021 -> 1021 (0 events that could have succeeded were rejected) failing day's gain (slowdown): tracking no slot 671 -> 0, event slot-rounds 19312 -> 1646 (91.48% less), calls to the broken dependency 1016 -> 213 the same gain under down: tracking no slot 0 -> 0, event slot-rounds 4144 -> 1285, calls 1166 -> 213 scaled to AY3's 10 minutes: calls saved under slowdown 48180, tracking requests rescued 40260 rule mode detection round slot-rounds to detection opens rejected valid events consecutive 5 healthy - - 0 0 consecutive 5 partial - - 0 0 consecutive 5 slowdown 4 110 10 - consecutive 5 down 4 15 10 - ratio 0.50 / 20 healthy - - 0 0 ratio 0.50 / 20 partial - - 0 0 ratio 0.50 / 20 slowdown 9 220 10 - ratio 0.50 / 20 down 9 30 10 - ratio 0.10 / 20 healthy - - 4 386 ratio 0.10 / 20 partial 88 117 9 698 ratio 0.10 / 20 slowdown 67 22 11 - ratio 0.10 / 20 down 67 3 11 - sensitivity: if open duration were 200 rounds instead of 100, calls made under slowdown would go 213 -> 208; under the ratio 0.10 rule, rejected valid events on a failure-free day would go 386 -> 741 ``` These numbers belong to the measurement class and depend on the seed; the same seed gives the same table. The first two rows give the pattern's failure-free-day cost, and the cost is surprisingly small. On a healthy day the runs with and without the breaker are **identical**: 5000 tracking responses, 1135 event responses, 0 false opens, 0 rejected valid events. The cost is countable but small — the breaker is queried on all 1166 calls and keeps a 20-record window per dependency. **A correctly sized threshold becomes invisible on a failure-free day.** The third and fourth rows give the gain. Under slowdown, without a breaker, 671 of 5000 tracking requests find no slot — the round-level counterpart of lesson 1's blast-radius table. With the breaker, that number becomes **0**: the tracking query is no longer affected by the slowdown of a dependency it has nothing to do with. The event leg's slot-rounds fall from 19,312 to 1646, 91.48 percent less, and calls to the broken dependency drop from 1016 to 213. Scaled to AY3's 10-minute slowdown, that is 48,180 calls saved and 40,260 tracking requests rescued. The fifth and sixth rows show what the breaker **does not** rescue. Under down, tracking requests with no slot is already 0 without a breaker; a fast failure holds no slot, so there is no spread and nothing for the breaker to protect. The gain sits elsewhere: calls made fall from 1166 to 213, slot-rounds from 4144 to 1285. For a dependency that is down, the breaker cuts **not the spread, but the wasted work**. One row also confesses the pattern's own cost: 953 events are rejected in the with-breaker slowdown run. The breaker does not rescue the event flow, it **fences it off**. Those 953 rejected state events are still real events and have to go somewhere; where is not this pattern's question. #### The Threshold Is Chosen by Failure Mode The second table runs three threshold rules across four modes, and shows that no single threshold can serve all four. **Consecutive failure count is the cheapest rule, but it never sees partial failure.** The consecutive-5 rule catches both slowdown and down at round 4, and never opens falsely on a healthy day — but never opens under partial failure either. The reason is arithmetic: at a one-in-eight failure ratio, five consecutive failures is rare. Under partial failure that is correct; seven-eighths of calls still succeed, and cutting off would cost more than it gains. **It is not detection's delay that changes by mode, but its cost.** In the consecutive-5 rule, both down and slowdown are detected at round 4; concurrent calls accumulate five failures at the same rate. But slot-rounds held until detection are 15 under down and 110 under slowdown — **7.3 times.** Under ratio-0.50 the gap widens further: 30 against 220. A threshold should be chosen not by call count but by how many slot-rounds that count costs; waiting twenty calls on a slow dependency equals waiting a hundred and twenty. #### The Cost of a False Open The third rule shows the reverse. The ratio-0.10 threshold catches slowdown and down far more cheaply — only 22 and 3 slot-rounds until detection — and even sees partial failure at round 88. Its cost sits in the other two columns. On a healthy day this rule opens 4 times and **rejects 386 valid events** — while nothing is wrong with the dependency. Under partial failure it opens 9 times and rejects 698 valid events, even though those events had a seven-in-eight chance of succeeding. **A false open is cutting off a healthy dependency, and its cost is rejected valid requests.** The sensitivity row says this cost depends on open duration: raising it from 100 to 200 rounds would drop calls made under slowdown from 213 to 208 — almost no gain — but raise rejected valid events on a healthy day from 386 to 741. **Extending open duration buys little on a failing day and costs a lot on a failure-free one**, because the gain depends on the breaker opening once, not on how long the failure lasts, while the loss is multiplied directly by the open duration. #### Summary - A circuit breaker takes a dependency out of the circuit; its closed, open, and half-open states were established in the Application Architecture course and are not repeated here. Across services, it protects the caller's shared slot pool. - A correctly sized threshold is invisible on a failure-free day: the runs with and without a breaker give the same numbers, 0 false opens. The cost is one check per call and a 20-record window per dependency. - The gain under slowdown is large: tracking requests finding no slot go 671 → 0, the event leg's slot-rounds 19,312 → 1646 (91.48 percent less), calls to the broken dependency 1016 → 213. AY3's 10 minutes save 48,180 calls and rescue 40,260 tracking requests. - Under down there is no spread to begin with (tracking no slot 0 → 0); the gain is wasted work: calls 1166 → 213, slot-rounds 4144 → 1285. - Detection's cost changes by mode, not its delay: the consecutive-5 rule catches both modes at round 4, but slot-rounds held until detection are 15 under down and 110 under slowdown. - A false open cuts off a healthy dependency: ratio-0.10 rejects 386 valid events on a healthy day, 698 under partial failure; raising open duration to 200 rounds pushes the healthy-day loss to 741, while the failing-day gain only drops from 213 to 208. #### Next Step The breaker drew a boundary, but where it sits must be read carefully: protection starts **after detection**. Under ratio-0.50, slowdown is detected at round 9, and during those nine rounds the broken dependency draws 220 slot-rounds from the shared pool. Lowering the threshold shrinks that share but grows false opens; the two cannot shrink together. And a breaker sits on each dependency separately, while the pool is single: a slowdown with no breaker, or one under its threshold, still consumes the pool and drags down the neighboring flow. One question remains — can the spread be cut without depending on detection at all. The next lesson splits the pool itself: each flow gets its own slots, the resource a flow can consume is bounded from the start, and what that costs on a failure-free day is counted. ### Bulkhead Pattern Source: https://academia.sh/en/courses/resilience-patterns/bulkhead-pattern Splitting the resource pool by flow: comparing a single pool, where one slow dependency consumes the whole pool, against a bulkhead layout in the same failure scenario, the bulkhead protecting the neighboring flow without depending on detection at all, its failure-free-day cost measured in idle slots and reduced utilization, and the trade-off bulkhead size draws between the two days. The circuit breaker cut the spread, but its protection started only after detection: until the threshold filled, the broken dependency drew slots from the shared pool — 220 slot-rounds under ratio-0.50. Lowering the threshold shrank that share but grew false opens. This lesson's question: can the spread be cut without detection at all. It can, and the way is to split the pool. The **bulkhead pattern** splits the caller's resources into separate pools by flow or by dependency; a flow can only exhaust its own pool, and when it does, the neighboring flow feels nothing. The name comes from a ship hull's watertight compartments: when one compartment floods, the ship does not sink, that compartment does. **Bulkhead** shares its root with the file **splitting** in the Shell Programming course and the remainder in **modulo hashing** from the Traffic Layer course, but is a separate concept: here, what gets split is the **resource pool**. The difference from the breaker fits in one sentence. The breaker **understands** the dependency is broken and stops the call; the bulkhead understands nothing, it only bounds from the start the resource a broken flow can consume. One decides, the other sets a limit. #### Same Pool, Two Layouts The run below is again a **model**. It takes up lesson 1's gateway slot pool; the only difference is that arrivals are no longer smooth but **Poisson**, drawn from a seeded generator. The bulkhead's cost shows up only under bursty arrival: under smooth flow, each flow fills its own share exactly, and sharing gains nothing. The failure scenario is lesson 1's: carrier validation slows down, the event leg's hold climbs from 6 to 22 rounds, and by AY3 this happens three times a month, ten minutes each. ```js // bulkhead/run.mjs — comparing a single pool against a bulkhead layout in the same failure scenario. // MODEL: a round is an abstract step, arrivals are Poisson from a seeded generator, the mode is a parameter. const TOTAL = 32, CYCLE = 100, T = 4000, START = 1000; // AY2, AY1; 40 s run, failure at second 10 const TRACKING = 416.67, EVENT = 97.22; // K01: peak read and peak write req/s const HOLD = { tracking: 4, event: 6 }, SLOW = 22; // lesson 01: gateway leg durations function run({ split, mode }) { // split: single pool if null let s = 20260801 % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const poisson = (lam) => { const L = Math.exp(-lam); let k = 0, p = 1; do { k += 1; p *= random(); } while (p > L); return k - 1; }; const pool = split === null ? { shared: TOTAL } : { tracking: split, event: TOTAL - split }; const busy = Object.fromEntries(Object.keys(pool).map((h) => [h, []])); // busy slots' end round const poolFor = (flow) => (split === null ? "shared" : flow); const c = { trackingResponses: 0, trackingDropped: 0, eventResponses: 0, eventDropped: 0, idle: 0, busyRounds: 0 }; for (let t = 0; t < T; t += 1) { for (const h of Object.keys(pool)) busy[h] = busy[h].filter((x) => x > t); c.busyRounds += Object.values(busy).reduce((a, x) => a + x.length, 0); const arrivals = []; for (let n = poisson(TRACKING / CYCLE); n > 0; n -= 1) arrivals.push("tracking"); for (let n = poisson(EVENT / CYCLE); n > 0; n -= 1) arrivals.push("event"); for (let i = arrivals.length - 1; i > 0; i -= 1) { const j = Math.floor(random() * (i + 1)); [arrivals[i], arrivals[j]] = [arrivals[j], arrivals[i]]; } for (const flow of arrivals) { const h = poolFor(flow); const duration = flow === "event" && mode === "slowdown" && t >= START ? SLOW : HOLD[flow]; if (busy[h].length < pool[h]) { busy[h].push(t + duration); c[`${flow}Responses`] += 1; continue; } c[`${flow}Dropped`] += 1; for (const o of Object.keys(pool)) if (o !== h) c.idle += pool[o] - busy[o].length; } } return c; } const b = (x, n = 2) => x.toFixed(n); const METRICS = ["trackingResponses", "trackingDropped", "eventResponses", "eventDropped", "idle"]; const HEADERS = ["tracking responses", "tracking dropped", "event responses", "event dropped", "idle slots"]; console.log(`model: ${T} rounds (${T / CYCLE} s), failure at round ${START}, ${TOTAL} slots total; ` + `Poisson arrivals (seed 20260801), tracking ${(TRACKING / CYCLE).toFixed(4)} and event ${(EVENT / CYCLE).toFixed(4)} arrivals/round`); console.log(`\n${"layout".padEnd(22)}${"day".padEnd(12)}` + HEADERS.map((x) => x.padStart(20)).join("") + `${"average busy slots".padStart(20)}`); const r = {}; for (const [name, split] of [["single pool (32)", null], ["bulkhead (24/8)", 24]]) for (const mode of ["healthy", "slowdown"]) { const k = run({ split, mode }); r[`${name}|${mode}`] = k; console.log(`${name.padEnd(22)}${mode.padEnd(12)}` + METRICS.map((x) => String(k[x]).padStart(20)).join("") + `${b(k.busyRounds / T).padStart(20)}`); } const single0 = r["single pool (32)|healthy"], bulk0 = r["bulkhead (24/8)|healthy"]; const single1 = r["single pool (32)|slowdown"], bulk1 = r["bulkhead (24/8)|slowdown"]; const scale = (600 * CYCLE) / (T - START); // AY3: scaling to a 10-minute failure console.log(`\nfailure-free day's cost: tracking dropped ${single0.trackingDropped} -> ${bulk0.trackingDropped}, event dropped ` + `${single0.eventDropped} -> ${bulk0.eventDropped}; average busy slots ${b(single0.busyRounds / T)} -> ${b(bulk0.busyRounds / T)} ` + `(utilization ${b(single0.busyRounds / T / TOTAL, 3)} -> ${b(bulk0.busyRounds / T / TOTAL, 3)}); idle slots ${single0.idle} -> ${bulk0.idle}`); console.log(`failing day's gain: tracking dropped ${single1.trackingDropped} -> ${bulk1.trackingDropped} ` + `(${b((100 * (single1.trackingDropped - bulk1.trackingDropped)) / single1.trackingDropped, 1)}% less), event dropped ` + `${single1.eventDropped} -> ${bulk1.eventDropped}; tracking requests rescued in AY3's 10 minutes ` + `${((single1.trackingDropped - bulk1.trackingDropped) * scale).toFixed(0)}`); console.log(`\n${"pool split".padEnd(20)}${"healthy tracking dropped".padStart(26)}` + `${"healthy event dropped".padStart(23)}${"slowdown tracking dropped".padStart(27)}` + `${"slowdown event dropped".padStart(24)}`); for (const split of [null, 28, 24, 20, 16]) { const s = run({ split, mode: "healthy" }), y = run({ split, mode: "slowdown" }); const label = split === null ? `single pool (${TOTAL})` : `${split}/${TOTAL - split}`; console.log(`${label.padEnd(20)}${String(s.trackingDropped).padStart(26)}${String(s.eventDropped).padStart(23)}` + `${String(y.trackingDropped).padStart(27)}${String(y.eventDropped).padStart(24)}`); } const sensitivity = run({ split: 27, mode: "healthy" }); console.log(`sensitivity: with 27/5 instead of 24/8, failure-free tracking dropped would go ${bulk0.trackingDropped} -> ` + `${sensitivity.trackingDropped}, event dropped ${bulk0.eventDropped} -> ${sensitivity.eventDropped}; the event pool ` + `runs at ${b(((EVENT / CYCLE) * HOLD.event) / 8, 3)} utilization at 8 slots, ` + `${b(((EVENT / CYCLE) * HOLD.event) / 5, 3)} at 5`); ``` ``` model: 4000 rounds (40 s), failure at round 1000, 32 slots total; Poisson arrivals (seed 20260801), tracking 4.1667 and event 0.9722 arrivals/round layout day tracking responses tracking dropped event responses event dropped idle slots average busy slots single pool (32) healthy 16542 78 3931 27 0 17.31 single pool (32) slowdown 13966 2654 3358 600 0 24.00 bulkhead (24/8) healthy 16448 172 3552 406 4239 16.77 bulkhead (24/8) slowdown 16448 172 1947 2011 19196 18.90 failure-free day's cost: tracking dropped 78 -> 172, event dropped 27 -> 406; average busy slots 17.31 -> 16.77 (utilization 0.541 -> 0.524); idle slots 0 -> 4239 failing day's gain: tracking dropped 2654 -> 172 (93.5% less), event dropped 600 -> 2011; tracking requests rescued in AY3's 10 minutes 49640 pool split healthy tracking dropped healthy event dropped slowdown tracking dropped slowdown event dropped single pool (32) 78 27 2654 600 28/4 11 1754 11 2876 24/8 172 406 172 2011 20/12 868 37 868 1416 16/16 2591 0 2591 942 sensitivity: with 27/5 instead of 24/8, failure-free tracking dropped would go 172 -> 26, event dropped 406 -> 1303; the event pool runs at 0.729 utilization at 8 slots, 1.167 at 5 ``` These numbers belong to the measurement class and depend on the seed. #### The Bulkhead Does Not Change the Failing Day Under the bulkhead layout, the tracking query's dropped-request count is **172** on a healthy day and **172** on a slowdown day. The same number. Whether the dependency slows down or not changes nothing for the tracking flow — because the event flow's slots are capped at eight, and those eight can never touch the tracking flow's twenty-four. In a single pool, the same failure drops 2654 tracking requests. The bulkhead cuts that to 172: 93.5 percent less. Scaled to AY3's 10-minute slowdown, that is 49,640 tracking requests rescued. And this protection **depends on no detection at all**; the breaker's nine-round detection window does not exist here, because no decision is made. What the pattern does not do sits in the same row. Under the bulkhead, the event flow's dropped requests **rise** from 600 to 2011. The bulkhead does not rescue the event flow, it chokes it earlier instead; what it rescues is the neighbor. The pattern's name already says so — the flooding compartment keeps filling. #### The Failure-Free Day's Cost: Lost Sharing The first two rows give the cost, and it is not small. On a healthy day, the single pool drops 78 tracking and 27 event requests; 105 total. The same day, the bulkhead layout drops 172 tracking and 406 event requests; 578 total. **With no failure at all, the bulkhead multiplies dropped requests by 5.5.** The reason sits in the `idle slots` column. Whenever a request drops under the bulkhead, the slots sitting idle in the other pool are added up, and on a healthy day this total comes to 4239: 4239 times, a request was rejected while an idle slot sat in the neighboring pool. In a single pool this number is zero by definition, because an idle slot does not care whose it is. Average busy slots fall from 17.31 to 16.77, utilization from 0.541 to 0.524 — not doing the same work with fewer resources, but doing it with less of the same resource. What is lost is called **sharing**: when two bursty flows share a single pool, one's peak overlaps the other's trough, and the combined peak comes out smaller than the sum of the peaks. The bulkhead forbids that overlap. This is the failure-free day's cost, and it is directly measurable: 94 extra dropped tracking requests, 379 extra dropped event requests, 0.017 less utilization. #### Bulkhead Size Trades the Two Days Against Each Other The second table shows the sizing choice, and that no single split is right. A 28/4 split protects the tracking flow almost perfectly: under slowdown, only 11 requests drop, against the single pool's 2654. Its cost sits on the failure-free day — the event flow's four slots are not enough, and 1754 events drop on a healthy day. A 16/16 split does the opposite: the event flow never drops on a failure-free day, but the tracking flow loses 2591 requests — as much as the single pool loses on the failing day. **A wrongly sized bulkhead does on a failure-free day as much damage as a failing one.** 24/8 sits between the two, and its choice is measurable: the event flow's average load is 0.9722 arrivals/round times 6 rounds, that is, 5.83 slots; eight slots is 1.37 times that. The sensitivity row gives the cost of narrowing the share — at 27/5, tracking dropped falls from 172 to 26, but event dropped climbs from 406 to 1303, because the event pool's utilization goes from 0.729 to 1.167, past capacity. **A bulkhead shrunk below its own flow's usual load fails without waiting for a failure.** The bulkhead and the breaker do not replace each other. The bulkhead cuts the spread without detection, but does not cut calls to the broken dependency: in the bulkhead slowdown run, the event flow still calls the broken dependency and still exhausts its own eight slots. The breaker stops those calls, but has to understand first. Each moves a different number — the bulkhead moves the neighbor's dropped requests, the breaker moves calls to the broken dependency. #### Summary - A bulkhead pattern splits the caller's resource pool by flow; a flow can only exhaust its own pool. Its difference from the breaker is that it decides nothing — the boundary is set from the start, no detection needed. - The bulkhead makes the failing day invisible to the neighboring flow: under the bulkhead layout, the tracking query's dropped requests are 172 on both the healthy and the slowdown day. In a single pool, the same failure drops 2654 requests; the gain is 93.5 percent, 49,640 requests in AY3's 10 minutes. - The bulkhead does not rescue the broken flow, it chokes it earlier: the event flow's dropped requests rise from 600 to 2011. - The failure-free day's cost is lost sharing: dropped requests go from 105 to 578 (5.5 times), utilization from 0.541 to 0.524; 4239 times a request is rejected while an idle slot sits in the neighboring pool. - Size trades the two days against each other: 28/4 drops only 11 tracking requests under slowdown but 1754 event requests on a failure-free day; 16/16 never drops the event flow but loses 2591 tracking requests on a failure-free day. - A bulkhead must carry its own flow's usual load: the event flow needs 5.83 slots, 8 slots is 1.37 times that; at 5 slots utilization becomes 1.167 and the bulkhead fails without a failure. #### Next Step Together, the bulkhead and the breaker draw one boundary: the broken dependency stays in its own bulkhead and, after a while, is not called at all. Both stand on the same assumption — a failed call is made **once**. A real caller does not meet failure that way; it retries, since most errors are transient and a second attempt usually succeeds. Retrying changes everything these two patterns measure: bulkhead slots run out faster, more failures land in the breaker's window, and, most importantly, the broken dependency receives more requests from the system already straining it. The next lesson measures that multiplier: how far the request rate reaching the dependency multiplies if every chain layer retries, how the storm stretches the failure out, and which layer should hold the retry budget. ### Retries and Storm Risk Source: https://academia.sh/en/courses/resilience-patterns/retries-and-storm-risk Retrying turning into a storm under the call amplification: measuring how far the call rate reaching a broken dependency multiplies when every layer of the chain retries, separating the arithmetic upper bound from the multiplier actually observed under backoff, tying the multiplier to the retry allowance under an outage and to the failure ratio under degradation, and cutting the storm by spreading a retry budget across the chain. The bulkhead and the circuit breaker fenced off the spread, but both stood on the same assumption: a failed call is made once. A real caller does not behave that way. Most errors are transient, a second attempt usually succeeds, and retries get added to nearly every layer. This lesson's question is what that decision does **to the broken dependency**. The decision to retry, exponential backoff, a cap, full jitter, and the retry budget were established and measured in the Application Architecture course; the Caching, Queues and Asynchronous Processing course measured backoff schedules. In every run below, backoff and full jitter are explicit and the seed is fixed — so **only the multiplier** decides the comparison. Jitter's arithmetic is not recomputed here. What gets measured is something else. The chain is three layers deep: the gateway calls the event endpoint, the event endpoint calls the state store. Each layer makes a reasonable decision on its own — "retry three times on failure" — and none knows the others retry too. The result multiplies: three layers retrying three times each can put twenty-seven calls on the bottom dependency for one external request. This is called a **retry storm**, and by definition it happens at the worst moment. #### Two Failure Profiles The run below is a **model**: a round is an abstract step, arrivals are Poisson from a seeded generator, and the dependency's capacity is calls per round. No real cluster, queue, or chaos tool is set up. **AY6 — the dependency's usual capacity is 6 calls per round (600 req/s).** Rationale: K01's peak write rate is 97.22 req/s, that is, 0.9722 calls per round; a sixfold margin makes overflow rare under bursty arrival. Its sensitivity is given below together with the retry budget. The failure runs under two profiles, because the multiplier behaves differently in each. **Outage:** the dependency answers no call for one second (capacity 0) — lesson 1's down mode. **Degradation:** its capacity drops to a sixth for ten seconds (capacity 1) — a portion of calls still succeed. ```js // storm/run.mjs — the retry storm under the call amplification. MODEL: a round is an abstract step, // arrivals are Poisson, backoff is exponential with full jitter (its arithmetic measured in M14/K06). const CYCLE = 100, T = 4000, START = 500; // AY1: 1 round = 10 ms; 40 s run, failure at second 5 const EVENT = 97.22, USUAL = 6; // K01 peak write req/s; AY6: usual capacity calls/round const BASE = 2, CAP = 32, RARE = 0.025; // backoff base/cap (rounds); AY4 const SCENARIOS = { // window duration and capacity kept during the window healthy: { duration: 0, capacity: USUAL }, "outage (1 s)": { duration: 100, capacity: 0 }, "degradation (10 s)": { duration: 1000, capacity: 1 }, }; function run({ tries, budget, scenario }) { let s = 20260802 % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const poisson = (lam) => { const L = Math.exp(-lam); let k = 0, p = 1; do { k += 1; p *= random(); } while (p > L); return k - 1; }; const { duration, capacity: broken } = SCENARIOS[scenario]; const allowed = tries.reduce((a, x) => a * x, 1); // allowed retries = product across layers const schedule = new Map(); const enqueue = (t, i) => { if (t < T) (schedule.get(t) ?? schedule.set(t, []).get(t)).push(i); }; const c = { requests: 0, calls: 0, primary: 0, extra: 0, successful: 0, dropped: 0, peak: 0, windowCalls: 0 }; for (let t = 0; t < T; t += 1) { const inWindow = duration > 0 && t >= START && t < START + duration; const capacity = inWindow ? broken : USUAL; for (let n = poisson(EVENT / CYCLE); n > 0; n -= 1) { c.requests += 1; enqueue(t, { n: 0 }); } const thisRound = schedule.get(t) ?? []; c.calls += thisRound.length; c.peak = Math.max(c.peak, thisRound.length); if (inWindow) c.windowCalls += thisRound.length; thisRound.forEach((item, i) => { if (item.n === 0) c.primary += 1; else c.extra += 1; if (i < capacity && (duration > 0 || random() >= RARE)) { c.successful += 1; return; } const budgetOk = budget === null || c.extra <= budget * c.primary; if (item.n + 1 >= allowed || budgetOk === false) { c.dropped += 1; return; } const backoff = Math.min(CAP, BASE * 2 ** item.n); // exponential backoff + full jitter enqueue(t + 1 + Math.floor(random() * backoff), { n: item.n + 1 }); }); schedule.delete(t); } return { ...c, allowed, multiplier: c.calls / c.requests, windowRate: duration === 0 ? 0 : c.windowCalls / duration }; } const b = (x, n = 2) => x.toFixed(n); const CONFIGS = [ // chain: gateway -> event endpoint -> state store ["no retries", [1, 1, 1], null], ["gateway only (3)", [3, 1, 1], null], ["two layers (3x3)", [3, 3, 1], null], ["three layers (3x3x3)", [3, 3, 3], null], ["three layers + budget 0.30", [3, 3, 3], 0.3], ]; console.log(`model: ${T} rounds (${T / CYCLE} s), failure at round ${START}; usual capacity ${USUAL} calls/round ` + `(${USUAL * CYCLE} req/s), event arrivals ${(EVENT / CYCLE).toFixed(4)}/round; seed 20260802`); const r = {}; for (const scenario of Object.keys(SCENARIOS)) { const p = SCENARIOS[scenario]; console.log(` -- ${scenario}${p.duration === 0 ? " (AY4: 0.025 rare failure)" : `: for ${p.duration} rounds, capacity ${p.capacity}`} --`); console.log(`${"configuration".padEnd(25)}${"allowed".padStart(8)}${"calls".padStart(8)}${"multiplier".padStart(11)}` + `${"peak calls/round".padStart(18)}${"successful".padStart(11)}${"dropped".padStart(9)}` + (p.duration === 0 ? "" : `${"window calls/round".padStart(20)}${"times usual rate".padStart(19)}`)); for (const [name, tries, budget] of CONFIGS) { const k = run({ tries, budget, scenario }); r[`${name}|${scenario}`] = k; console.log(`${name.padEnd(25)}${String(k.allowed).padStart(8)}${String(k.calls).padStart(8)}` + `${`x${b(k.multiplier)}`.padStart(11)}${String(k.peak).padStart(18)}${String(k.successful).padStart(11)}` + `${String(k.dropped).padStart(9)}` + (p.duration === 0 ? "" : `${b(k.windowRate).padStart(20)}${`x${b(k.windowRate / (EVENT / CYCLE), 1)}`.padStart(19)}`)); } } const at = (name, scenario) => r[`${name}|${scenario}`]; const OUTAGE = "outage (1 s)", DEGRADATION = "degradation (10 s)"; const [s0, s27] = [at("no retries", "healthy"), at("three layers (3x3x3)", "healthy")]; const [k0, k3, k9, k27, kb] = CONFIGS.map(([name]) => at(name, OUTAGE)); const [z0, z9, z27, zb] = [at("no retries", DEGRADATION), at("two layers (3x3)", DEGRADATION), at("three layers (3x3x3)", DEGRADATION), at("three layers + budget 0.30", DEGRADATION)]; console.log(` failure-free day's cost: calls ${s0.calls} -> ${s27.calls} (multiplier x${b(s27.multiplier)}), ` + `reaching the dependency ${b((s0.calls / T) * CYCLE)} -> ${b((s27.calls / T) * CYCLE)} req/s; ` + `no difference between allowed 1 and 27, because the failure ratio is 0.025`); console.log(`failing day's gain (outage): dropped requests ${k0.dropped} -> ${k3.dropped} -> ${k9.dropped} -> ` + `${k27.dropped}; successful ${k0.successful} -> ${k27.successful}`); console.log(`the storm's growth (outage): window calls/round ${b(k0.windowRate)} -> ${b(k3.windowRate)} -> ` + `${b(k9.windowRate)} -> ${b(k27.windowRate)} = x${b(k27.windowRate / (EVENT / CYCLE), 1)} the usual rate; ` + `arithmetic upper bound x${k27.allowed}`); console.log(`under degradation the multiplier follows not allowance but failure ratio: allowed 9 and 27 give the same calls ` + `(${z9.calls} and ${z27.calls}), window rate ${b(z27.windowRate)}`); console.log(`budget 0.30 spread across the chain: outage calls ${k27.calls} -> ${kb.calls}, window rate ` + `${b(k27.windowRate)} -> ${b(kb.windowRate)}, dropped ${k27.dropped} -> ${kb.dropped}; degradation calls ` + `${z27.calls} -> ${zb.calls}, dropped ${z0.dropped} -> ${z27.dropped} -> ${zb.dropped}`); console.log(`upper bound independent of the run: at d layers retrying r times, multiplier r^d; for r=3, ` + `${[1, 2, 3, 4].map((d) => `d=${d} -> ${3 ** d}`).join(", ")}`); const THRESHOLD = 20; // AY2: timeout 20 rounds = K01's 200 ms threshold const worst = Array.from({ length: 27 }, (_, n) => 1 + Math.min(CAP, BASE * 2 ** n)) .reduce((a, x) => a + x, 0); console.log(`27 retries' worst-case total wait is ${worst} rounds = ${worst / CYCLE} s, ${b(worst / THRESHOLD, 1)} ` + `times K01's ${THRESHOLD}-round read threshold`); const narrow = run({ tries: [3, 3, 3], budget: 0.1, scenario: OUTAGE }); console.log(`sensitivity: at budget 0.10 instead of 0.30, outage calls would go ${kb.calls} -> ${narrow.calls}, ` + `window rate ${b(kb.windowRate)} -> ${b(narrow.windowRate)}, dropped ${kb.dropped} -> ${narrow.dropped}`); ``` ``` model: 4000 rounds (40 s), failure at round 500; usual capacity 6 calls/round (600 req/s), event arrivals 0.9722/round; seed 20260802 -- healthy (AY4: 0.025 rare failure) -- configuration allowed calls multiplier peak calls/round successful dropped no retries 1 3832 x1.00 6 3736 96 gateway only (3) 3 3953 x1.03 6 3851 0 two layers (3x3) 9 3953 x1.03 6 3851 0 three layers (3x3x3) 27 3953 x1.03 6 3851 0 three layers + budget 0.30 27 3953 x1.03 6 3851 0 -- outage (1 s): for 100 rounds, capacity 0 -- configuration allowed calls multiplier peak calls/round successful dropped window calls/round times usual rate no retries 1 3855 x1.00 5 3775 80 0.80 x0.8 gateway only (3) 3 4071 x1.05 8 3782 94 2.89 x3.0 two layers (3x3) 9 4435 x1.15 13 3850 20 5.77 x5.9 three layers (3x3x3) 27 4523 x1.17 17 3870 0 6.24 x6.4 three layers + budget 0.30 27 4065 x1.05 10 3777 103 2.88 x3.0 -- degradation (10 s): for 1000 rounds, capacity 1 -- configuration allowed calls multiplier peak calls/round successful dropped window calls/round times usual rate no retries 1 3855 x1.00 5 3534 321 0.95 x1.0 gateway only (3) 3 4913 x1.28 8 3754 94 2.01 x2.1 two layers (3x3) 9 5296 x1.39 9 3803 0 2.39 x2.5 three layers (3x3x3) 27 5296 x1.39 9 3803 0 2.39 x2.5 three layers + budget 0.30 27 4288 x1.11 7 3643 220 1.40 x1.4 failure-free day's cost: calls 3832 -> 3953 (multiplier x1.03), reaching the dependency 95.80 -> 98.83 req/s; no difference between allowed 1 and 27, because the failure ratio is 0.025 failing day's gain (outage): dropped requests 80 -> 94 -> 20 -> 0; successful 3775 -> 3870 the storm's growth (outage): window calls/round 0.80 -> 2.89 -> 5.77 -> 6.24 = x6.4 the usual rate; arithmetic upper bound x27 under degradation the multiplier follows not allowance but failure ratio: allowed 9 and 27 give the same calls (5296 and 5296), window rate 2.39 budget 0.30 spread across the chain: outage calls 4523 -> 4065, window rate 6.24 -> 2.88, dropped 0 -> 103; degradation calls 5296 -> 4288, dropped 321 -> 0 -> 220 upper bound independent of the run: at d layers retrying r times, multiplier r^d; for r=3, d=1 -> 3, d=2 -> 9, d=3 -> 27, d=4 -> 81 27 retries' worst-case total wait is 793 rounds = 7.93 s, 39.6 times K01's 20-round read threshold sensitivity: at budget 0.10 instead of 0.30, outage calls would go 4065 -> 3926, window rate 2.88 -> 1.54, dropped 103 -> 90 ``` These numbers belong to the measurement class and depend on the seed. #### The Multiplier Sleeps on a Failure-Free Day The first table gives the pattern's failure-free-day cost, and it is small: calls climb from 3832 to 3953, the multiplier is 1.03, the rate reaching the dependency goes from 95.80 to 98.83 req/s. In exchange, dropped requests fall from 96 to 0 — on a failure-free day, retrying fully absorbs rare timeouts for a three-percent cost. What matters is the column that **never changes**. As allowed retries climb from 1 to 27, the call count stays flat. The reason is arithmetic: at a 0.025 failure ratio, the odds a request needs a third attempt are below one in a thousand, so the twenty-seven-retry allowance never gets used. **The call amplification sleeps on a failure-free day; the failure ratio is what wakes it.** A chain carrying excess retry headroom shows up in no metric under normal conditions. #### Under an Outage, the Storm Grows With Layers The second table shows the wakeup. While the dependency answers no call for one second, the in-window call rate grows with each added layer: 0.80 with no retries, 2.89 at one layer, 5.77 at two, 6.24 calls/round at three. That last figure is **6.4 times** the usual rate, and it is aimed at a dependency that is already down. The arithmetic upper bound must be kept apart from the observed multiplier. Three layers at three retries each allow 27; the observed figure is 6.4. Backoff makes the difference — retries spread out over time, so not all fit in the window. **The upper bound is the risk the design carries; the observed multiplier is the share of that risk realized at these parameters.** The bound is independent of the run: at depth four, it is 81. The table also gives the caller-side gain, and it is real: dropped requests are 80 in the no-retry run, **0** at three layers. Because backoff spreads retries past the window, every request with enough retry headroom eventually succeeds. Something also goes the wrong way between two rows. In the single-layer run, dropped requests are 94 — **more** than the no-retry run's 80. The reason sits in the peak column: after the window, the storm climbs to 8 calls per round, above the usual 6-call capacity. Three retries do not survive this second wave once the outage ends. **Insufficient retry headroom can be worse than none at all:** it grows the load without rescuing the request. #### Under Degradation, the Failure Ratio Sets the Multiplier, Not the Allowance The third table runs the same configurations under a profile where capacity drops to one instead of zero, and the ranking reverses. Two layers and three layers give **the same number**: 5296 calls, 2.39 calls/round in the window. Even as the allowance climbs from 9 to 27, nothing changes. The reason is the reverse of the second table's. Because capacity is one, a portion of calls still succeeds; a request failing nine times in a row is rare. What sets the multiplier is not the allowance but **the probability that a retry fails.** The rule: the allowance sets an upper bound, the failure mode chooses how much of it is realized. A full outage uses the allowance to the end; partial degradation does not. The multiplier still climbing to 2.5 times under degradation is not small: a dependency whose capacity has fallen to a sixth receives two and a half times the requests from the very system straining it. **This is what stretches out the degradation.** #### The Retry Budget Is Spread Across the Chain The last row gives the fix, and the fix is not trimming the retry count layer by layer; it was already shown that each layer chooses a reasonable count and the totals multiply. A **retry budget** caps the ratio of extra retries to primary calls, and that cap holds across the whole chain. With the budget at 0.30, outage window rate falls from 6.24 to 2.88 calls/round, 3.0 times the usual rate; total calls drop from 4523 to 4065. Under degradation, window rate falls from 2.39 to 1.40, total calls from 5296 to 4288. The cost is plain: dropped requests rise from 0 to 103 under outage, 0 to 220 under degradation. **The budget gives back a portion of the requests the caller rescued, so the dependency can breathe.** The sensitivity row shows how far the budget can be tightened: at 0.10, outage window rate falls to 1.54, and dropped requests **fall** from 103 to 90 — here, a tighter budget improves both sides at once, because a smaller storm also shrinks the second wave after the window. The budget's right value depends less on how many requests the failure rescues than on how many the storm crushes. #### Summary - Each layer of the chain retries on its own, and allowances multiply: three layers at three retries give an upper bound of 27; at d layers and r retries the bound is r^d, 81 at d=4. - The multiplier sleeps on a failure-free day: as the allowance climbs from 1 to 27 the call count does not change, the multiplier is 1.03, and dropped requests fall from 96 to 0. Excess retry headroom shows up in no metric under normal conditions. - Under a full outage, the storm grows with each layer: window call rate goes 0.80 → 2.89 → 5.77 → 6.24 calls/round, 6.4 times the usual rate. The observed multiplier stays below the arithmetic bound (27), since backoff spreads retries over time. - Insufficient retry headroom can be worse than none: dropped requests are 94 in the single-layer run against 80 with no retries; after the window, the storm pushes the peak above usual capacity, to 8 calls/round. - Under partial degradation, the failure ratio sets the multiplier, not the allowance: allowances of 9 and 27 give the same result (5296 calls, 2.39 calls/round), since no request exhausts nine retries. - A retry budget is placed across the whole chain: at 0.30, the outage storm falls from 6.24 to 2.88 calls/round, degradation from 2.39 to 1.40; the cost is 103 dropped requests under outage, 220 under degradation. #### Next Step Every measurement here held one number fixed: the wait between retries. But wait time and retry allowance cannot be chosen independently, since their product decides **how long the caller waits**. The last row gives that: twenty-seven retries, with backoff capped at thirty-two rounds, keep a request alive for a worst case of 793 rounds, 7.93 seconds — 39.6 times K01's twenty-round read threshold. So the retry policy's real limit is not the retry count but the time the caller has, and that time was never budgeted here. The next lesson takes it up and spreads it across the chain: how much share the end-to-end threshold leaves each step, how the remaining share splits among the calls beneath it, and how canceled work is counted when the budget runs out on a slowing dependency. ### Timeout Budgets Source: https://academia.sh/en/courses/resilience-patterns/timeout-budgets How the budget runs out at a slowing dependency: splitting the arm's remaining share across consecutive sub-calls with a greedy, equal, or weighted policy, the greedy split making the healthy step appear at fault, counting the caller's held slot separately from the dependency's abandoned work, and the tight split writing a gain only together with a cancellation signal. The retry policy is settled: try count, wait between tries, jitter width. One number stays open and decides more than the policy itself — how long the caller waits before giving up. Without it, the backoff grows toward open space, not a limit. The Application Layer and Service Interaction course established this duration as a **budget**: the end-to-end threshold splits across the chain's steps, the remaining time passes with the call, and the retry count multiplies it. That computation is an input here, not retold. This lesson goes one step further: when a chain step **slows down**, where does the budget run out, how does the share split across sub-calls, and how much work does the dependency still do after the caller gives up. #### Failure Mode The chain comes from this course's shared structure. The tracking query enters at the gateway and calls two arms in parallel. The delivery-operations arm reads a single record from its own store; the billing arm reads **two records** — first the tariff, then the shipment's size — and computes the fee from both. The Application Layer course counted four step calls per external request; once the billing arm's store step splits into two consecutive reads, the count climbs to five. **AY10 — the tariff store slows down.** It does not go down, answer incorrectly, or suffer a network partition; only its response time climbs to tens of times its usual six-millisecond median. Duration **12 minutes**, frequency **once a month**. Rationale: the slowdown starts with a maintenance window or a query-plan change and lasts until noticed and rolled back; twelve minutes sits a little above K01's ten-minute recovery assumption. Its sensitivity is below, not added to K01's table. Slowdown is harder than being down. A down dependency rejects the connection immediately, and the caller learns this in under a millisecond. One that is slowing down rejects nothing, it just makes the caller wait — and with no limit set, that wait is paid for with the caller's resources. The budget is the only mechanism deciding how much of it is accepted. #### Splitting the Remaining Share The arm's share is fixed: K01's 200-millisecond read threshold minus the Application Layer course's U3 assumption of a 20-millisecond gateway share leaves 180 milliseconds, which the billing arm must distribute across two consecutive reads. Three policies exist. The **greedy** split gives the entire remainder to each call: the first read can wait up to 180 milliseconds, the second gets whatever is left. The **equal** split halves the share. The **weighted** split allocates it by the typical durations' ratio — tariff 6 milliseconds, size 12, so one to two. All three look nearly identical on a failure-free day, since both reads stay well under their share. The difference appears the moment one read exceeds it. ```js // budget/chain.mjs — MODEL of the tracking query chain. Logical clock: step durations are // read from a cycle, real time is not measured; no service, network, or store is set up. export const K01 = { threshold: 200, readPeak: 416.67, edgePeak: 513.89, responseBytes: 480 }; export const GATEWAY_SHARE = 20; // K03 service-design/05, assumption U3 export const TYPICAL = { delivery: 8, tariff: 6, size: 12 }; // model parameter (ms) const D = (body, tail) => [...body, ...body, tail]; // a long tail once in sixteen export const REGIME = { // sub-call duration (ms), sixteen-step cycle "failure-free": { delivery: D([8, 9, 8, 10, 8, 9, 8, 10], 40), tariff: D([6, 7, 6, 8, 6, 7, 6, 8], 80), size: D([12, 13, 12, 14, 12, 13, 12, 14], 100), }, "tariff slowdown": { // AY10: only the tariff store slows down delivery: D([8, 9, 8, 10, 8, 9, 8, 10], 40), tariff: D([150, 400, 220, 400, 180, 300, 260, 400], 400), size: D([12, 13, 12, 14, 12, 13, 12, 14], 100), }, }; // Splitting the remaining share for the arm across two consecutive sub-calls. export function shares(policy, remaining) { if (policy === "greedy") return [remaining, remaining]; if (policy === "equal") return [remaining / 2, remaining / 2]; const t = TYPICAL.tariff + TYPICAL.size; // weighted: ratio of the typical durations return [(remaining * TYPICAL.tariff) / t, (remaining * TYPICAL.size) / t]; } export function run({ regime, policy, cancel, n = 48 }) { const R = K01.threshold - GATEWAY_SHARE; // remaining share for the arm const s = { answered: 0, dropped: 0, wastedWork: 0, totalDuration: 0, failed: { delivery: 0, tariff: 0, size: 0 }, endToEnd: [] }; const cursor = { delivery: 0, tariff: 0, size: 0 }; const duration = (name) => REGIME[regime][name][cursor[name]++ % REGIME[regime][name].length]; const call = (name, limit) => { // one sub-call, timeout `limit` if (limit <= 0) return { ok: false, elapsed: 0 }; const d = duration(name); if (d <= limit) return { ok: true, elapsed: d }; if (cancel === false) s.wastedWork += d - limit; // caller gave up, dependency keeps working return { ok: false, elapsed: limit }; }; for (let i = 0; i < n; i += 1) { const a = call("delivery", R); // delivery-ops arm: one store call const p = shares(policy, R); // billing arm: two consecutive records const b1 = call("tariff", Math.min(p[0], R)); const b2 = call("size", Math.min(p[1], R - b1.elapsed)); for (const [name, r] of [["delivery", a], ["tariff", b1], ["size", b2]]) { if (r.ok === false) s.failed[name] += 1; } const elapsed = GATEWAY_SHARE + Math.max(a.elapsed, b1.elapsed + b2.elapsed); s.endToEnd.push(elapsed); s.totalDuration += elapsed; if (a.ok && b1.ok && b2.ok && elapsed <= K01.threshold) s.answered += 1; else s.dropped += 1; } const sorted = [...s.endToEnd].sort((x, y) => x - y); return { ...s, median: sorted[n >> 1], worst: sorted[n - 1], average: s.totalDuration / n, n }; } ``` Both reads are attempted on every request; the billing arm decides only after adding both outcomes. This carries the measurement: if the first read eats the budget, the second read experiences this as **its own failure**. ```js // budget/run.mjs — comparing the three split policies over two days import { K01, GATEWAY_SHARE, TYPICAL, shares, run } from "./chain.mjs"; const R = K01.threshold - GATEWAY_SHARE; const POLICY = ["greedy", "equal", "weighted"]; const CALLS = 5; // sub-calls per external request: 2 arms + 1 + 2 records const DEADLINE_BYTES = 8; // field carrying the remaining time const slot = (ms) => (K01.readPeak * ms) / 1000; // L = lambda x W console.log(`end-to-end threshold ${K01.threshold} ms (K01), gateway share ${GATEWAY_SHARE} ms (K03/U3), remaining for the arm ${R} ms`); console.log(`typical durations: delivery ${TYPICAL.delivery}, tariff ${TYPICAL.tariff}, size ${TYPICAL.size} ms`); console.log(`${"policy".padEnd(11)}${"tariff share(ms)".padStart(17)}${"size share(ms)".padStart(15)}`); for (const p of POLICY) { const [a, b] = shares(p, R); console.log(`${p.padEnd(11)}${a.toFixed(1).padStart(17)}${b.toFixed(1).padStart(15)}`); } for (const regime of ["failure-free", "tariff slowdown"]) { console.log(`\n-- ${regime} (48 external requests, no cancellation signal) --`); console.log(`${"policy".padEnd(11)}${"answered".padStart(9)}${"dropped".padStart(8)}${"blame:tariff".padStart(13)}` + `${"blame:size".padStart(11)}${"median".padStart(8)}${"worst".padStart(8)}${"avg.".padStart(7)}` + `${"held slots".padStart(13)}${"wasted work(ms)".padStart(16)}`); for (const p of POLICY) { const r = run({ regime, policy: p, cancel: false }); console.log(`${p.padEnd(11)}${String(r.answered).padStart(9)}${String(r.dropped).padStart(8)}` + `${String(r.failed.tariff).padStart(13)}${String(r.failed.size).padStart(11)}` + `${String(r.median).padStart(8)}${String(r.worst).padStart(8)}${r.average.toFixed(1).padStart(7)}` + `${slot(r.average).toFixed(1).padStart(13)}${String(r.wastedWork).padStart(16)}`); } } console.log(`\n-- tariff slowdown: slots the system keeps busy (peak read ${K01.readPeak}/s) --`); console.log(`${"policy".padEnd(11)}${"caller".padStart(9)}${"abandoned".padStart(12)}` + `${"total(no cancel)".padStart(17)}${"total(cancel)".padStart(16)}`); for (const p of POLICY) { const r = run({ regime: "tariff slowdown", policy: p, cancel: false }); const c = slot(r.average), t = slot(r.wastedWork / r.n); console.log(`${p.padEnd(11)}${c.toFixed(1).padStart(9)}${t.toFixed(1).padStart(12)}` + `${(c + t).toFixed(1).padStart(17)}${c.toFixed(1).padStart(16)}`); } const bytes = K01.edgePeak * CALLS * DEADLINE_BYTES; const a = run({ regime: "failure-free", policy: "greedy", cancel: false }); console.log(`\n-- failure-free day's cost (computed value) --`); console.log(`deadline field ${DEADLINE_BYTES} B x ${CALLS} sub-calls = ${CALLS * DEADLINE_BYTES} B/request,` + ` ${((CALLS * DEADLINE_BYTES) / K01.responseBytes * 100).toFixed(1)}% of the tracking response (${K01.responseBytes} B)`); console.log(`at ${K01.edgePeak} req/s peak edge -> ${bytes.toFixed(2)} B/s = ${(bytes / 1024).toFixed(2)} KiB/s`); for (const p of ["equal", "weighted"]) { const r = run({ regime: "failure-free", policy: p, cancel: false }); const diff = r.dropped - a.dropped; console.log(`${p.padEnd(11)}: ${diff}/${r.n} more dropped requests than greedy` + ` = ${((diff / r.n) * 100).toFixed(2)}% -> ${((K01.readPeak * diff) / r.n).toFixed(2)} req/s at peak read`); } console.log(`\n-- AY10's sensitivity: how long the tariff store stays slow --`); console.log(`${"duration(min)".padStart(14)}${"a month".padStart(8)}${"minutes/mo".padStart(11)}${"of failure share(28.2 min)".padStart(27)}`); for (const [min, times] of [[12, 1], [12, 2], [24, 1]]) { console.log(`${String(min).padStart(14)}${String(times).padStart(8)}${String(min * times).padStart(11)}` + `${`${(((min * times) / 28.2) * 100).toFixed(1)}%`.padStart(27)}`); } ``` ``` end-to-end threshold 200 ms (K01), gateway share 20 ms (K03/U3), remaining for the arm 180 ms typical durations: delivery 8, tariff 6, size 12 ms policy tariff share(ms) size share(ms) greedy 180.0 180.0 equal 90.0 90.0 weighted 60.0 120.0 -- failure-free (48 external requests, no cancellation signal) -- policy answered dropped blame:tariff blame:size median worst avg. held slots wasted work(ms) greedy 48 0 0 0 40 200 46.2 19.2 0 equal 46 2 0 2 40 190 45.8 19.1 20 weighted 46 2 2 0 40 180 45.3 18.9 40 -- tariff slowdown (48 external requests, no cancellation signal) -- policy answered dropped blame:tariff blame:size median worst avg. held slots wasted work(ms) greedy 6 42 36 42 200 200 197.8 82.4 5540 equal 0 48 48 2 123 200 126.0 52.5 9700 weighted 0 48 48 0 93 180 96.4 40.2 11120 -- tariff slowdown: slots the system keeps busy (peak read 416.67/s) -- policy caller abandoned total(no cancel) total(cancel) greedy 82.4 48.1 130.5 82.4 equal 52.5 84.2 136.7 52.5 weighted 40.2 96.5 136.7 40.2 -- failure-free day's cost (computed value) -- deadline field 8 B x 5 sub-calls = 40 B/request, 8.3% of the tracking response (480 B) at 513.89 req/s peak edge -> 20555.60 B/s = 20.07 KiB/s equal : 2/48 more dropped requests than greedy = 4.17% -> 17.36 req/s at peak read weighted : 2/48 more dropped requests than greedy = 4.17% -> 17.36 req/s at peak read -- AY10's sensitivity: how long the tariff store stays slow -- duration(min) a month minutes/mo of failure share(28.2 min) 12 1 12 42.6% 12 2 24 85.1% 24 1 24 85.1% ``` All these numbers belong to the **computed value** class: they are counted over a deterministic sequence of durations. The time unit is a logical clock, not a measured latency; the run gives the same output independent of the machine. #### Failure-Free Day's Cost The budget carries two costs even on a failure-free day. The first is the field itself: passing the remaining time with the call adds an eight-byte field to every sub-call, 40 bytes per external request across five — 8.3 percent of K01's 480-byte tracking response, and 20.07 KiB/s of internal traffic at the peak edge rate. Small but not zero, and it grows linearly as the chain deepens. The second is costlier: **a tight split cuts off healthy requests.** In the failure-free regime greedy answers forty-eight of forty-eight requests; equal and weighted answer forty-six, dropping two each. These are false positives on a day with no failure at all — the model's long tail, arriving once in sixteen, exceeds the step's share: 4.17 percent, 17.36 requests/s at peak read. Which step gets cut off depends on the policy: under equal it is size (share 90, tail 100); under weighted, tariff (share 60, tail 80). Choosing the share by the step's tail lowers this cost but does not zero it. Splitting the budget is a decision to cut a tail, and the cut tail is always made of real requests. #### What Changes on a Failing Day The second table runs under AY10 and the three policies produce three distinct behaviors. **Greedy rescues six requests.** The tariff returns in 150 or 180 milliseconds on some rounds; with a 180-millisecond share these succeed, leaving enough for the size read. Under equal and weighted, the same reads are cut off at 90 and 60 milliseconds, and forty-eight of forty-eight requests drop — the cost of a tight split continues on a failing day too. **The trade-off shows up in end-to-end duration.** Under greedy the median is 200 milliseconds: every dropped request eats the whole budget. Under equal it is 123, under weighted 93 — the caller learns the same failure in under half the time. Its resource counterpart is `held slots`: by queueing theory's $L = \lambda W$ relation, at K01's peak read rate of 416.67 requests/s, greedy keeps 82.4 concurrent caller slots busy, weighted 40.2. The relation is not this course's finding; it comes from the background jobs topic and is used here as a converter. **The blame columns hide the failure mode.** Under greedy, the tariff fails thirty-six times, the size **forty-two**. The size's regime never changed; all forty-two failures come from the tariff eating the budget. Under weighted, the size fails zero times. The difference is not a counting difference but a diagnostic one: greedy hides the slowing step behind the healthy one, and the name showing up most in the timeout log is not where the failure happened. This is **spread** through the budget. The steps look isolated — separate stores, separate records — but they share a single resource, time. Fixing the share is the name for dividing up that shared resource. #### Abandoned Work The last column flips the table read so far. When the caller gives up, the tariff store does not stop; it finishes the read it started. This is work nobody waits for, and the model counts it as `wasted work`: 5540 milliseconds under greedy, 11120 under weighted. A tight split **doubles** abandoned work — giving up earlier leaves a longer remainder ahead of the dependency. The third table settles it by adding both sides. With no cancellation signal, total busy slots are 130.5 under greedy, 136.7 under equal and weighted — nearly the same: splitting only moves the load from caller to dependency, nothing else. A tight split's gain shows up only with a **cancellation signal** — the caller tells the dependency it has given up, and the read stops. Then abandoned work zeroes out and total slots become 82.4 against 40.2. This is the deciding sentence: **deadline propagation is not complete without a cancellation signal.** Passing the remaining time with the call says when the caller will give up; the signal says that it has. Without the first, the budget cannot be split; without the second, splitting only relocates cost. The last table is AY10's sensitivity. A single twelve-minute slowdown spends 42.6 percent of K01's monthly 28.2-minute failure share. Twice a month, or once for twenty-four minutes, the share climbs to 85.1 percent, leaving 4.2 minutes for the rest of the month. Splitting the budget does not bring these minutes back — requests still drop — but it fixes how many resources a dropped request occupies and confirms which step the failure shows up in. #### Summary - The arm's remaining share (200 ms threshold minus 20 ms gateway share = 180 ms) splits three ways: greedy 180/180, equal 90/90, weighted 60/120. - The failure-free day's cost has two items: a deadline field adding 40 bytes per external request (8.3 percent of the response, 20.07 KiB/s at peak edge), and a tight split cutting off valid requests — 2 in 48, 4.17 percent, 17.36 requests/s at peak read. - Under AY10, greedy rescues six requests but keeps the median at 200 ms; weighted rescues none but brings the median to 93 ms, dropping held slots from 82.4 to 40.2. - Under greedy, the healthy step (size) appears to fail forty-two times, under weighted zero times; the split sets not just the resource but the **diagnosis** of the failure. - A tight split raises abandoned work from 5540 to 11120 ms; with no cancellation signal, total busy slots run 130.5 against 136.7 — no gain. With one, the total becomes 82.4 against 40.2 — a gain only with cancellation. - A single twelve-minute slowdown spends 42.6 percent of K01's monthly 28.2-minute failure share; twice a month, 85.1 percent. #### Next Step The budget decided how long a single request would wait and distributed that decision across the chain. One assumption stayed silent: the caller is **able** to make the call for every request. Facing a slowing dependency, the budget cuts the request off, reclaims the resource, and shows the failure at the right step — but does not stop the next request from arriving. Requests keep coming at the same rate, each spending the same budget on the same slow step. The problem's other face is on the write path: status events from the carrier flow in at a fixed rate, and if the consumer feeding the projection falls behind, work piles up somewhere the budget never touches. The next lesson takes up that pile and rounds the question around: how does the system **tell** the party feeding it work that it can no longer keep up, how far does that notice spread, and where does work collect if it is never made. ### Backpressure Source: https://academia.sh/en/courses/resilience-patterns/backpressure The overload signal propagating backward from the service boundary: measuring propagation depth across four settings from zero to the limit, partial propagation only moving the loss one link back, the signal reaching the limit turning an accepted-then-lost write into a never-accepted write, and an unlimited buffer turning loss into staleness. The previous lesson set the budget to a single request's duration and taught how to reclaim its resource on time. One thing the budget never touches: the **rate at which requests arrive**. Facing a slowing dependency, the budget cuts each request off, but the next one still arrives and spends the same budget on the same slow step. The budget says how long a request waits; it says nothing to whoever feeds the system work. This lesson asks that missing notice on the write path. Status events from the carrier flow in at a fixed rate; if the consumer reading the event log and feeding the read model falls behind, work piles up somewhere. The Caching, Queues and Asynchronous Processing course established the **single-queue** case — a bounded queue as a latency-budget decision, overflow policies, blocking passing the slowdown to the producer, and an unaware layer in front emptying out the limit — not retold here. The question here is the chain: how many **links back** the signal is carried, where work collects where it is not, and where propagation must stop. #### Failure Mode and Chain The write path has four parts: the carrier's notification endpoint sends the event, the **entry** service accepts it, the **log** writer persists it, the **projection** consumer updates the read model. The source sits outside the system, beyond its control — the carrier belongs to another organization. **AY11 — the projection consumer falls behind.** It does not go down or write incorrectly; its speed drops from 120 events per round to 60. Duration **20 minutes**, frequency **three times a month**. Rationale: consumer slowdown usually comes from a projection schema change or a rebuild job overlap, lasting until that job finishes. Its sensitivity is below, not added to K01's table. AY11 is not an outage; the tracking query keeps answering, only the projection it reads falls behind, so it is not written to K01's monthly 43.2-minute downtime budget. A lost write goes into a different measure, its threshold set in this course's recovery topic. A round is an abstract step, corresponding in the model to a one-second window, and this cycle coefficient is an assumption. The source rate is K01's peak write count: 97.22 events per round. Arrival is not flat but oscillating — a multiplier cycle averaging 1.00 represents fluctuation within the peak hour. #### Propagation Depth The signal is a single piece of information carried backward from the consumer: "my successor is full, I cannot take your work." Each link can heed this or ignore it. In the model this is a parameter — **depth**, how many links back it travels from the consumer. At depth 0 no link heeds it; at depth 3 it reaches the service boundary and the carrier's event **is not accepted**. ```js // backpressure/chain.mjs — MODEL of the write path. A round is an abstract step; the cycle // coefficient is an assumption (one round = one second). No real queue, log, or consumer is set up. export const K01 = { writePeak: 97.22 }; // K01: carrier state event, peak write/s // Links are ordered from source to tip. `rate` is the number of events processed in one round. export const LINK = [ { name: "entry", capacity: 200, rate: 250 }, // event admission endpoint { name: "log", capacity: 150, rate: 150 }, // event log writer { name: "projection", capacity: 100, rate: 120 }, // consumer feeding the read model ]; // Oscillation of arrivals within the peak hour: a multiplier cycle averaging 1.00. export const OSCILLATION = [0.6, 0.6, 0.6, 2.2]; // Link i pauses if it heeds its successor's fullness; if not, it drops the event. // depth: how many links back from the consumer the signal is carried. 3 = to the limit. const heeds = (i, depth) => depth >= LINK.length - 1 - i; export function run({ roundCount, depth, consumptionRate, oscillate = true, unlimited = false }) { const rate = LINK.map((h, i) => (i === LINK.length - 1 ? consumptionRate : h.rate)); const capacity = LINK.map((h) => (unlimited ? Infinity : h.capacity)); const buffer = LINK.map(() => []); const s = { produced: 0, completed: 0, rejected: 0, finishedAge: 0, sourceStopped: 0, lost: LINK.map(() => 0), paused: LINK.map(() => 0), }; let carry = 0; for (let t = 1; t <= roundCount; t += 1) { for (let i = LINK.length - 1; i >= 0; i -= 1) { // drain from the tip toward the source let n = Math.min(rate[i], buffer[i].length); while (n > 0) { if (i === LINK.length - 1) { // last link: work completes s.finishedAge = Math.max(s.finishedAge, t - buffer[i].shift()); s.completed += 1; n -= 1; continue; } if (buffer[i + 1].length < capacity[i + 1]) { buffer[i + 1].push(buffer[i].shift()); n -= 1; continue; } if (heeds(i, depth)) { s.paused[i] += 1; break; } // successor full: pause s.lost[i + 1] += 1; buffer[i].shift(); n -= 1; // no signal: lose it } } const multiplier = oscillate ? OSCILLATION[(t - 1) % OSCILLATION.length] : 1; carry += K01.writePeak * multiplier; while (carry >= 1) { // source: carrier notification endpoint carry -= 1; s.produced += 1; if (buffer[0].length < capacity[0]) { buffer[0].push(t); continue; } if (depth >= LINK.length) { // rejected at the limit: event not accepted s.rejected += 1; s.sourceStopped += 1; } else s.lost[0] += 1; // accepted, then lost } } s.backlog = buffer.reduce((a, b) => a + b.length, 0); const pending = buffer.flat(); s.pendingAge = pending.length === 0 ? 0 : roundCount - Math.min(...pending); s.totalLost = s.lost.reduce((a, b) => a + b, 0); return s; } ``` Two counters are kept separate, and the lesson's whole distinction is here. A **lost** event is one the system accepted, then dropped; the carrier counts it sent and never resends it. A **rejected** event is never accepted at the boundary; the carrier knows this and resends it under its own policy. ```js // backpressure/run.mjs — signal propagation depth: five settings across two days import { K01, LINK, run } from "./chain.mjs"; const ROUNDS = 1200; // AY11: 20 minutes, one round = one second const CONSUMPTION_NORMAL = 120, CONSUMPTION_SLOW = 60; // projection consumer's rate per round const SETTING = [ ["unlimited buffer", { depth: 0, unlimited: true }], ["no signal", { depth: 0 }], ["1 link", { depth: 1 }], ["2 links", { depth: 2 }], ["to the limit", { depth: 3 }], ]; console.log(`${ROUNDS} rounds, source K01 peak write ${K01.writePeak}/round (oscillating, average multiplier 1.00)`); console.log(`links: ${LINK.map((h, i) => `${h.name}(cap ${h.capacity}, rate ${i === 2 ? "consumption" : h.rate})`).join(" -> ")}\n`); for (const [regime, consumptionRate] of [["failure-free: consumption 120/round", CONSUMPTION_NORMAL], ["AY11: consumption 60/round", CONSUMPTION_SLOW]]) { console.log(`-- ${regime} --`); console.log(`${"setting".padEnd(16)}${"completed".padStart(11)}${"rejected".padStart(11)}${"lost".padStart(9)}` + `${"loss location".padStart(20)}${"backlog".padStart(8)}${"pending age".padStart(13)}${"finished age".padStart(13)}`); for (const [label, y] of SETTING) { const r = run({ roundCount: ROUNDS, consumptionRate, ...y }); console.log(`${label.padEnd(16)}${String(r.completed).padStart(11)}${String(r.rejected).padStart(11)}` + `${String(r.totalLost).padStart(9)}${r.lost.join("/").padStart(20)}${String(r.backlog).padStart(8)}` + `${String(r.pendingAge).padStart(13)}${String(r.finishedAge).padStart(13)}`); } console.log(); } const unlimited = run({ roundCount: ROUNDS, depth: 0, unlimited: true, consumptionRate: CONSUMPTION_NORMAL }); const signaled = run({ roundCount: ROUNDS, depth: 3, consumptionRate: CONSUMPTION_NORMAL }); const flat = run({ roundCount: ROUNDS, depth: 3, consumptionRate: CONSUMPTION_NORMAL, oscillate: false }); console.log(`-- failure-free day's cost (against the unlimited buffer) --`); console.log(`produced ${signaled.produced}, valid events rejected at the limit ${signaled.rejected}` + ` = ${((signaled.rejected / signaled.produced) * 100).toFixed(2)}%`); console.log(`same setting with oscillation off: ${flat.rejected} rejections -> the source of rejection is not average load but oscillation`); console.log(`unlimited buffer's pending age ${unlimited.pendingAge} rounds, signal to the limit ${signaled.pendingAge} rounds`); console.log(`paused link-rounds: ${signaled.paused.join("/")}; events where the source was stopped ${signaled.sourceStopped}`); console.log(`signal check cost: ${LINK.length} links x ${K01.writePeak} events/s = ${(LINK.length * K01.writePeak).toFixed(2)} checks/s`); console.log(`\n-- AY11's sensitivity: consumer rate (1200 rounds) --`); console.log(`${"consumption/round".padStart(19)}${"gap/round".padStart(10)}${"unlimited backlog".padStart(19)}` + `${"no-signal lost".padStart(16)}${"rejected at limit".padStart(19)}${"lost/rejected".padStart(14)}`); for (const h of [90, 60, 30]) { const a = run({ roundCount: ROUNDS, depth: 0, unlimited: true, consumptionRate: h }); const y = run({ roundCount: ROUNDS, depth: 0, consumptionRate: h }); const t = run({ roundCount: ROUNDS, depth: 3, consumptionRate: h }); console.log(`${String(h).padStart(19)}${(K01.writePeak - h).toFixed(2).padStart(10)}${String(a.backlog).padStart(19)}` + `${String(y.totalLost).padStart(16)}${String(t.rejected).padStart(19)}` + `${(y.totalLost / t.rejected).toFixed(3).padStart(14)}`); } ``` ``` 1200 rounds, source K01 peak write 97.22/round (oscillating, average multiplier 1.00) links: entry(cap 200, rate 250) -> log(cap 150, rate 150) -> projection(cap 100, rate consumption) -- failure-free: consumption 120/round -- setting completed rejected lost loss location backlog pending age finished age unlimited buffer 116301 0 0 0/0/0 362 3 4 no signal 82282 0 34064 4164/14950/14950 317 2 3 1 link 97224 0 19114 4164/14950/0 325 3 4 2 links 112124 0 4164 4164/0/0 375 3 4 to the limit 112124 4164 0 0/0/0 375 3 4 -- AY11: consumption 60/round -- setting completed rejected lost loss location backlog pending age finished age unlimited buffer 71814 0 0 0/0/0 44849 460 460 no signal 71814 0 44494 4164/14950/25380 355 3 4 1 link 71814 0 44404 4164/40240/0 445 5 6 2 links 71814 0 44399 44399/0/0 450 7 8 to the limit 71814 44399 0 0/0/0 450 7 8 -- failure-free day's cost (against the unlimited buffer) -- produced 116663, valid events rejected at the limit 4164 = 3.57% same setting with oscillation off: 0 rejections -> the source of rejection is not average load but oscillation unlimited buffer's pending age 3 rounds, signal to the limit 3 rounds paused link-rounds: 598/897/0; events where the source was stopped 4164 signal check cost: 3 links x 97.22 events/s = 291.66 checks/s -- AY11's sensitivity: consumer rate (1200 rounds) -- consumption/round gap/round unlimited backlog no-signal lost rejected at limit lost/rejected 90 7.22 9029 34064 8579 3.971 60 37.22 44849 44494 44399 1.002 30 67.22 80753 80394 80303 1.001 ``` All these numbers belong to the **computed value** class: they are counted over a deterministic arrival sequence. #### Partial Propagation Moves Loss, Does Not Remove It The second table is the lesson's main finding. Under AY11, the lost-event count is 44,494 at depth 0, 44,404 at depth 1, 44,399 at depth 2 — nearly identical. Only `loss location` changes: first the projection's gate (25,380), then the log's gate (40,240), then the entry's gate (44,399). Partial backpressure rescues no event; it only relocates the loss. Two things are paid for this: the backlog climbs from 355 to 450, and the pending event's age rises from 3 rounds to 7 — holding work back is itself a cost. In the fourth row the number zeroes out: once the signal reaches the service boundary, 44,399 events are not **lost**, they are **rejected**. The completed count is unchanged (71,814) — whatever the consumer's capacity is; what changes is what happens to the events that do not fit. A lost event counts as sent for the carrier and never comes again; a rejected one was never sent and can be resent. **Backpressure rescues no event until it reaches the service boundary.** The background jobs topic said that unless applied across the whole chain, it counts as applied nowhere in it; this table adds which link the boundary is — the only place the signal can stop is where the system talks to the outside. #### Where Propagation Must Stop The signal cannot reach the carrier — another organization's system this one cannot slow down, only decline. Propagation therefore ends not at the chain's fourth link but at the third one's outer gate, and there it changes form: **inside it is a hold, at the boundary a rejection.** The first table shows the first cost. On a failure-free day — consumer faster than producer — the signal carried to the limit rejects 4164 valid events: 3.57 percent of production. With oscillation off, the same setting rejects nothing — so rejection's source is not average load but the momentary fluctuation of arrivals. When the buffer cannot absorb that fluctuation, backpressure rounds an absorbable peak into a rejection. The same row has a second cost: 598 and 897 paused link-rounds, where a link had work but stopped because its successor was full — unused capacity. The unlimited-buffer row pays neither cost: on a failure-free day it completes 116,301 events and rejects none. Its cost shows up under AY11 instead: no event is lost, but 44,849 pile up, and the oldest pending one is **460 rounds** old — about seven and a half minutes. The projection is that far behind, and the tracking query shows a state that old. An unlimited buffer does not eliminate loss; it rounds it into **staleness**. The three options write to three places: memory and staleness (unlimited buffer), loss (signal-free limited buffer), or the source (signal to the limit). The choice depends on the data — a lost status event permanently shortens the tracking chain by one entry, so the second option is unacceptable here. #### As the Gap Grows, the Gain Shrinks The last table is AY11's sensitivity, drawing backpressure's limit. At a consumer rate of 90 the gap is only 7.22 events per round; the signal-free arrangement loses 34,064 events while the signal to the limit rejects only 8579 — a ratio of 3.971. Here backpressure **rescues** three-quarters of the work, because the work piling up between peaks gets drained away. At a rate of 60 the ratio falls to 1.002; at 30, to 1.001. As the gap grows, backpressure rescues no work at all, only turning loss into rejection. This still has value — a rejected event can be resent — but the two gains differ and must be stated explicitly. At small gaps backpressure is an **absorption** mechanism; at large gaps a **labeling** one. #### Summary - The backpressure signal carries a propagation depth: how many links back from the consumer heed their successor's fullness. Depth 3 means the signal reaches the service boundary. - Partial propagation rescues no event: under AY11 the lost-event count at depths 0, 1, and 2 is 44,494, 44,404, and 44,399 — the same number. Only the loss location changes, with the backlog (355 → 450) and pending age (3 → 7 rounds). - Once the signal reaches the limit, 44,399 events stop being lost and become rejected; a lost event was sent for the carrier, a rejected one was not and can be resent. - The failure-free cost is the peaks oscillation absorbs: 4164 valid events rejected (3.57 percent of production), zeroing out with oscillation off; plus 598 and 897 paused link-rounds and 291.66 signal checks per second. - An unlimited buffer does not eliminate loss, it turns it into staleness: under AY11, 44,849 events pile up and the oldest pending event is 460 rounds old. - The gain's size depends on the gap: at a 7.22-event gap, backpressure cuts loss from 34,064 to 8579 (ratio 3.971); at 37.22 events the ratio is 1.002 — labeling, not rescuing. #### Next Step The signal reached the limit and turned into a rejection there. That was unavoidable, but the rejection **itself** was not designed here: when 4164 valid events were rejected, which ones, on what basis, and what was told to the other side all came down to one blunt rule — buffer full, reject. On the read side the question is sharper. At the edge there are 513.89 requests per second, and three flows share one gate: the tracking query, the carrier's status event, the end-of-day billing request. When capacity falls short, are all three throttled together, or is one kept and the others cut. Two questions tangle here — cutting what exceeds a known rate is not the same as rejecting by current capacity. The next lesson separates the two and counts which request is kept at K01's 513.89 requests/s peak. ### Throttling and Load Shedding Source: https://academia.sh/en/courses/resilience-patterns/throttling-and-load-shedding Two separate decisions in controlled service denial: throttling cutting off what exceeds a known rate at the source, load shedding rejecting by shedding-class order according to instantaneous capacity, throttling being unable to see capacity loss while shedding lets idle capacity be used, and counting how much of K01's 513.89 requests/s peak the two mechanisms protect together. The previous lesson carried the backpressure signal to the service boundary and showed it turning into a rejection there. The rejection **itself** was not designed: the rule "reject if the buffer is full" did not choose which request would go. At the edge there are 513.89 requests per second, and three flows pass through the same gate — the tracking query, the carrier's status event, the end-of-day billing request. When capacity falls short, are all three throttled the same. Two decisions are tangled in that question; this lesson separates them. #### Two Separate Criteria **Throttling** cuts off what exceeds a known rate. Its criterion is the flow itself: a ceiling derived from the usual rate is set per flow, and a request exceeding it is rejected. The decision never looks at the system's current state — even with capacity wide open, a request over the ceiling does not get through. **Load shedding** rejects according to instantaneous capacity. Its criterion is the system's current fullness: while capacity suffices, nothing is rejected; as saturation nears, requests are rejected by a shedding-class order — the same request accepted when idle, rejected when full. Admission control, setting the threshold on concurrency, and load shedding preventing collapse were established in the Caching, Queues and Asynchronous Processing course and are not retold here. The question here is what the two do **together**. Two naming collisions need separating up front. The Traffic Layer course's **gateway offloading** shares the words but is separate: it moves work like authentication or TLS termination out of the application to the gateway — the work still happens, only its location changes. The load shedding here **never accepts** the request at all. Similarly, `throttling` in the Frontend Quality course means limiting a connection's bandwidth; here it is the same idea applied to request rate. #### Gate The model is a single admission gate at the edge layer; three flows share the same capacity, and work is counted in **units**. **AY12 — capacity and unit costs.** The edge layer processes 700 units per round. A tracking query and a status event cost one unit each; an end-of-day billing request costs 60. Rationale: the batch request is rare but runs a read at K01's 833.33 records/s, two orders of magnitude above one read query. Usual demand at these values is 603.89 units per round — 86.3 percent of capacity. **AY13 — two failure scenarios.** **AY13a:** the end-of-day billing job moves into the peak hour; its rate climbs from 1.5 per round to 4.5, duration 30 minutes, twice a month. Rationale: K01's four-hour window is fixed; a job running long or starting late enters the peak hour. **AY13b:** an edge node goes down; capacity drops by a quarter (700 → 525), duration 8 minutes, twice a month. Rationale: losing a node and its replacement arriving stays under K01's ten-minute recovery assumption. Their sensitivities are below, not added to K01's table. The shedding-class order follows what each flow costs to lose: losing a status event permanently shortens the tracking chain (class 1), rejecting a tracking query makes the user wait but it can be retried (class 2), the end-of-day job can be deferred (class 3). Shedding starts with the lowest-priority class. ```js // gate/edge.mjs — MODEL of the admission gate at the edge layer. A round is an abstract step // (one round = one second; the cycle coefficient is an assumption). No real node or server is set up. export const K01 = { tracking: 416.67, event: 97.22, edgePeak: 513.89 }; // AY12: unit costs and edge capacity. One unit = one tracking query's work. export const FLOW = [ { name: "event", unit: 1, rate: K01.event, tier: 1 }, // carrier status write { name: "tracking", unit: 1, rate: K01.tracking, tier: 2 }, // read returned to the user { name: "batch", unit: 60, rate: 1.5, tier: 3 }, // end-of-day billing request ]; export const CAPACITY = 700; // AY12: units processed per round export const CEILING_RATIO = 1.2; // throttling ceiling = usual rate x 1.2 export const OSCILLATION = [0.85, 0.85, 0.85, 1.45]; // arrival oscillation of the tracking flow export function run({ roundCount, throttle, shed, capacity = CAPACITY, batchRate = 1.5, oscillate = true }) { const rates = FLOW.map((a) => (a.name === "batch" ? batchRate : a.rate)); const ceiling = FLOW.map((a) => a.rate * CEILING_RATIO); // derived from the known usual rate const carry = FLOW.map(() => 0); const queue = []; // { i, unit, round } const s = { arrived: FLOW.map(() => 0), accepted: FLOW.map(() => 0), throttled: FLOW.map(() => 0), shedRejected: FLOW.map(() => 0), oldestAge: 0, }; const LIMIT = 2 * capacity; // one round's work may wait for (let t = 1; t <= roundCount; t += 1) { const arrivals = []; const remainingCeiling = [...ceiling]; // ceiling does not carry over rounds FLOW.forEach((a, i) => { // arrival: fractional rate with carry const multiplier = a.name === "tracking" && oscillate ? OSCILLATION[(t - 1) % OSCILLATION.length] : 1; carry[i] += rates[i] * multiplier; while (carry[i] >= 1) { carry[i] -= 1; s.arrived[i] += 1; if (throttle && remainingCeiling[i] < 1) { s.throttled[i] += 1; continue; } remainingCeiling[i] -= 1; arrivals.push(i); } }); let pending = queue.reduce((x, o) => x + o.unit, 0); arrivals.sort((x, y) => FLOW[x].tier - FLOW[y].tier); // higher tier accepted first for (const i of arrivals) { if (shed && pending + FLOW[i].unit > LIMIT) { s.shedRejected[i] += 1; continue; } queue.push({ i, unit: FLOW[i].unit, round: t }); pending += FLOW[i].unit; s.accepted[i] += 1; } let budget = capacity; // the round's processing capacity while (queue.length > 0 && queue[0].unit <= budget) { const o = queue.shift(); budget -= o.unit; s.oldestAge = Math.max(s.oldestAge, t - o.round); } } s.backlog = queue.reduce((x, o) => x + o.unit, 0); s.pendingAge = queue.length === 0 ? 0 : roundCount - queue[0].round; return s; } ``` ```js // gate/run.mjs — throttling and load shedding measured separately and together over three days import { K01, FLOW, CAPACITY, CEILING_RATIO, run } from "./edge.mjs"; const ROUNDS = 600; // comparison window; totals scale linearly with duration const SETTING = [ ["neither", { throttle: false, shed: false }], ["throttling only", { throttle: true, shed: false }], ["shedding only", { throttle: false, shed: true }], ["both", { throttle: true, shed: true }], ]; const DAY = [ ["failure-free", {}], ["AY13a: batch job moves into peak hour (1.5 -> 4.5 req/round)", { batchRate: 4.5 }], ["AY13b: an edge node goes down (capacity 700 -> 525)", { capacity: 525 }], ]; const usual = FLOW.reduce((x, a) => x + a.rate * a.unit, 0); console.log(`capacity ${CAPACITY} units/round; usual demand ${usual.toFixed(2)} units/round` + ` (utilization ${((usual / CAPACITY) * 100).toFixed(1)}%)`); console.log(`flows: ${FLOW.map((a) => `${a.name}(tier ${a.tier}, ${a.unit} unit, ${a.rate}/round)`).join(", ")}`); console.log(`throttling ceilings (usual rate x ${CEILING_RATIO}): ` + FLOW.map((a) => `${a.name} ${(a.rate * CEILING_RATIO).toFixed(2)}`).join(", ")); for (const [day, extra] of DAY) { console.log(`\n-- ${day} --`); console.log(`${"setting".padEnd(18)}${"accepted event/tracking/batch".padStart(30)}` + `${"throttled".padStart(15)}${"shed".padStart(16)}${"oldest age".padStart(13)}` + `${"backlog".padStart(8)}${"pending age".padStart(13)}`); for (const [label, y] of SETTING) { const r = run({ roundCount: ROUNDS, ...y, ...extra }); console.log(`${label.padEnd(18)}${r.accepted.join("/").padStart(30)}` + `${r.throttled.join("/").padStart(15)}${r.shedRejected.join("/").padStart(16)}` + `${String(r.oldestAge).padStart(13)}${String(r.backlog).padStart(8)}` + `${String(r.pendingAge).padStart(13)}`); } } console.log(`\n-- the peak AY13b keeps --`); const shed = run({ roundCount: ROUNDS, throttle: false, shed: true, capacity: 525 }); const kept = (shed.accepted[0] + shed.accepted[1]) / ROUNDS; console.log(`with shedding on, accepted event+tracking = ${kept.toFixed(2)} req/round;` + ` K01 peak edge ${K01.edgePeak} req/s -> ratio ${(kept / K01.edgePeak).toFixed(4)}`); console.log(`batch flow's unit load ${(FLOW[2].rate * FLOW[2].unit).toFixed(2)}/round, node loss's gap` + ` ${(FLOW.reduce((x, a) => x + a.rate * a.unit, 0) - 525).toFixed(2)}/round`); const [MIN, TIMES, FAILURE_SHARE] = [8, 2, 28.2]; // AY13b duration and K01's monthly failure share (min) console.log(`gateless, AY13b is an outage: ${MIN} min x ${TIMES} = ${MIN * TIMES} min/mo =` + ` ${(((MIN * TIMES) / FAILURE_SHARE) * 100).toFixed(1)}% of K01's failure share (${FAILURE_SHARE} min)`); console.log(`gated, the same duration is not an outage but a partial rejection:` + ` ${(shed.shedRejected.reduce((x, y) => x + y, 0) / ROUNDS).toFixed(2)} req/round rejected`); console.log(`\n-- failure-free day's cost --`); const th = run({ roundCount: ROUNDS, throttle: true, shed: false }); const thFlat = run({ roundCount: ROUNDS, throttle: true, shed: false, oscillate: false }); const sh = run({ roundCount: ROUNDS, throttle: false, shed: true }); const neither = run({ roundCount: ROUNDS, throttle: false, shed: false }); console.log(`throttling: ${th.throttled[1]} valid tracking queries rejected = ` + `${((th.throttled[1] / th.arrived[1]) * 100).toFixed(2)}% of arrivals, ${((K01.tracking * th.throttled[1]) / th.arrived[1]).toFixed(2)} req/s at peak read`); console.log(`same setting with oscillation off: ${thFlat.throttled[1]} rejections -> what exceeds the ceiling is not average rate but the peak`); console.log(`shedding: ${sh.shedRejected.reduce((x, y) => x + y, 0)} rejections, oldest age ${sh.oldestAge} round(s)` + ` (gateless: ${neither.oldestAge} round(s))`); console.log(`every request passes two checks: flow ceiling and utilization -> 2 checks per request`); console.log(`\n-- AY13's sensitivity --`); console.log(`${"scenario".padEnd(26)}${"both: event/tracking/batch".padStart(31)}${"total rejected".padStart(15)}`); for (const [label, extra] of [["batch 3.0 req/round", { batchRate: 3 }], ["batch 6.0 req/round", { batchRate: 6 }], ["capacity 595 (15% loss)", { capacity: 595 }], ["capacity 455 (35% loss)", { capacity: 455 }]]) { const r = run({ roundCount: ROUNDS, throttle: true, shed: true, ...extra }); const rejected = r.throttled.reduce((x, y) => x + y, 0) + r.shedRejected.reduce((x, y) => x + y, 0); console.log(`${label.padEnd(26)}${r.accepted.join("/").padStart(31)}${String(rejected).padStart(15)}`); } ``` ``` capacity 700 units/round; usual demand 603.89 units/round (utilization 86.3%) flows: event(tier 1, 1 unit, 97.22/round), tracking(tier 2, 1 unit, 416.67/round), batch(tier 3, 60 unit, 1.5/round) throttling ceilings (usual rate x 1.2): event 116.66, tracking 500.00, batch 1.80 -- failure-free -- setting accepted event/tracking/batch throttled shed oldest age backlog pending age neither 58331/250002/900 0/0/0 0/0/0 1 122 0 throttling only 58331/234372/600 0/15630/300 0/0/0 0 0 0 shedding only 58331/250002/900 0/0/0 0/0/0 1 122 0 both 58331/234372/600 0/15630/300 0/0/0 0 0 0 -- AY13a: batch job moves into peak hour (1.5 -> 4.5 req/round) -- setting accepted event/tracking/batch throttled shed oldest age backlog pending age neither 58331/250002/2700 0/0/0 0/0/0 67 52266 66 throttling only 58331/234372/600 0/15630/2100 0/0/0 0 0 0 shedding only 58331/250002/1872 0/0/0 0/0/828 1 680 0 both 58331/234372/600 0/15630/2100 0/0/0 0 0 0 -- AY13b: an edge node goes down (capacity 700 -> 525) -- setting accepted event/tracking/batch throttled shed oldest age backlog pending age neither 58331/250002/900 0/0/0 0/0/0 81 48983 80 throttling only 58331/234372/600 0/15630/300 0/0/0 29 15624 28 shedding only 58331/229912/454 0/0/0 0/20090/446 1 525 0 both 58331/229852/455 0/15630/300 0/4520/145 1 525 0 -- the peak AY13b keeps -- with shedding on, accepted event+tracking = 480.40 req/round; K01 peak edge 513.89 req/s -> ratio 0.9348 batch flow's unit load 90.00/round, node loss's gap 78.89/round gateless, AY13b is an outage: 8 min x 2 = 16 min/mo = 56.7% of K01's failure share (28.2 min) gated, the same duration is not an outage but a partial rejection: 34.23 req/round rejected -- failure-free day's cost -- throttling: 15630 valid tracking queries rejected = 6.25% of arrivals, 26.05 req/s at peak read same setting with oscillation off: 0 rejections -> what exceeds the ceiling is not average rate but the peak shedding: 0 rejections, oldest age 1 round(s) (gateless: 1 round(s)) every request passes two checks: flow ceiling and utilization -> 2 checks per request -- AY13's sensitivity -- scenario both: event/tracking/batch total rejected batch 3.0 req/round 58331/234372/600 16830 batch 6.0 req/round 58331/234372/600 18630 capacity 595 (15% loss) 58331/234372/600 15930 capacity 455 (35% loss) 58331/214820/5 36077 ``` All these numbers belong to the **computed value** class: they are counted over a deterministic arrival sequence. #### Failure-Free Day's Cost **Throttling costs on a failure-free day too.** The tracking flow's ceiling is 500.00 requests per round; when arrival oscillation peaks at 604.17, the difference gets rejected. Over six hundred rounds, 15,630 valid tracking queries are lost — 6.25 percent of arrivals, 26.05 requests per second at K01's peak read rate. With oscillation off, the same setting rejects nothing — so what exceeds the ceiling is not average rate but the peak. Capacity was idle at the time too (`oldest age` and `backlog` both zero): throttling rejects without letting available capacity be used. **Load shedding costs nothing on a failure-free day.** Fullness never nears its limit, so nothing is rejected, and `oldest age` matches the gateless arrangement: 1 round. Its cost is not rejection but **reacting late**. The shared cost is small: every request passes two checks, flow ceiling and fullness. #### Two Failures, Two Mechanisms The second and third tables show the two mechanisms catching **different failure modes**. **Under AY13a (the batch job moves into peak hour), throttling wins.** In the gateless arrangement, delay climbs to 67 rounds and backlog to 52,266 units — a collapse. Holding the batch flow's ceiling at 1.80 per round, throttling cuts 2100 extra requests at the gate; delay zero, backlog zero. Shedding also prevents the collapse but differently: letting idle capacity be used, it accepts 1872 batch requests — **3.1 times** what throttling accepts — at the cost of a one-round delay and a 680-unit backlog. **Under AY13b (an edge node goes down), throttling is useless.** No flow exceeds its ceiling — the flows run at usual rates. Throttling rejects nothing, and with capacity down to 525, delay climbs to 29 rounds and backlog to 15,624 units. Shedding, in the same scenario, keeps delay at 1 round and backlog at 525 units. The difference comes from the criterion: throttling's ceiling is derived from **a known rate** and does not change with capacity, so a capacity loss violates no ceiling and throttling cannot see it. The last row runs both together and gives the best result in both scenarios: neither makes the other unnecessary. #### The Peak Protected The original question was which request gets protected. Under AY13b with shedding on, satisfied status events plus tracking queries total 480.40 requests per round — **0.9348** of K01's 513.89 requests/s peak. The status-event flow is fully protected (58,331 requests); what gets cut is 20,090 tracking queries and 446 batch requests. The arithmetic is in the table: the node loss's gap is 78.89 units per round, the batch flow's load 90.00. Cutting the batch flow entirely would close the gap — but shedding reacts only at saturation, so the queue hits its limit at arrival peaks and tracking queries get cut in those rounds too. The shedding-class order decides what gets cut **first**, not that nothing gets cut. Without the gate, this would be an outage: delay climbs to 81 rounds and no request is satisfied on time. At two occurrences a month, that is 16 minutes — 56.7 percent of K01's 28.2-minute failure share. With the gate, the same duration is a partial rejection, not an outage — zero minutes written to the budget, at the cost of 34.23 rejected requests per round. The last table is sensitivity. Whether the batch rate is 3.0 or 6.0, or capacity loss stays at 15 percent, the satisfied-request count does not change: throttling pins the ceiling, so the anomaly's size at the source never reaches the outcome. At 35 percent loss, though, tracking falls to 214,820 and batch from 600 to 5 requests — the shedding-class order exhausts the lowest-priority class first, then enters the protected one. #### Summary - Throttling cuts what exceeds a known rate at the source (criterion: flow); load shedding rejects by shedding-class order according to instantaneous capacity (criterion: fullness); the Traffic Layer course's gateway offloading moves work to the gateway, while this load shedding never accepts the request at all. - The failure-free day's cost belongs only to throttling: 15,630 valid tracking queries rejected (6.25 percent of arrivals, 26.05 requests/s at peak read), zeroing out with oscillation off. Shedding rejects nothing on a failure-free day. - Under AY13a (batch job moves into peak hour), the gateless arrangement collapses with a 67-round delay; throttling zeroes it, and shedding also prevents the collapse while letting idle capacity be used, passing 3.1 times more batch requests. - Under AY13b (an edge node goes down), throttling rejects nothing and delay climbs to 29 rounds — its criterion is the flow, so it cannot see the capacity loss. Shedding keeps delay at 1 round. - Shedding protects 0.9348 of K01's 513.89 requests/s peak under AY13b and keeps the status-event flow fully up; in the gateless arrangement, the same 16 minutes would have eaten 56.7 percent of K01's 28.2-minute failure share. #### Next Step The gate kept its promise to the accepted request and openly rejected what it did not accept. One gap remained. The 20,090 tracking queries cut under AY13b went entirely to waste — yet a tracking query's response is not a single piece: the shipment's status, estimated delivery time, and fee are separate fields from separate dependencies. When one dependency goes down, dropping the whole request throws away the fields that were already available too. The next lesson takes this up: deciding which part of the response is core, still answering when non-core fields do not arrive, and counting what that costs on a failure-free day. ### Graceful Degradation Source: https://academia.sh/en/courses/resilience-patterns/graceful-degradation Separating the response's core set and answering with missing fields: measuring field skip and stale fallback against K01's 480-byte tracking response, a 15-minute outage zeroing out when an optional dependency goes down, the same pattern only cutting the outage to 14.02 minutes when a core dependency goes down, and isolation's limit written by that difference. The previous lesson built the gate and openly rejected the rejected request. One check on the rejection was left undone: when a request drops, everything gathered for it up to that point drops with it. A tracking query's response is not a single piece — the shipment's state, route, estimated delivery time, and fee are separate fields from separate dependencies. When one dependency goes down, dropping the whole request also throws away the other three fields already available. This lesson takes up the last option: deciding which part of the response is core, and still answering when non-core fields do not arrive. Its name is **graceful degradation**, not to be confused with **graceful shutdown**: graceful shutdown is a process finishing the work it holds while shutting down and taking no new work, established in the Server-Side Fundamentals and Caching, Queues and Asynchronous Processing courses. Graceful degradation, by contrast, is a still-standing system shrinking **the content of its response**. #### Fields and the Core Set **AY14 — core set, fallback store, and two failure scenarios.** The core set is the `state` and `route` fields; `estimate` and `fee` are optional. Rationale: the tracking page is meaningless without showing where the shipment is and what state it is in, while estimated delivery and fee are supporting information. The fallback store keeps the last known value for 5000 shipments, and a stale value is served for at most 60 rounds. **AY14a:** the billing service stops, duration 15 minutes, once a month. **AY14b:** the projection read model stops, duration 15 minutes, once a month. Both durations sit a little above K01's ten-minute recovery-time assumption, and neither is added to K01's table. The fields' byte sizes split K01's 480-byte tracking response: state 180, route 120, estimate 90, fee 90. Three policies are compared. **All or nothing** drops the request if any field is missing. **Field skip** requires the core but drops a missing optional field from the response. **Stale fallback** additionally serves the last known value — itself a form of degradation, its width set by the staleness bound. ```js // degradation/response.mjs — MODEL of the tracking response: fields are gathered from separate // dependencies. A round is an abstract step (one round = one second) and each round represents // one sample request from the flow. No real service, network, or store is set up. export const K01 = { readPeak: 416.67, responseBytes: 480, failureShare: 28.2 }; // AY14: core set {state, route}; total bytes give K01's 480-byte tracking response. export const FIELD = [ { name: "state", source: "projection", core: true, bytes: 180 }, { name: "route", source: "projection", core: true, bytes: 120 }, { name: "estimate", source: "estimate", core: false, bytes: 90 }, { name: "fee", source: "billing", core: false, bytes: 90 }, ]; export const MARKER_BYTES = 6; // freshness marker per field export const FALLBACK_RECORDS = 5000; // shipment count kept in the fallback store export const STALENESS_BOUND = 60; // AY14: a stale value is served for at most 60 rounds const TRANSIENT = 40; // transient timeout interval on a healthy dependency export function run({ policy, mode, warmup = 120, failure = 900 }) { const fallback = new Map(); // field -> round of the last fresh value const s = { full: 0, reduced: 0, dropped: 0, freshField: 0, staleField: 0, missingField: 0, falseDegradation: 0, totalBytes: 0, lastResponseRound: 0, }; for (let t = 1; t <= warmup + failure; t += 1) { const failing = t > warmup; const outcome = []; FIELD.forEach((a, i) => { const health = failing ? mode[a.source] : "healthy"; const transient = health === "healthy" && (t + i * 10) % TRANSIENT === 0; if (health === "healthy" && transient === false) { fallback.set(a.name, t); outcome.push({ a, status: "fresh" }); return; } const age = fallback.has(a.name) ? t - fallback.get(a.name) : Infinity; if (policy === "fallback" && age <= STALENESS_BOUND) { outcome.push({ a, status: "stale" }); if (transient && failing) s.falseDegradation += 1; // fresh value existed, transient timeout cut it return; } outcome.push({ a, status: "missing" }); }); if (failing === false) continue; // warmup rounds are not counted const missingCore = outcome.some((o) => o.a.core && o.status === "missing"); const anyMissing = outcome.some((o) => o.status !== "fresh"); if (policy === "all" ? anyMissing : missingCore) { s.dropped += 1; continue; } let bytes = 0; for (const o of outcome) { if (o.status === "fresh") s.freshField += 1; else if (o.status === "stale") s.staleField += 1; else { s.missingField += 1; continue; } bytes += o.a.bytes + MARKER_BYTES; } s.totalBytes += bytes; s.lastResponseRound = t - warmup; if (anyMissing) s.reduced += 1; else s.full += 1; } s.responses = s.full + s.reduced; s.avgBytes = s.responses === 0 ? 0 : s.totalBytes / s.responses; s.round = failure; return s; } ``` Warmup rounds represent the failure starting **after** the system has run healthy for a while. The fallback store can only serve a value that was filled in during that time; a field never seen before has no fallback. ```js // degradation/run.mjs — comparing three policies over three days, and isolation's limit import { K01, FIELD, MARKER_BYTES, FALLBACK_RECORDS, STALENESS_BOUND, run } from "./response.mjs"; const FAILURE = 900; // AY14: 15 minutes = 900 rounds const POLICY = [["all or nothing", "all"], ["field skip", "skip"], ["+ stale fallback", "fallback"]]; const HEALTHY = { projection: "healthy", estimate: "healthy", billing: "healthy" }; const DAY = [ ["failure-free", HEALTHY], ["AY14a: billing stops (optional field)", { ...HEALTHY, billing: "down" }], ["AY14b: projection stops (core field)", { ...HEALTHY, projection: "down" }], ]; console.log(`${FAILURE} rounds (after 120 rounds of warmup); staleness bound ${STALENESS_BOUND} rounds`); console.log(`fields: ` + FIELD.map((a) => `${a.name}(${a.source}, ${a.core ? "core" : "optional"}, ${a.bytes} B)`).join(", ")); console.log(`total field bytes ${FIELD.reduce((x, a) => x + a.bytes, 0)} = K01 tracking response ${K01.responseBytes} B\n`); for (const [day, mode] of DAY) { console.log(`-- ${day} --`); console.log(`${"policy".padEnd(20)}${"responses".padStart(10)}${"full".padStart(5)}${"reduced".padStart(11)}` + `${"dropped".padStart(8)}${"fresh/stale/missing field".padStart(26)}${"avg. bytes".padStart(11)}${"last response round".padStart(20)}`); for (const [label, p] of POLICY) { const r = run({ policy: p, mode, failure: FAILURE }); console.log(`${label.padEnd(20)}${String(r.responses).padStart(10)}${String(r.full).padStart(5)}` + `${String(r.reduced).padStart(11)}${String(r.dropped).padStart(8)}` + `${`${r.freshField}/${r.staleField}/${r.missingField}`.padStart(26)}${r.avgBytes.toFixed(1).padStart(11)}` + `${String(r.lastResponseRound).padStart(20)}`); } console.log(); } const a = run({ policy: "all", mode: HEALTHY, failure: FAILURE }); const y = run({ policy: "fallback", mode: HEALTHY, failure: FAILURE }); console.log(`-- failure-free day's cost --`); console.log(`freshness marker ${MARKER_BYTES} B x ${FIELD.length} fields = ${MARKER_BYTES * FIELD.length} B/response` + ` = ${((MARKER_BYTES * FIELD.length) / K01.responseBytes * 100).toFixed(1)}% of K01's response,` + ` ${((K01.readPeak * MARKER_BYTES * FIELD.length) / 1024).toFixed(2)} KiB/s at peak read`); console.log(`fallback store ${FALLBACK_RECORDS} shipments x ${K01.responseBytes} B = ` + `${((FALLBACK_RECORDS * K01.responseBytes) / 1024 / 1024).toFixed(2)} MiB; every fresh field is one store write ->` + ` ${(K01.readPeak * FIELD.length).toFixed(2)} writes/s`); console.log(`the strict policy drops ${a.dropped}/${FAILURE} requests on a failure-free day` + ` -> ${((K01.readPeak * a.dropped) / FAILURE).toFixed(2)} req/s at peak read`); console.log(`with stale fallback, the same day drops ${y.dropped}, serves ${y.staleField} fields stale` + ` = ${((y.staleField / (FAILURE * FIELD.length)) * 100).toFixed(2)}% of fields`); console.log(`\n-- impact on the downtime budget (AY14: 15 min, once a month) --`); console.log(`${"scenario".padEnd(9)}${"policy".padEnd(21)}${"answered".padStart(11)}` + `${"outage(min)".padStart(12)}${"of failure share(28.2)".padStart(23)}`); for (const [label, mode] of [["AY14a", DAY[1][1]], ["AY14b", DAY[2][1]]]) { for (const [pad, p] of POLICY) { const r = run({ policy: p, mode, failure: FAILURE }); const min = 15 * (r.dropped / FAILURE); console.log(`${label.padEnd(9)}${pad.padEnd(21)}${String(r.responses).padStart(11)}` + `${min.toFixed(2).padStart(12)}${`${((min / K01.failureShare) * 100).toFixed(1)}%`.padStart(23)}`); } } console.log(`\n-- isolation's limit: what each source going down leaves --`); console.log(`${"failed source".padEnd(16)}${"affected field".padStart(15)}${"core?".padStart(12)}` + `${"responses".padStart(10)}${"avg. bytes".padStart(11)}${"last response round".padStart(20)}`); for (const source of ["billing", "estimate", "projection"]) { const affected = FIELD.filter((x) => x.source === source); const r = run({ policy: "fallback", mode: { ...HEALTHY, [source]: "down" }, failure: FAILURE }); console.log(`${source.padEnd(16)}${affected.map((x) => x.name).join("+").padStart(15)}` + `${(affected.some((x) => x.core) ? "yes" : "no").padStart(12)}` + `${String(r.responses).padStart(10)}${r.avgBytes.toFixed(1).padStart(11)}${String(r.lastResponseRound).padStart(20)}`); } ``` ``` 900 rounds (after 120 rounds of warmup); staleness bound 60 rounds fields: state(projection, core, 180 B), route(projection, core, 120 B), estimate(estimate, optional, 90 B), fee(billing, optional, 90 B) total field bytes 480 = K01 tracking response 480 B -- failure-free -- policy responses full reduced dropped fresh/stale/missing field avg. bytes last response round all or nothing 810 810 0 90 3240/0/0 504.0 899 field skip 856 810 46 44 3378/0/46 498.8 900 + stale fallback 900 810 90 0 3510/90/0 504.0 900 -- AY14a: billing stops (optional field) -- policy responses full reduced dropped fresh/stale/missing field avg. bytes last response round all or nothing 0 0 0 900 0/0/0 0.0 0 field skip 856 0 856 44 2545/0/879 405.4 900 + stale fallback 900 0 900 0 2633/127/840 414.4 900 -- AY14b: projection stops (core field) -- policy responses full reduced dropped fresh/stale/missing field avg. bytes last response round all or nothing 0 0 0 900 0/0/0 0.0 0 field skip 0 0 0 900 0/0/0 0.0 0 + stale fallback 59 0 59 841 115/121/0 504.0 59 -- failure-free day's cost -- freshness marker 6 B x 4 fields = 24 B/response = 5.0% of K01's response, 9.77 KiB/s at peak read fallback store 5000 shipments x 480 B = 2.29 MiB; every fresh field is one store write -> 1666.68 writes/s the strict policy drops 90/900 requests on a failure-free day -> 41.67 req/s at peak read with stale fallback, the same day drops 0, serves 90 fields stale = 2.50% of fields -- impact on the downtime budget (AY14: 15 min, once a month) -- scenario policy answered outage(min) of failure share(28.2) AY14a all or nothing 0 15.00 53.2% AY14a field skip 856 0.73 2.6% AY14a + stale fallback 900 0.00 0.0% AY14b all or nothing 0 15.00 53.2% AY14b field skip 0 15.00 53.2% AY14b + stale fallback 59 14.02 49.7% -- isolation's limit: what each source going down leaves -- failed source affected field core? responses avg. bytes last response round billing fee no 900 414.4 900 estimate estimate no 900 414.4 900 projection state+route yes 59 504.0 59 ``` All these numbers belong to the **computed value** class: they are counted over a deterministic sequence of rounds. #### Failure-Free Day's Cost Three costs are paid, all three even without a failure. **The field it carries.** Every field carries a freshness marker: whether the value is fresh or stale, and if stale, how old. Six bytes, four fields, 24 bytes per response — 5.0 percent of K01's 480-byte tracking response, 9.77 KiB/s at the peak read rate. Without this marker, degradation is silent, and silent degradation is its most dangerous form: the client mistakes a stale fee for a current one. **The fallback store.** The last known value for five thousand shipments takes 2.29 MiB, but the real cost is not memory, it is writes: every fresh field updates the store, 1666.68 writes per second at peak read — a load paid continuously even with no failure, and the pattern's most expensive item. **Fields served stale.** With stale fallback on, even on a failure-free day, 90 of 3600 fields are served stale — 2.50 percent. These are transient timeouts: the fresh value would have arrived the next round, but the fallback engaged immediately. The fallback path's own freshness is itself a cost. Against this, the first table also shows the strict policy's own cost: on a failure-free day it drops 90 of 900 requests, 41.67 requests per second at peak read, with no dependency down at all. The strict policy is not a baseline; it is a defect in its own right. #### The Failing Day's Gain The AY14a table shows what the pattern buys. When the billing service stops for fifteen minutes, the strict policy drops 900 of 900 requests: the entire fifteen minutes is an outage, eating 53.2 percent of K01's monthly 28.2-minute failure share — a single failure spending more than half the month's allowance. Field skip answers 856 requests in the same failure, and the outage drops to 0.73 minutes — 2.6 percent of the share. Adding stale fallback answers 900 of 900, and **the outage zeroes out**. The response's average size drops from 504 to 414.4 bytes; the user sees where the shipment is and its state, but not the fee. A design decision becomes visible here. The fee field is served stale for the first sixty rounds, then dropped once the staleness bound is exceeded: the table shows 127 stale and 840 missing. A stale fee is close to true for a while; after fifteen minutes it is outright wrong. The bound is the name for that "while." #### Isolation's Limit The AY14b table writes this topic's closing sentence. When the projection read model — the sole source of both core fields — stops, all three policies collapse. The strict policy and field skip drop 900 of 900 requests: fifteen minutes of outage, 53.2 percent of the share. Stale fallback answers only 59 requests and falls silent at round 59; outage 14.02 minutes, 49.7 percent. The gain is one minute, exactly the staleness bound. Degradation does not carry the core, it only **postpones**. The last table sums this up in three rows: an optional source going down leaves the response count at 900; a core source going down drops it to 59, with the last response given at round 59. This says something about the topic as a whole. Every pattern here **contains** the failure: it takes the broken dependency out of the loop, separates resource pools, bounds retries, budgets time, reports overload to the source, rejects requests exceeding capacity, trims the response's perimeter. None of them **removes** the failure. The broken part stays broken; the only difference is that the system around it does not break with it. When a core part breaks, containment has nothing left to give. What is needed then is a copy to **stand in for** the broken part — and no such copy was ever designed in this topic. Where the copy stands, when it takes over, how many requests drop during the takeover, and how much a wrongly made takeover decision costs are all open questions. #### Summary - Graceful degradation splits the response's fields into core and optional sets and answers if the core is present; distinct from graceful shutdown — one is a shutting-down process finishing its work, the other a still-standing system shrinking its response. - The failure-free day's cost has three items: a 24-byte freshness marker per response (5.0 percent of K01's response, 9.77 KiB/s at peak read), a 2.29 MiB fallback store with 1666.68 writes per second, and 2.50 percent of fields served stale even without a failure. - Under AY14a (optional dependency down), the strict policy drops every one of 900 requests and, at 15.00 minutes, eats 53.2 percent of K01's 28.2-minute failure share; field skip brings this to 0.73 minutes, stale fallback to 0.00. The response's average size drops from 504 to 414.4 bytes. - Stale fallback does not carry the core, it postpones it: under AY14b (core dependency down), only 59 requests are answered, the outage falls from 15.00 to 14.02 minutes — a gain exactly the staleness bound. - Every pattern in this topic contains the failure, none removes it; when a core part breaks, containment has nothing left to give. #### Next Step The gap this topic leaves fits in one sentence: there is nothing to stand in for the broken part. The isolation patterns sealed the failure inside a box and kept the system outside it standing; the part inside is still broken, and no mechanism to fix it was designed. AY14b's 14.02 minutes is exactly the measure of that gap — containment cannot reach past it. The next topic looks inside the box and asks the first question: if a copy stands ready to replace the broken part, how is its takeover decided. The decision itself is a source of failure, because a slowed-down part and a stopped one look the same from the outside, and a wrongly made takeover decision can cost more than never taking over at all. That topic takes up the failover window, what triggers the decision, and how many minutes a takeover writes to the downtime budget. ## Recovery and Continuity Design after the failure has already happened: choosing between the active–passive and active–active setup by failover cost, the health endpoint feeding automatic removal, converting recovery time and data loss objectives into numbers and comparing them against the outage budget, redundancy at the region and zone level, replicating scale units as stamps, and testing the plan with a drill. ### Failover Design Source: https://academia.sh/en/courses/resilience-patterns/failover-design The three decisions failover leaves after the setup choice: choosing the consecutive-failed-check threshold that triggers promotion numerically between false failover and detection delay, the second transition window the failback policy produces, counting the writes a replica that responds but cannot do its job silently swallows, and converting all these transitions into the monthly outage budget's failure share. The previous topic contained the failure with eight patterns: the circuit breaker routed around a broken dependency, the bulkhead pattern separated resource pools, the timeout budget bounded the wait, graceful degradation kept the core function standing. All shared one limit: the failure was **contained**, not removed. The broken part is still broken, and how a replacement replica takes over was never designed. The setup itself was measured in the Introduction to System Design course: in active–passive, writes drop until failover completes; in active–active, there is no failover window, but conflicting writes occur. That measurement is an **input** here, not repeated. This lesson's question sits one layer higher: when failover triggers, what happens when the old active replica returns, why a replica that looks up but cannot do its job never triggers failover, and how many seconds these transitions cost from the outage budget. #### The Three Decisions Left After the Setup The setup choice says whether a replica accepts writes; it does not design the transition itself. Three decisions remain. **Triggering.** How many consecutive failed health checks start the promotion. At threshold 1, the first failed check starts the transition; at threshold 4, four are awaited. What the check **asks** is the next lesson's subject; here it only asks "does it respond", which makes it shallow. **Failback.** When the old active replica recovers, is activity handed back to it, or does the new replica stay active permanently. Failback is itself a transition with its own window. **Distinguishing.** How a replica that responds but cannot do its job is seen. Since this replica passes the health check, no threshold catches it. Two replicas both believing themselves active is a separate problem; the lease duration and the two-leader round were established in the Application Layer and Service Interaction course and are not repeated here. #### The Setup The setup below is a **model**: no real cluster, zone, or container is set up. A round is an abstract step — the failover model's arrangement from the Introduction to System Design course continues. The failure schedule is a parameter: replica `A` briefly stops responding twice and recovers on its own, actually stops twice, and goes **half-failed** once (up, passing the shallow check, but unable to write to the state store). Replica `B` is always up in the model, a simplification that leaves both replicas failing together unaccounted for. ```js // failover/model.mjs — active-passive failover MODEL. A round is an abstract step (the // arrangement from the Availability Patterns lesson, M19/K01); the detection threshold, // promotion, and failback windows are parameters in rounds, not measured durations. No real // cluster is set up. export const ROUND = 130; export const BRIEF = [[12, 12], [20, 21]]; // A does not respond but recovers on its own export const STOPPED = [[30, 75], [90, 96]]; // A actually stopped export const HALF_FAILED = [[100, 120]]; // A is up, passes the shallow check, but cannot write to the store const PROMOTION = 1; // assumption: promotion takes one round const FAILBACK_THRESHOLD = 3; // assumption: consecutive healthy checks required for failback const within = (t, a) => a.some(([b, c]) => t >= b && t <= c); export const isResponsive = (t) => !within(t, BRIEF) && !within(t, STOPPED); export const isHealthy = (t) => isResponsive(t) && !within(t, HALF_FAILED); export function run({ threshold, failback, round: N = ROUND }) { const s = { dropped: 0, silent: 0, failovers: 0, falseFailovers: 0, failbacks: 0, window: 0 }; let active = "A", consecutive = 0, healthy = 0, doneAt = null, direction = null; for (let t = 1; t <= N; t += 1) { const responds = active === "A" ? isResponsive(t) : true; // B is always up in the model if (doneAt !== null) { // transition window is running s.window += 1; s.dropped += 1; if (t === doneAt) { // retroactive classification: the old active replica recovered on its own, the failover was unnecessary if (direction === "failover" && isResponsive(t + 1)) s.falseFailovers += 1; active = direction === "failover" ? "B" : "A"; doneAt = null; direction = null; consecutive = 0; healthy = 0; } continue; } if (active === "A" && responds === false) { // shallow check: response only consecutive += 1; s.dropped += 1; if (consecutive >= threshold) { doneAt = t + PROMOTION; direction = "failover"; s.failovers += 1; } continue; } consecutive = 0; if (active === "B" && failback === "automatic") { healthy = isResponsive(t) ? healthy + 1 : 0; if (healthy >= FAILBACK_THRESHOLD) { doneAt = t + PROMOTION; direction = "failback"; s.failbacks += 1; healthy = 0; continue; } } if (active === "A" && isHealthy(t) === false) s.silent += 1; // write accepted, never reached the store } return s; } export function noFailover(N = ROUND) { // no failover: A is whatever it is let dropped = 0, silent = 0, accepted = 0; for (let t = 1; t <= N; t += 1) { if (isResponsive(t) === false) dropped += 1; else if (isHealthy(t) === false) silent += 1; else accepted += 1; } return { accepted, dropped, silent }; } ``` ```js // failover/measure.mjs — measuring the detection threshold and failback policy in the same failure schedule import { run, noFailover, ROUND, BRIEF, STOPPED, HALF_FAILED } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); const length = (a) => a.reduce((t, [b, c]) => t + (c - b + 1), 0); console.log(`${ROUND} rounds. Replica A: ${length(BRIEF)} rounds briefly unresponsive, ` + `${length(STOPPED)} rounds stopped, ${length(HALF_FAILED)} rounds half-failed. B is always up in the model.`); const nf = noFailover(); console.log(`no failover: accepted ${nf.accepted}, dropped ${nf.dropped}, silent lost update ${nf.silent}`); console.log(); console.log("threshold | failovers | false failovers | failbacks | transition round | dropped | silent lost update"); console.log("----------|-----------|------------------|-----------|-------------------|---------|--------------------"); for (const threshold of [1, 2, 3, 4]) { const r = run({ threshold, failback: "automatic" }); console.log(`${s(threshold, 9)} | ${s(r.failovers, 9)} | ${s(r.falseFailovers, 16)} | ${s(r.failbacks, 9)} | ` + `${s(r.window, 17)} | ${s(r.dropped, 7)} | ${s(r.silent, 19)}`); } console.log(); console.log("failback | threshold | failovers | failbacks | transition round | dropped | active replica changes"); console.log("----------|-----------|-----------|-----------|-------------------|---------|------------------------"); for (const failback of ["automatic", "permanent"]) { const r = run({ threshold: 3, failback }); console.log(`${failback.padEnd(9)} | ${s(3, 9)} | ${s(r.failovers, 9)} | ${s(r.failbacks, 9)} | ` + `${s(r.window, 17)} | ${s(r.dropped, 7)} | ${s(r.failovers + r.failbacks, 22)}`); } ``` ``` 130 rounds. Replica A: 3 rounds briefly unresponsive, 53 rounds stopped, 21 rounds half-failed. B is always up in the model. no failover: accepted 53, dropped 56, silent lost update 21 threshold | failovers | false failovers | failbacks | transition round | dropped | silent lost update ----------|-----------|------------------|-----------|-------------------|---------|-------------------- 1 | 4 | 2 | 4 | 8 | 12 | 20 2 | 3 | 1 | 3 | 6 | 13 | 20 3 | 2 | 0 | 2 | 4 | 13 | 20 4 | 2 | 0 | 2 | 4 | 15 | 20 failback | threshold | failovers | failbacks | transition round | dropped | active replica changes ----------|-----------|-----------|-----------|-------------------|---------|------------------------ automatic | 3 | 2 | 2 | 4 | 13 | 4 permanent | 3 | 1 | 0 | 1 | 7 | 1 ``` #### Two Sides of the Threshold The no-failover row is the baseline: without failover, 56 of 130 rounds accept no write. With threshold 3, dropped rounds in the same schedule fall to 13 — the pattern's gain on the failing day. The threshold column shows the trade-off in two directions. **As the threshold shrinks, false failovers rise.** Threshold 1 treated both self-recovering outages as transitions: 2 of 4 failovers are false, since the old active replica started responding again just as promotion finished. Threshold 2 caught one of these; thresholds 3 and 4 triggered on neither. **As the threshold grows, detection is delayed.** The time from a real stop's start to the transition's completion lengthens with the threshold: dropped rounds are 13 at threshold 3, 15 at threshold 4. The total is not one-directional: rounds dropped at threshold 1 and threshold 3 come out close, 12 and 13, because the false failover's window takes back the detection speed threshold 1 gained. **The threshold is not a speed decision but a balance of two costs**, and where it sits depends on the failure schedule. The silent lost update column is 20 in all four rows. It does not change with the threshold, because the half-failed replica passes the shallow check and the counter never increases. #### Failback Is a Second Window The second table compares two failback policies at the same threshold. Under automatic failback, the active replica changes four times (2 failovers, 2 failbacks), a transition round count of 4; under the permanent policy, one change and one transition round. Dropped rounds fall from 13 to 7. The difference comes from two places. First, every failback produces its own window. Second, and more important, automatic failback **reconnects the system to the same replica**: `A` recovers after its first stop, activity is handed back to it, and it stops a second time shortly after. Under the permanent policy this second stop produces no outage at all, since the active replica is now `B`. Automatic failback's payoff is operational: the replicas' roles return to a known arrangement, bought with 6 extra dropped rounds. #### The Replica That Responds But Cannot Write The half-failed window is this model's quietest part. The replica is up, passes the health check, and accepts requests — but the write it accepted never reaches the state store. Writes were **accepted and lost** over 21 rounds without the pattern, 20 rounds with it. The outage budget never sees these rounds, because the system was responding. #### The Numbers for Two Days Rounds convert to a budget only with a cycle-length assumption. This course's own assumptions are not added to K01's table; they are cited by their own names. **KU1 — a round is one health-check interval, 2 seconds.** **KU2 — 2 real stops a month**, since K01's failure share tolerates 2.82 failures a month. **KU3 — 20 transient unresponsive periods a month**, since brief pauses are more frequent than real stops. Recovery time, peak write, and failure share are read from K01. ```js // failover/budget.mjs — converting the model's rounds into K01's outage budget and request counts const ROUND_SEC = 2; // KU1 (assumption): one round is one health-check interval, 2 seconds const REAL_STOPS_MONTH = 2; // KU2 (assumption): 2 real stops a month const BRIEF_MONTH = 20; // KU3 (assumption): 20 brief unresponsive periods a month const RECOVERY_SEC = 600; // K01: recovery time assumption, 10 minutes const FAILURE_SHARE = 28.2; // K01: the outage budget's monthly failure share (minutes) const PEAK_WRITE = 97.22; // K01: peak write requests/s const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PROMOTION = 1; const b = (x, n = 2) => x.toFixed(n); console.log(`${"threshold".padStart(9)}${"transition window".padStart(19)}${"real stops (month)".padStart(20)}` + `${"false failovers (month)".padStart(25)}${"total outage".padStart(15)}${"of failure share".padStart(18)}`); for (const threshold of [1, 2, 3, 4]) { const window = (threshold + PROMOTION) * ROUND_SEC; const realStop = REAL_STOPS_MONTH * window; const falseFailover = threshold <= 2 ? BRIEF_MONTH * (window + PROMOTION * ROUND_SEC) : 0; // threshold 3+ never triggers on a brief outage const total = realStop + falseFailover; console.log(`${String(threshold).padStart(9)}${(window + " sec").padStart(19)}${(b(realStop, 1) + " sec").padStart(20)}` + `${(b(falseFailover, 1) + " sec").padStart(25)}${(b(total / 60, 2) + " min").padStart(15)}` + `${("%" + b((100 * total) / 60 / FAILURE_SHARE, 2)).padStart(18)}`); } const noFailover = REAL_STOPS_MONTH * RECOVERY_SEC; console.log(`\nno failover: ${REAL_STOPS_MONTH} stops/month x ${RECOVERY_SEC} sec = ${b(noFailover / 60, 1)} min, ` + `%${b((100 * noFailover) / 60 / FAILURE_SHARE, 1)} of failure share`); const threshold3 = REAL_STOPS_MONTH * (3 + PROMOTION) * ROUND_SEC; console.log(`same two stops with threshold 3: ${threshold3} sec, ${b(noFailover / threshold3, 1)}x shorter`); console.log(`dropped in the window: ${b((3 + PROMOTION) * ROUND_SEC * PEAK_WRITE)} writes, ` + `${b((3 + PROMOTION) * ROUND_SEC * PEAK_EDGE)} edge requests (at peak load)`); const SILENT_ROUND = 20; // from the model: rounds accepted during the half-failed window but never reaching the store console.log(`\nhalf-failed window: ${SILENT_ROUND} rounds x ${ROUND_SEC} sec = ${SILENT_ROUND * ROUND_SEC} sec, ` + `${b(SILENT_ROUND * ROUND_SEC * PEAK_WRITE)} writes silently lost`); console.log(`this window is invisible in the outage budget: the system was responding`); console.log(); for (const replicas of [2, 3]) { // active-active: the failed replica carries 1/k of the pool console.log(`active-active, ${replicas} replicas: in the removal window, ` + `${b(((3 + PROMOTION) * ROUND_SEC * PEAK_WRITE) / replicas)} writes drop, ` + `in the half-failed window ${b((SILENT_ROUND * ROUND_SEC * PEAK_WRITE) / replicas)} writes are silently lost`); } ``` ``` threshold transition window real stops (month) false failovers (month) total outage of failure share 1 4 sec 8.0 sec 120.0 sec 2.13 min %7.57 2 6 sec 12.0 sec 160.0 sec 2.87 min %10.17 3 8 sec 16.0 sec 0.0 sec 0.27 min %0.95 4 10 sec 20.0 sec 0.0 sec 0.33 min %1.18 no failover: 2 stops/month x 600 sec = 20.0 min, %70.9 of failure share same two stops with threshold 3: 16 sec, 75.0x shorter dropped in the window: 777.76 writes, 4111.12 edge requests (at peak load) half-failed window: 20 rounds x 2 sec = 40 sec, 3888.80 writes silently lost this window is invisible in the outage budget: the system was responding active-active, 2 replicas: in the removal window, 388.88 writes drop, in the half-failed window 1944.40 writes are silently lost active-active, 3 replicas: in the removal window, 259.25 writes drop, in the half-failed window 1296.27 writes are silently lost ``` **The failing day's gain** is the first two lines compared. Without failover, the same two stops last as long as the recovery time: 20.0 minutes, 70.9 percent of the failure share — nearly the entire budget, and a third stop breaks the target that month. With threshold 3, the same two stops take 16 seconds, a 75x reduction, spending 0.95 percent of the failure share. This ratio does not depend on the machine, only on the ratio of recovery time to transition window. **The failure-free day's cost** is in the false-failover column. At threshold 1, 20 transient unresponsive periods a month produce 120 seconds of outage though none is a real failure; at threshold 2, 160 seconds — in both rows exceeding the failing day's cost, 12 seconds at threshold 2. At threshold 3 it drops to zero, trading the real-stop window up from 4 to 8 seconds. A second failure-free-day cost never shows up in the arithmetic: the passive replica carries its full capacity and answers no requests at all. The window's one-time bill is also recorded: an eight-second transition drops 777.76 writes and 4,111.12 edge requests at peak load — a concentration an outage budget, which measures an average, does not show. The last two lines tie the setup difference to a number. In the active–active setup, the transition window drops only the failed replica's share, not the whole service: 388.88 writes with two replicas, 259.25 with three. The same division holds for the silent lost update — 1,296.27 instead of 3,888.80 with three replicas. Active–active does not make the failure cheaper, it **divides** it; K01's conflicting writes are the price of that division. #### Summary - Three decisions remain after the setup choice: the threshold that triggers promotion, the failback policy toward the old active replica, and distinguishing a replica that responds but cannot do its job. - The threshold balances two costs: threshold 1 turned both self-recovering outages into transitions (2 false failovers), threshold 4 delayed detection (15 dropped rounds); total dropped rounds stayed in a narrow 12–15 range. - Failback produces its own window: 4 transitions and 13 dropped rounds under the automatic policy, 1 transition and 7 dropped rounds under the permanent policy. - No threshold catches the half-failed replica: silent lost updates are 20 rounds at all four thresholds; the counter never increases because the shallow check gets a response. - With the KU1–KU3 assumptions, the failing day's gain is a drop from 20.0 minutes to 16 seconds: a 75x reduction, and 0.95 percent of the failure share instead of 70.9 percent. - The failure-free day's cost is 120 seconds of false-failover outage a month at threshold 1, 160 seconds at threshold 2; it zeroes out at threshold 3, in exchange for the real-stop window rising from 4 to 8 seconds. #### Next Step Every decision in this lesson rested on a single input: the yes or no the health check returns. When that input is wrong, both the threshold and the failback policy lose their meaning — in the model's half-failed window, 3,888.80 writes were lost while the system looked healthy. The next lesson designs the check's **content**: whether the endpoint looks at its dependencies, the risk that adding a dependency to the check marks all replicas unhealthy at once, separating being ready to take traffic from being up, and where the check interval sits between detection delay and false failover. ### Health Endpoint Monitoring Source: https://academia.sh/en/courses/resilience-patterns/health-endpoint-monitoring Designing the content of the health endpoint that feeds failover: measuring the shallow check, readiness check, and deep check in the same failure schedule, how adding a shared dependency to the check evicts every replica from the pool at once, the check interval's trade-off between detection delay and false eviction, and counting the load the check adds to the dependency. Every decision in the previous lesson rested on a single input: the yes or no the health check returns. How the threshold was chosen, when failback happened, even whether failover happened at all depended on this one bit's accuracy. In the model's half-failed window, 3,888.80 writes were lost while the system looked healthy — because the check only asked "are you responding". The health check that feeds failover and the load balancer pool was established in the Relational Database Administration and The Traffic Layer courses; the endpoint's form on the serving side was introduced in the Server-Side Fundamentals course. This lesson designs that endpoint's **content**: what question it asks, whether it puts its dependencies in scope, and how often it asks. Where its collected metrics go, the dashboard, and the alert threshold are not this course's subject; they belong to the Performance Anti-Patterns and Monitoring course. #### The Four Questions the Endpoint Can Ask **The shallow check** only tests that the process is up and can respond. It produces its answer without touching any dependency, so it is cheap and never gives a false negative. **The readiness check** is a separate question: the process may be up but not ready to take traffic. A newly started replica may not have filled its cache, read its configuration, or finished its migration. Being up and being ready are two separate states, and merging them into one endpoint loses one of the two. **The deep check** touches the replica's dependencies. Two kinds of dependency are separated here, and the separation is this lesson's main result: a dependency on the replica's **own path** (whether this replica can write to the state store) versus a **shared** dependency (whether the state store itself is up). The first differs between replicas, the second is the same for all of them. The setup below is a **model**: three replicas, one shared state store, and a billing store that feeds only the batch flow. A round is an abstract step. The request mix per round is chosen close to K01's read–write ratio: 8 reads, 2 writes, 1 batch. ```js // health/model.mjs — the content of the health endpoint is a MODEL. Three replicas, one shared // state store, and a store used only by the batch feed. A round is an abstract step. The request // mix per round is chosen close to K01's read/write ratio: 8 reads, 2 writes, 1 batch. export const ROUND = 130; export const K3_STOPPED = 82; // k3 stops in this round; detection delay is measured from here export const REPLICA = ["k1", "k2", "k3"]; const P = (t, a) => a.some(([b, c]) => t >= b && t <= c); export const schedule = { halfFailed: { k1: [[20, 45]], k2: [], k3: [] }, // looks like it is writing, never reaches the store warming: { k1: [], k2: [[1, 8], [60, 66]], k3: [] }, // process is up, not ready for traffic stopped: { k1: [], k2: [], k3: [[82, 95]] }, // process does not respond brief: { k1: [], k2: [], k3: [[50, 50], [58, 59]] }, // brief unresponsiveness, recovers on its own stateStore: [[100, 110]], // shared dependency goes down billingStore: [[115, 125]], // affects only the batch feed }; export const stateStoreUp = (t) => !P(t, schedule.stateStore); export const billingStoreUp = (t) => !P(t, schedule.billingStore); export const processUp = (k, t) => !P(t, schedule.stopped[k]) && !P(t, schedule.brief[k]); export const warming = (k, t) => P(t, schedule.warming[k]); export const pathHealthy = (k, t) => !P(t, schedule.halfFailed[k]); // the replica's own write path export const canRead = (k, t) => processUp(k, t) && !warming(k, t); // Five endpoint contents. Each returns a single yes/no per replica. export const ENDPOINT = { shallow: (k, t) => processUp(k, t), readiness: (k, t) => canRead(k, t), deepLocal: (k, t) => canRead(k, t) && pathHealthy(k, t), deepShared: (k, t) => canRead(k, t) && pathHealthy(k, t) && stateStoreUp(t), deepAll: (k, t) => canRead(k, t) && pathHealthy(k, t) && stateStoreUp(t) && billingStoreUp(t), }; const READS = 8, WRITES = 2, BATCH = 1; // requests per round: close to K01's read/write ratio export function run(endpointName, { interval = 1, round: N = ROUND } = {}) { const endpoint = ENDPOINT[endpointName]; const s = { outOfPoolRounds: 0, droppedReads: 0, droppedWrites: 0, droppedBatch: 0, silent: 0, wastedReject: 0, checks: 0, falseEviction: 0, detection: null }; let round = 0, decision = Object.fromEntries(REPLICA.map((k) => [k, true])), previousDecision = { ...decision }; for (let t = 1; t <= N; t += 1) { if ((t - 1) % interval === 0) { // check round previousDecision = { ...decision }; for (const k of REPLICA) decision[k] = endpoint(k, t); s.checks += REPLICA.length; for (const k of REPLICA) // did a brief outage evict it from the pool if (previousDecision[k] && decision[k] === false && P(t, schedule.brief[k])) s.falseEviction += 1; if (s.detection === null && decision.k3 === false && P(t, schedule.stopped.k3)) s.detection = t - K3_STOPPED; } const pool = REPLICA.filter((k) => decision[k]); s.outOfPoolRounds += REPLICA.length - pool.length; const someCanRead = REPLICA.some((k) => canRead(k, t)); for (let i = 0; i < READS; i += 1) { if (pool.length === 0) { s.droppedReads += 1; if (someCanRead) s.wastedReject += 1; continue; } const k = pool[round++ % pool.length]; if (canRead(k, t) === false) s.droppedReads += 1; } for (let i = 0; i < WRITES; i += 1) { if (pool.length === 0) { s.droppedWrites += 1; continue; } const k = pool[round++ % pool.length]; if (canRead(k, t) === false) { s.droppedWrites += 1; continue; } if (pathHealthy(k, t) === false) { s.silent += 1; continue; } // accepted, never reached the store if (stateStoreUp(t) === false) s.droppedWrites += 1; } for (let i = 0; i < BATCH; i += 1) { if (pool.length === 0 || billingStoreUp(t) === false) { s.droppedBatch += 1; continue; } round += 1; } } return s; } ``` ```js // health/measure.mjs — five endpoint contents in the same failure schedule, then the effect of the check interval import { run, ROUND, REPLICA } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); console.log(`${ROUND} rounds, ${REPLICA.length} replicas, 8 reads + 2 writes + 1 batch request per round.`); console.log("schedule: k1 20-45 half-failed, k2 1-8 and 60-66 warming, k3 50 and 58-59 brief " + "unresponsive, 82-95 stopped, state store 100-110, billing store 115-125 down."); console.log(); console.log("endpoint content | out of pool | dropped reads | dropped writes | dropped batch | silent lost update | wasted reject"); console.log("-----------------|-------------|----------------|-----------------|----------------|---------------------|--------------"); for (const name of ["shallow", "readiness", "deepLocal", "deepShared", "deepAll"]) { const r = run(name); console.log(`${name.padEnd(17)} | ${s(r.outOfPoolRounds, 11)} | ${s(r.droppedReads, 14)} | ` + `${s(r.droppedWrites, 15)} | ${s(r.droppedBatch, 14)} | ${s(r.silent, 19)} | ${s(r.wastedReject, 12)}`); } console.log(); console.log("interval | checks | detection delay | false eviction | dropped reads | silent lost update"); console.log("---------|--------|------------------|-----------------|----------------|--------------------"); for (const interval of [1, 2, 4, 8]) { const r = run("deepLocal", { interval }); console.log(`${s(interval, 8)} | ${s(r.checks, 6)} | ${s(`${r.detection} round`, 16)} | ` + `${s(r.falseEviction, 15)} | ${s(r.droppedReads, 14)} | ${s(r.silent, 19)}`); } ``` ``` 130 rounds, 3 replicas, 8 reads + 2 writes + 1 batch request per round. schedule: k1 20-45 half-failed, k2 1-8 and 60-66 warming, k3 50 and 58-59 brief unresponsive, 82-95 stopped, state store 100-110, billing store 115-125 down. endpoint content | out of pool | dropped reads | dropped writes | dropped batch | silent lost update | wasted reject -----------------|-------------|----------------|-----------------|----------------|---------------------|-------------- shallow | 17 | 40 | 32 | 11 | 17 | 0 readiness | 32 | 0 | 22 | 11 | 17 | 0 deepLocal | 58 | 0 | 22 | 11 | 0 | 0 deepShared | 91 | 88 | 22 | 22 | 0 | 88 deepAll | 124 | 176 | 44 | 22 | 0 | 176 interval | checks | detection delay | false eviction | dropped reads | silent lost update ---------|--------|------------------|-----------------|----------------|-------------------- 1 | 390 | 0 round | 2 | 0 | 0 2 | 195 | 1 round | 1 | 11 | 0 4 | 99 | 3 round | 0 | 19 | 0 8 | 51 | 7 round | 0 | 40 | 3 ``` #### How Far Depth Should Go The first three rows show what depth gains. The shallow check kept the warming replica in the pool, dropping 40 reads and 32 writes routed to it; adding the readiness check brought dropped reads to 0 and dropped writes to 22. The remaining 22 writes occur in the eleven rounds the shared store is down, and no endpoint content rescues them — the replica may be healthy, but there is nowhere to write. The column's name comes from the Scaling the Data Layer course: a **silent lost update** is an accepted write invalidated without ever telling its owner. There the cause was conflict; here it is the half-failed replica — the measure is the same. The local deep check removes it: 17 to 0. The half-failed replica is now out of the pool, because the endpoint honestly answers "can I write". Its price shows in the out-of-pool column, 32 to 58 — the answer to the previous lesson's unsolved problem, at the sole cost of removing the broken replica from the pool. **The fourth row is where the sign flips.** When the shared state store joins the check, the moment it goes down all three replicas give the same answer and leave the pool at once. Once the pool is empty, not only writes but **reads** are rejected too: 88 dropped reads, all wasted rejections — a request rejected while at least one replica could have answered it, which is exactly what that column means. The edge cache and the read-from-replica path were up; the check shut them out too. Dropped batch work rose from 11 to 22 alongside. The fifth row repeats the mistake. When the store feeding only end-of-day billing joins the check, the replicas answering tracking queries also leave the pool once that store goes down: wasted rejects rise from 88 to 176 — no link existed between the billing flow's store and the tracking query's path, yet the check tied the two together. The rule reads from these three rows: **the endpoint should ask whether the replica is worse off than its neighbors, not whether the world is fine.** A condition that answers the same on every replica, once placed in the endpoint, empties the pool the moment it breaks — the spread comes from the check itself. This is why a shared dependency's failure is the circuit breaker's and graceful degradation's subject, measured in the previous topic, not the health endpoint's. #### How Often the Interval Should Run The second table runs the same endpoint content (`deepLocal`) at four check intervals. At interval 1, checks number 390, detection delay is 0 rounds, dropped reads 0 — but there are two **false evictions**: brief self-recovering outages evicted the replica from the pool. At interval 4, false evictions zero out, and in exchange the round `k3` stops is noticed 3 rounds later, dropping 19 reads routed to it in that span. At interval 8, delay rises to 7 rounds, dropped reads to 40, and the half-failed replica, left uninspected between checks, produces 3 silent lost updates. The trade-off has the same shape as the previous lesson's threshold trade-off but is a separate axis: the threshold sets how many consecutive checks must fail, the interval sets how infrequent checks are. The two multiply — interval 4 and threshold 3 mean a real failure can, in the worst case, go unnoticed for 12 rounds. #### The Numbers for Two Days The check's failure-free-day cost has two line items: the check load placed on the dependency, and the requests false eviction shifts onto the remaining replicas. The first is the deep check's direct bill. ```js // health/budget.mjs — the check's failure-free-day cost: the check load it puts on the // dependency and the requests false eviction shifts onto the remaining replicas const ROUND_SEC = 2; // KU1 (assumption): one round is one check interval, 2 seconds const STORE_REQ = 138.89; // K01: requests/s reaching the store const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const BRIEF_MONTH = 20; // KU3 (assumption): 20 transient unresponsive periods a month const b = (x, n = 2) => x.toFixed(n); console.log(`${"replicas".padStart(9)}${"interval".padStart(10)}${"checks/s".padStart(11)}` + `${"store load added".padStart(19)}${"checks/day".padStart(14)}`); for (const [replicas, interval] of [[3, 1], [3, 4], [12, 1], [12, 4], [48, 1], [48, 4]]) { const sec = interval * ROUND_SEC, rate = replicas / sec; console.log(`${String(replicas).padStart(9)}${(sec + " sec").padStart(10)}${b(rate).padStart(11)}` + `${("%" + b((100 * rate) / STORE_REQ)).padStart(19)}${Math.round(rate * 86_400).toLocaleString("en-US").padStart(14)}`); } console.log(`\nfalse eviction: when one of 3 replicas leaves the pool, the remaining two each carry ` + `${b(PEAK_EDGE / 2)} instead of ${b(PEAK_EDGE / 3)} req/s (${b((PEAK_EDGE / 2) / (PEAK_EDGE / 3))}x)`); for (const interval of [1, 2]) { const share = interval === 1 ? 2 / 3 : 1 / 3; // from the model: how many of the 3 brief outages were caught console.log(`interval ${interval}: ${b(BRIEF_MONTH * share, 1)} false evictions a month, each cutting ` + `capacity by ${b(100 / 3, 1)}% for up to ${interval * ROUND_SEC} sec`); } ``` ``` replicas interval checks/s store load added checks/day 3 2 sec 1.50 %1.08 129,600 3 8 sec 0.38 %0.27 32,400 12 2 sec 6.00 %4.32 518,400 12 8 sec 1.50 %1.08 129,600 48 2 sec 24.00 %17.28 2,073,600 48 8 sec 6.00 %4.32 518,400 false eviction: when one of 3 replicas leaves the pool, the remaining two each carry 256.94 instead of 171.30 req/s (1.50x) interval 1: 13.3 false evictions a month, each cutting capacity by 33.3% for up to 2 sec interval 2: 6.7 false evictions a month, each cutting capacity by 33.3% for up to 4 sec ``` With three replicas and a two-second interval, the deep check adds 1.50 queries a second to the store — 1.08 percent of K01's 138.89 req/s reaching it. The number grows linearly with replica count and runs into a limit there: at 48 replicas, the same interval means 17.28 percent added load. **The deep check's bill scales with replica count, not workload** — a scaling decision also changes the check policy. Multiplying the interval by four divides the bill by four. The second line item is false eviction. When one replica leaves a three-replica pool, the remaining two carry 256.94 req/s instead of 171.30, a 1.50x increase. At interval 1 the model catches two-thirds of the false evictions; with 20 transient unresponsive periods a month assumed (KU3, from the previous lesson), a third of capacity vanishes for two seconds 13.3 times a month. At interval 2 the count drops to 6.7, but each event's window doubles — the failure-free day's cost, paid with no failure at all. **The failing day's gain** sits between the first table's `shallow` and `deepLocal` rows: in the same failure schedule, 40 dropped reads fall to 0, 32 dropped writes to 22, 17 silent lost updates to 0, and wasted rejects stay at 0. The same table also shows where the gain ends: one step too much depth produces 88 wasted rejects, two steps too much produces 176. #### Summary - The health endpoint is not one question: the shallow check tests that the process is up, the readiness check that it can take traffic, the deep check that the dependency path works. - Adding the readiness check saved 40 reads and 10 writes routed to the warming replica; adding the local deep check brought silent lost updates from 17 to 0, at the cost of out-of-pool rounds rising from 32 to 58. - Adding a shared dependency to the check produces spread: when the store went down all three replicas left the pool, and 88 reads were rejected while answerable; adding the flow-specific store too raised this to 176. - The rule: the endpoint asks whether a replica is worse off than its neighbors; put a condition in it that answers the same on every replica, and the pool empties. - The check interval is a separate axis: interval 1 gives 2 false evictions and 0 rounds of delay, interval 8 gives 0 false evictions but 7 rounds of delay, 40 dropped reads, and 3 silent lost updates. - The failure-free day's cost is counted: at 3 replicas and a 2-second interval, the deep check adds 1.08 percent load to the store, 17.28 percent at 48 replicas; false eviction raises the remaining replicas' load to 1.50x, 13.3 times a month. #### Next Step These two lessons counted how long a failure takes to notice and to hand over: threshold, interval, promotion, and failback rounds — always the same quantity, **time to recovery**. But the writes lost in the previous lesson's half-failed window are a different quantity neither lesson measured: how much data is left behind when the system comes back? The two questions call for separate thresholds, both paid from the same budget. The next lesson names these thresholds, ties them to backup frequency and replication lag to turn them into numbers, and compares them against K01's 43.2-minute monthly budget. ### Recovery Objectives Source: https://academia.sh/en/courses/resilience-patterns/recovery-objectives Defining the recovery time objective and the data loss objective as two separate thresholds: converting the data loss objective into a number of lost events via the backup interval and replication lag, computing how many failures the recovery time objective fits into K01's monthly failure share, and counting the failure-free-day cost of driving data loss to zero. The previous two lessons counted how long a failure takes to notice and to hand over: threshold, interval, promotion, and failback rounds. What they measured was always the same quantity, the time for the service to come back. But the 3,888.80 writes lost in the first lesson's half-failed window are a different quantity that neither lesson measured: how much data is left behind when the system comes back? This lesson names two thresholds and separates them. The **recovery time objective** is the longest time allowed to pass from a failure's start to the service's return. The **data loss objective** is the most data loss accepted after recovery, written as a duration: "at most one minute of writes". Both are **thresholds**. The `time to recovery` from the Introduction to System Design course is a **measurement** — how long a failure that actually happened took; the objective here is the permitted upper bound, set in advance as a design decision. The same separation holds for service availability: the measured percentage and the targeted percentage are separate numbers, and one does not validate the other. #### Two Thresholds Held by Separate Mechanisms The recovery time objective is held by **failover**: the sum of detection, promotion, and routing rounds must stay under the target. The data loss objective is held by **data replication**: it determines how close the recovered replica's data is to the moment of failure. A system can fail over fast and lose a lot of data, or lose no data and recover slowly. A recovery claim only carries meaning with both numbers together: "failover completed in eight seconds" is incomplete without "and how many events were lost". Two inputs convert the data loss objective into a number. The **backup interval**: in the worst case, the failure happens right before the next backup, and a full interval's worth of writes is lost. **Replication lag**: under asynchronous replication, writes that left the source but never reached the target are lost. Replication's mechanics were established in the Scaling the Data Layer course and are not repeated here; the lag is only an input here. ```js // objective/loss.mjs — tying the data loss objective to the number of lost events via the backup // interval and replication lag. All arithmetic; the inputs are K01's computed values and this // lesson's assumptions. const DAY = 86_400; const EVENTS_DAY = 2_800_000; // K01: daily state events (V3 x V4 = 400,000 x 7) const EVENTS_SEC = EVENTS_DAY / DAY; // average write rate const PEAK = 97.22; // K01: peak write requests/s const EVENTS_SHIPMENT = 7; // K01: state events per shipment const GROWTH_GB = 0.976; // K01: daily data growth (976 MB) const FULL_GB = 712.48; // K01: stored data const b = (x, n = 2) => x.toFixed(n); console.log(`average write ${b(EVENTS_SEC)} events/s, peak ${PEAK} events/s`); console.log(`${"backup interval".padStart(18)}${"worst-case loss".padStart(17)}` + `${"lost events (avg)".padStart(19)}${"lost events (peak)".padStart(20)}${"lost shipments".padStart(16)}`); for (const [name, sec] of [["24 hr", 86_400], ["6 hr", 21_600], ["1 hr", 3600], ["15 min", 900], ["5 min", 300], ["1 min", 60]]) { const avg = EVENTS_SEC * sec, peak = PEAK * sec; console.log(`${name.padStart(18)}${(sec + " sec").padStart(17)}` + `${Math.round(avg).toLocaleString("en-US").padStart(19)}` + `${Math.round(peak).toLocaleString("en-US").padStart(20)}` + `${Math.round(avg / EVENTS_SHIPMENT).toLocaleString("en-US").padStart(16)}`); } console.log(`\n${"replication lag".padStart(19)}${"data loss objective".padStart(21)}` + `${"lost events (peak)".padStart(20)}${"lost shipments".padStart(16)}`); for (const L of [0, 0.5, 1, 5, 10]) { console.log(`${(b(L, 1) + " sec").padStart(19)}${(b(L, 1) + " sec").padStart(21)}` + `${b(PEAK * L).padStart(20)}${b((PEAK * L) / EVENTS_SHIPMENT).padStart(16)}`); } console.log(`\n${"backup interval".padStart(18)}${"backups/day".padStart(13)}` + `${"backups/30 days".padStart(17)}${"incremental GB/30 days".padStart(24)}${"full backup GB (weekly)".padStart(25)}`); for (const [name, sec] of [["24 hr", 86_400], ["1 hr", 3600], ["5 min", 300]]) { const perDay = DAY / sec; console.log(`${name.padStart(18)}${b(perDay, 0).padStart(13)}${b(perDay * 30, 0).padStart(17)}` + `${b(GROWTH_GB * 30).padStart(24)}${b(FULL_GB * 5).padStart(25)}`); } console.log(`stored backup data does not change with the interval: only the backup operation count changes`); ``` ``` average write 32.41 events/s, peak 97.22 events/s backup interval worst-case loss lost events (avg) lost events (peak) lost shipments 24 hr 86400 sec 2,800,000 8,399,808 400,000 6 hr 21600 sec 700,000 2,099,952 100,000 1 hr 3600 sec 116,667 349,992 16,667 15 min 900 sec 29,167 87,498 4,167 5 min 300 sec 9,722 29,166 1,389 1 min 60 sec 1,944 5,833 278 replication lag data loss objective lost events (peak) lost shipments 0.0 sec 0.0 sec 0.00 0.00 0.5 sec 0.5 sec 48.61 6.94 1.0 sec 1.0 sec 97.22 13.89 5.0 sec 5.0 sec 486.10 69.44 10.0 sec 10.0 sec 972.20 138.89 backup interval backups/day backups/30 days incremental GB/30 days full backup GB (weekly) 24 hr 1 30 29.28 3562.40 1 hr 24 720 29.28 3562.40 5 min 288 8640 29.28 3562.40 stored backup data does not change with the interval: only the backup operation count changes ``` #### Backups Alone Cannot Hold an Objective The first table shows a difference of orders of magnitude. Daily backups lose 2,800,000 state events in the worst case — a full day's shipment traffic; hourly backups lose 116,667 events and 16,667 shipments. Even at five minutes, 9,722 events are lost. These numbers reflect not a shortcoming but a **definition**: a backup is a copy of specific moments, and everything between two moments falls outside the copy. The second table gives the same quantity on the replication side, and the comparison is stark: a one-second lag loses 97.22 events at peak load — 300 times less than a five-minute backup interval's loss. **In a stream taking continuous writes, a meaningful data loss objective is held by replication, not backups**; a backup's job is covering a different failure mode — data overwritten with the wrong content, a case replication faithfully copies. The third table shows where tightening the backup interval costs nothing. With incremental backups, the data stored over thirty days is 29.28 GB at all three intervals, because what is stored is the change itself, independent of the interval. The 3,562.40 GB of weekly full backups is fixed too. The only number that changes is the backup operation count: 1 to 288 a day. Tightening the interval bills operation count and coordination, not storage. #### The Recovery Time Objective Looks at the Budget The recovery time objective cannot be chosen alone, because the objective and the monthly outage budget share the same resource. K01 established the budget: the 99.9% target gives 43.2 minutes a month, 15.0 minutes of which go to planned work, the remaining 28.2 to failure. The percentage arithmetic is not repeated here; these three numbers are inputs. ```js // objective/budget.mjs — comparing the recovery time objective against K01's monthly outage // budget, and the failure-free-day cost of driving the data loss objective to zero const FAILURE_SHARE = 28.2; // K01: the outage budget's monthly failure share (minutes) const MONTHLY = 43.2; // K01: monthly outage budget for 99.9% (minutes) const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PEAK_WRITE = 97.22; // K01: peak write requests/s const EVENTS_DAY = 2_800_000; // K01: daily state events const b = (x, n = 2) => x.toFixed(n); console.log(`${"recovery time objective".padStart(24)}${"fits in failure share".padStart(23)}` + `${"one failure's budget share".padStart(28)}${"dropped edge requests in window".padStart(33)}`); for (const [name, min] of [["8 sec (failover)", 8 / 60], ["5 min", 5], ["10 min", 10], ["30 min", 30], ["60 min", 60]]) { console.log(`${name.padStart(24)}${b(FAILURE_SHARE / min).padStart(23)}` + `${("%" + b((100 * min) / MONTHLY)).padStart(28)}` + `${Math.round(min * 60 * PEAK_EDGE).toLocaleString("en-US").padStart(33)}`); } console.log(`K01's recovery time assumption was 10 min and fit 2.82 failures into the failure share`); console.log(`\ncost of driving the data loss objective to zero (synchronous acknowledgment):`); console.log(` every write waits for one extra acknowledgment round -> ${EVENTS_DAY.toLocaleString("en-US")} extra rounds a day`); const WINDOW = 8; // lesson 01: failover window at threshold 3 (seconds) const STOPS_MONTH = 2; // KU2 (assumption): 2 real stops a month, per replica const LAG = 1; // KU6 (assumption): asynchronous replication lag, 1 second const sync = STOPS_MONTH * WINDOW * PEAK_WRITE; const async_ = STOPS_MONTH * LAG * PEAK_WRITE; console.log(` the write path binds to two replicas at once: in the standby replica's ${WINDOW}-sec window`); console.log(` ${b(sync)} writes visibly drop a month (the carrier can resend)`); console.log(` under asynchronous replication, ${b(async_)} events a month are silently lost (unrecoverable)`); console.log(` ratio ${b(sync / async_)}: the synchronous choice rejects more requests, loses less data`); ``` ``` recovery time objective fits in failure share one failure's budget share dropped edge requests in window 8 sec (failover) 211.50 %0.31 4,111 5 min 5.64 %11.57 154,167 10 min 2.82 %23.15 308,334 30 min 0.94 %69.44 925,002 60 min 0.47 %138.89 1,850,004 K01's recovery time assumption was 10 min and fit 2.82 failures into the failure share cost of driving the data loss objective to zero (synchronous acknowledgment): every write waits for one extra acknowledgment round -> 2,800,000 extra rounds a day the write path binds to two replicas at once: in the standby replica's 8-sec window 1555.52 writes visibly drop a month (the carrier can resend) under asynchronous replication, 194.44 events a month are silently lost (unrecoverable) ratio 8.00: the synchronous choice rejects more requests, loses less data ``` The table shows how the objective speaks to the budget. A ten-minute recovery time objective fits 2.82 failures into the failure share — K01's own number, placed beside the other objectives, not recalculated. A thirty-minute objective means 0.94 failures: **a single failure eats 69.44 percent of the monthly budget, and a second breaks the target.** At sixty minutes, one failure already holds 138.89 percent — the target is broken by a single failure. A recovery time objective cannot be chosen independent of the availability target; both are bound to the same budget. The top row puts the previous two lessons' gain on this scale. The eight-second window measured with failover fits 211.50 failures into the failure share, and a single failure holds only 0.31 percent of the budget. Automatic failover is therefore not a speed improvement but making the target **achievable**: manual recovery already breaks the target at three failures a month, while failover fits two hundred. #### The Numbers for Two Days The path to a zero data loss objective is synchronous acknowledgment: a write does not count as accepted until a second replica confirms it. **The failure-free day's cost** has two line items. The first is every write waiting for one extra acknowledgment round — 2,800,000 extra rounds a day, with no failure at all. The second, and sharper, is the write path binding to two replicas at once: now the **standby** replica's failure stops writes too. If the standby replica's eight-second window happens twice a month, 1,555.52 writes are rejected. **The failing day's gain** is in the same block's last two lines. Under the asynchronous setup, a one-second lag silently loses 194.44 state events a month, unrecoverable. The synchronous setup zeroes this out. The ratio is 8.00: the synchronous choice openly rejects eight requests for every one event it would otherwise lose. This trade holds up for the shipment tracking service, because a rejected status update can be resent by the carrier, while a lost one cannot be recovered by anyone. This is exactly where K01 split the two patterns: the same failure can be paid either as outage or as lost data. The decision is made per flow. A tracking query is a read, and the data loss objective does not concern it. A zero data loss objective is defensible for the state event. End-of-day billing, though, runs in a four-hour window and can be restarted; both its recovery time objective and its data loss objective are loose — as long as the state events feeding it are complete. #### Summary - The recovery time objective and the data loss objective are separate thresholds held by separate mechanisms: the first by failover, the second by data replication; a recovery claim is incomplete without both numbers. Both are thresholds, not measurements: K01's `time to recovery` is a failure's actual duration, the objective here is the permitted upper bound. - The backup interval directly determines data loss: daily backups lose up to 2,800,000 events in the worst case, hourly 116,667, five-minute 9,722. - A one-second replication lag loses 97.22 events — one three-hundredth of a five-minute backup interval's loss; in a stream taking continuous writes, the data loss objective is held by replication. - Tightening the backup interval does not bill storage: the incremental data stored over thirty days is 29.28 GB at all three intervals; the only number that changes is the backup operation count, rising from 1 to 288 a day. - The recovery time objective is bound to the budget: a 30-minute objective fits 0.94 failures into the failure share (one failure is 69.44 percent of the budget), while the 8-second window measured with failover fits 211.50. - Zeroing out data loss costs the failure-free day 2,800,000 extra acknowledgment rounds a day and 1,555.52 rejected writes a month; in exchange, 194.44 permanent losses a month disappear (ratio 8.00). #### Next Step Every calculation in this lesson carried one implicit assumption: there is a replica to recover, and that replica was untouched by the failure. The assumption fails if two replicas sit on the same rack, the same power line, or behind the same network switch — this is why K01's replica arithmetic was an upper bound, with the assumption that "replicas fail independently" explicitly flagged there. The next lesson takes on **where** replicas are placed: the placement unit's definition, the capacity that survives losing one unit, the unpropagated write that cross-unit replication loses, and how much moving redundancy up one level costs on a failure-free day. ### Redundancy Zones Source: https://academia.sh/en/courses/resilience-patterns/redundancy-zones Measuring where copies are placed: separating the zone and the region as placement units, comparing three placement plans by surviving capacity and lost unpropagated writes under machine, zone, and region loss, computing the capacity that must sit idle to withstand the loss of one unit, and counting the overhead that cross-region replication carries on a failure-free day. Every calculation in the previous lesson carried one implicit assumption: there is a replica to recover, and that replica was untouched by the failure. K01 had already flagged this assumption — replica arithmetic only holds once replicas fail independently, which is why the computed number is an upper bound. If two replicas share the same power line, the same network switch, or the same cooling unit, there is no independence: when one goes down, the other goes down too. This lesson measures **where** replicas are placed. Two placement units are separated. An **availability zone** is a placement unit with its own power and network path, one whose failure is assumed not to cross into its neighbor. A **region** is a geographically separate set of zones; the distance between two regions lengthens the propagation round. In this course, `zone` refers only to this physical placement unit. For the same reason, the set of components a failure touches is called **blast radius** in this course. #### A Placement Plan Selects a Failure Mode A placement plan determines which replicas fail **together**. If six replicas sit in a single zone, the loss of that zone takes all six at once and the six replicas mean nothing. If the same six replicas are spread across three zones, the loss of one zone takes two. If they are spread across two regions, the loss of one region takes three. The setup below is a **model**: no real region, zone, cloud environment, or container is set up. Replicas are objects carrying a region and zone tag, a round is an abstract step, and the propagation round changes with distance. The mechanics of replication were established in the Scaling the Data Layer course and are not repeated here; the only question here is **which write ends up nowhere** when a unit is lost. **KU7 — propagation rounds: 1 within the same zone, 2 between different zones in the same region, 4 between regions.** Rationale: propagation time grows with distance. Sensitivity is linear; if the remote round doubles, the unpropagated write count at region loss also doubles. This assumption is not added to K01's table. ```js // zone/model.mjs — placement is a MODEL: replicas carry a region and zone tag, propagation rounds // change with distance. A round is an abstract step; no real region, zone, or cloud environment is set up. export const ROUND = 40, FAILURE_ROUND = 20, WRITES_PER_ROUND = 6; export const PROPAGATION = { zone: 1, region: 2, remote: 4 }; // KU7 (assumption): propagation rounds export const PLAN = { "single-zone": [["r1", "z1"], ["r1", "z1"], ["r1", "z1"], ["r1", "z1"], ["r1", "z1"], ["r1", "z1"]], "three-zone": [["r1", "z1"], ["r1", "z1"], ["r1", "z2"], ["r1", "z2"], ["r1", "z3"], ["r1", "z3"]], "two-region": [["r1", "z1"], ["r1", "z2"], ["r1", "z3"], ["r2", "z1"], ["r2", "z2"], ["r2", "z3"]], }; export const SCENARIO = { machine: (y, i) => i === 0, zone: (y) => y[0] === "r1" && y[1] === "z1", region: (y) => y[0] === "r1", }; const delay = (x, y) => (x[0] !== y[0] ? PROPAGATION.remote : x[1] !== y[1] ? PROPAGATION.region : PROPAGATION.zone); export function run(planName, scenarioName) { const placement = PLAN[planName], n = placement.length; const down = placement.map((y, i) => SCENARIO[scenarioName](y, i)); const alive = (i, t) => !(down[i] && t >= FAILURE_ROUND); const writes = []; let cursor = 0, s = { accepted: 0, dropped: 0, lost: 0 }; for (let t = 1; t <= ROUND; t += 1) { for (const w of writes) { // propagations that arrived become visible at the target for (const m of w.inFlight) if (m.arrives === t) w.owner.add(m.target); w.inFlight = w.inFlight.filter((m) => m.arrives > t); } if (t === FAILURE_ROUND) { // the failed unit: loss of ownership and in-flight messages for (const w of writes) { w.inFlight = w.inFlight.filter((m) => alive(m.source, t) && alive(m.target, t)); const remaining = [...w.owner].filter((i) => alive(i, t)); if (remaining.length === 0 && w.inFlight.length === 0) { s.lost += 1; w.lost = true; } w.owner = new Set(remaining); } } const pool = placement.map((_, i) => i).filter((i) => alive(i, t)); for (let k = 0; k < WRITES_PER_ROUND; k += 1) { if (pool.length === 0) { s.dropped += 1; continue; } const i = pool[cursor++ % pool.length]; s.accepted += 1; const w = { owner: new Set([i]), inFlight: [], lost: false }; for (const j of pool) if (j !== i) w.inFlight.push({ source: i, target: j, arrives: t + delay(placement[i], placement[j]) }); writes.push(w); } } const survived = placement.filter((_, i) => alive(i, ROUND)).length; return { ...s, survived, total: n }; } ``` ```js // zone/measure.mjs — three placement plans, three failures: surviving replicas, dropped writes, lost writes import { run, PLAN, SCENARIO, ROUND, FAILURE_ROUND, WRITES_PER_ROUND, PROPAGATION } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); console.log(`${ROUND} rounds, ${WRITES_PER_ROUND} writes per round, failure at round ${FAILURE_ROUND}.`); console.log(`propagation rounds: same zone ${PROPAGATION.zone}, same region different zone ${PROPAGATION.region}, ` + `different region ${PROPAGATION.remote} (KU7)`); console.log(); console.log(`${"plan".padEnd(12)} | ${"failure".padEnd(7)} | ${s("surviving replicas", 19)} | ${s("surviving capacity", 19)} | ${s("dropped writes", 15)} | ${s("lost unpropagated", 18)}`); console.log(`${"-".repeat(12)}-|-${"-".repeat(7)}-|-${"-".repeat(19)}-|-${"-".repeat(19)}-|-${"-".repeat(15)}-|-${"-".repeat(18)}`); for (const plan of Object.keys(PLAN)) { for (const scenario of Object.keys(SCENARIO)) { const r = run(plan, scenario); console.log(`${plan.padEnd(12)} | ${scenario.padEnd(7)} | ${s(`${r.survived}/${r.total}`, 19)} | ` + `${s(`${((100 * r.survived) / r.total).toFixed(1)}%`, 19)} | ${s(r.dropped, 15)} | ${s(r.lost, 18)}`); } } ``` ``` 40 rounds, 6 writes per round, failure at round 20. propagation rounds: same zone 1, same region different zone 2, different region 4 (KU7) plan | failure | surviving replicas | surviving capacity | dropped writes | lost unpropagated -------------|---------|---------------------|---------------------|-----------------|------------------- single-zone | machine | 5/6 | 83.3% | 0 | 0 single-zone | zone | 0/6 | 0.0% | 126 | 114 single-zone | region | 0/6 | 0.0% | 126 | 114 three-zone | machine | 5/6 | 83.3% | 0 | 0 three-zone | zone | 4/6 | 66.7% | 0 | 2 three-zone | region | 0/6 | 0.0% | 126 | 114 two-region | machine | 5/6 | 83.3% | 0 | 1 two-region | zone | 5/6 | 83.3% | 0 | 1 two-region | region | 3/6 | 50.0% | 0 | 9 ``` #### Reading the Three Plans The machine rows are the same across all three plans: the loss of one replica does not stop the service in any arrangement. The replica count already covered this failure mode, and the placement plan changes nothing here. **The zone row separates the plans.** In the single-zone arrangement, the loss of the zone takes all six replicas: 126 writes drop and 114 writes end up nowhere. The same failure leaves four replicas standing in the three-zone arrangement, no writes drop, and only 2 writes are lost. All six replicas existed in both cases; the only difference was where they were placed. **The region row separates a second level.** The three-zone arrangement covers zone loss, but region loss takes it entirely down too — all three zones sit in the same region. Only the two-region arrangement survives region loss: half capacity, zero dropped writes, 9 lost unpropagated writes. **The cost of spreading out shows up in the last column.** In the two-region arrangement, even a single machine's loss costs 1 write, while the single-zone arrangement costs 0. The reason is the propagation round: when replicas sit farther apart, a write takes longer to reach a second replica, and more writes sit in a single replica during that window. **Longer propagation is redundancy's other face**; the same count climbs to 9 at region loss. #### How Much Capacity Must Sit Idle The surviving capacity column leaves a question open: is 66.7 percent capacity enough to carry the load? For it to be enough, part of the capacity must sit idle on a normal day. The calculation below gives that share as a function of zone count and counts the load that cross-region replication carries on a failure-free day. ```js // zone/capacity.mjs — the failure-free-day cost of redundancy: capacity sitting idle and writes // carried across regions; converting model rounds to K01 scale const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PEAK_WRITE = 97.22; // K01: peak write requests/s const EVENTS_DAY = 2_800_000; // K01: daily state events const GROWTH_GB = 0.976; // K01: daily data growth const ROUND_SEC = 2; // KU1 (assumption): one round is 2 seconds const b = (x, n = 2) => x.toFixed(n); console.log(`${"zones".padStart(5)}${"load per zone".padStart(17)}${"if one zone drops".padStart(18)}` + `${"must be provisioned".padStart(20)}${"sitting idle".padStart(13)}${"extra share".padStart(14)}`); for (const n of [2, 3, 4, 6]) { const normal = PEAK_EDGE / n, dropped = PEAK_EDGE / (n - 1), total = n * dropped; console.log(`${String(n).padStart(5)}${b(normal).padStart(17)}${b(dropped).padStart(18)}` + `${b(total).padStart(20)}${b(total - PEAK_EDGE).padStart(13)}${(b(100 * (total / PEAK_EDGE - 1), 1) + "%").padStart(14)}`); } const bytes = (GROWTH_GB * 1e9) / EVENTS_DAY; console.log(`\n${b(bytes)} bytes per event (K01: ${GROWTH_GB} GB / ${EVENTS_DAY.toLocaleString("en-US")} events a day)`); console.log(`replicating to two regions: ${b(PEAK_WRITE * bytes / 1024)} KB/s extra transfer at peak load, ` + `${b(GROWTH_GB * 1000)} MB a day`); console.log(`\nconverting lost unpropagated writes to K01 scale (KU1: round = ${ROUND_SEC} sec):`); for (const [name, round, share] of [["zone loss (3 zones)", 1, 3], ["region loss (2 regions)", 4, 2]]) { console.log(` ${name.padEnd(24)} propagation ${round} round(s) = ${round * ROUND_SEC} sec, ` + `${b(PEAK_WRITE / share)} writes/s per unit -> ${b((round * ROUND_SEC * PEAK_WRITE) / share)} writes`); } console.log(` ratio ${b(((4 * ROUND_SEC * PEAK_WRITE) / 2) / ((1 * ROUND_SEC * PEAK_WRITE) / 3))}: ` + `region loss leaves more unpropagated writes in the same failure duration`); ``` ``` zones load per zone if one zone drops must be provisioned sitting idle extra share 2 256.94 513.89 1027.78 513.89 100.0% 3 171.30 256.94 770.84 256.95 50.0% 4 128.47 171.30 685.19 171.30 33.3% 6 85.65 102.78 616.67 102.78 20.0% 348.57 bytes per event (K01: 0.976 GB / 2,800,000 events a day) replicating to two regions: 33.09 KB/s extra transfer at peak load, 976.00 MB a day converting lost unpropagated writes to K01 scale (KU1: round = 2 sec): zone loss (3 zones) propagation 1 round(s) = 2 sec, 32.41 writes/s per unit -> 64.81 writes region loss (2 regions) propagation 4 round(s) = 8 sec, 48.61 writes/s per unit -> 388.88 writes ratio 6.00: region loss leaves more unpropagated writes in the same failure duration ``` Idle capacity shrinks quickly with zone count. Withstanding the loss of one zone with two zones requires double the capacity: 1027.78 is provisioned to carry 513.89 requests/s, and half of it sits idle. The extra share drops to 50 percent with three zones, 33.3 percent with four zones, and 20 percent with six zones. The general form for $n$ zones is $1/(n-1)$. **Increasing the zone count makes redundancy cheaper**, which means the same service availability can be reached at two different prices. The transfer bill for cross-region replication, however, is small: 348.57 bytes per event, 33.09 KB/s at peak load, 976 MB a day. For this workload, the bill for region redundancy is not in the transfer but in the capacity and the lost writes. Under a different load — a workload carrying writes with large bodies — the ranking would reverse; the number depends on the workload, not on the principle. The last block converts model rounds to K01 scale. Unpropagated writes at the loss of one zone in the three-zone arrangement come to 64.81; at the loss of one region in the two-region arrangement, 388.88. The ratio is 6.00 and comes from two causes: cross-region propagation takes four times as long, and the per-region write share is larger. **Region redundancy removes the outage but grows the data loss**, and this is exactly the number to compare against the previous lesson's data loss objective: a one-second replication lag lost 97.22 events, region loss loses 388.88. #### The Numbers for Two Days **The failure-free day's cost** is three line items. In a three-zone placement, 50 percent of capacity sits idle — 256.95 requests/s never get used. Two-region replication carries every write a second time: 976 MB a day and 33.09 KB/s at peak load. Third, longer propagation produces loss even at small failures: at a single machine's loss, the two-region arrangement lost 1 write while the single-zone arrangement lost 0. **The failing day's gain** is in the same table's zone and region rows. In the single-zone arrangement, the loss of one zone drops 126 writes and loses 114, while the same failure in the three-zone arrangement gives 0 dropped writes and 2 lost. At region loss, the three-zone arrangement gives 126/114 while the two-region arrangement gives 0/9. Each level removes one failure mode and leaves the next one open; the placement decision is choosing which mode gets removed. #### Summary - A zone is a placement unit with its own power and network path; a region is a geographically separate set of zones. In this course, `zone` carries only this meaning. - A placement plan determines which replicas fail together: when six replicas are pooled in a single zone, zone loss drops 126 writes and loses 114; spread across three zones, the same failure gives 0 dropped writes and 2 lost. - Region loss is survived only by the two-region arrangement: 50 percent capacity, 0 dropped writes, 9 lost unpropagated writes. - Spreading out has its own cost in longer propagation rounds: the two-region arrangement loses 1 write even at a single machine's loss, the single-zone arrangement loses 0. - The capacity that must sit idle to withstand the loss of one zone is $1/(n-1)$: 100 percent with two zones, 50 percent with three zones, 20 percent with six zones. - At K01 scale, zone loss leaves 64.81 and region loss leaves 388.88 unpropagated writes (ratio 6.00); region redundancy removes the outage but forces the data loss objective. #### Next Step This lesson spread replicas out, but what got spread was always the same whole: six replicas, one service, one dataset. Surviving capacity at zone or region loss came out to 66.7 percent and 50 percent, because the lost unit carried a **fraction** of the service. This fraction arithmetic assumes something: the units can do work independently of each other. In most arrangements, though, the units keep something in common — a single routing map, a single number dispenser, a single configuration source — and the loss of that point takes all the units at once. The next lesson takes on splitting the service into self-contained, replicable units: keeping the links inside a unit from crossing it, the load per unit, and the blast radius of the single resource shared across units. ### Deployment Stamps and Geo-Replicas Source: https://academia.sh/en/courses/resilience-patterns/deployment-stamps-and-geo-replicas Splitting the service into self-contained, replicable units: counting the links that cross the stamp boundary, comparing each component's blast radius across three placements, showing that the scope of the single resource shared across stamps stays independent of stamp count, and converting the load per stamp into K01 numbers. The previous lesson spread replicas across zones and regions, but what got spread was always the same whole: one service, one dataset. Surviving capacity coming out to 66.7 percent or 50 percent at the loss of one unit assumed that the lost unit carried a **fraction** of the service. In most arrangements this assumption does not hold: the units keep something in common, and the loss of that point takes all of them at once. A **deployment stamp** is a self-contained, replicable copy of the service: the application, its store, its cache, and its queue together. The same structure is called a **scale unit** when it is used as a scaling unit — adding capacity means opening a new stamp. The term is not confused with the `version stamp` from the Application Layer and Service Interaction course; that is a marker showing which version a record was written in, while this is a deployment unit. #### What the Stamp Boundary Is A stamp is self-contained only if no link crosses its boundary. Every link that crosses the boundary requires something outside the stamp to stay up, and when that thing goes down, the stamp's independence ends. This is why this lesson's first measure is **the number of links crossing the stamp**. The setup below is a **model**: nine components, three flows, and three placements. Components are tagged as stamp-local or shared across stamps; a flow is the list of components it touches in order. No real cluster, region, or container is set up. ```js // stamp/model.mjs — a deployment stamp is a MODEL: components are tagged as stamp-local or // shared across stamps, flows are component arrays. No real cluster is set up. export const STAMPS = 4; export const FLOW = { // components each flow touches, in order tracking: ["routing", "edge", "tracking-service", "cache", "state-store"], event: ["routing", "edge", "event-queue", "state-store", "number-dispenser"], billing: ["billing-job", "state-store", "tariff-store"], }; export const LOAD = { tracking: 8, event: 2, billing: 1 }; // requests per round, close to K01's ratio const ALL = [...new Set(Object.values(FLOW).flat())]; export const PLACEMENT = { // anything not marked shared is stamp-local "single-piece": ALL, "shared-stamp": ["routing", "cache", "number-dispenser", "tariff-store"], "self-contained": ["routing"], }; export const COMPONENTS = ALL; export const scope = (placement, c) => (PLACEMENT[placement].includes(c) ? "shared" : "stamp"); export function stampCount(placement) { // the single-piece arrangement has no stamp concept return placement === "single-piece" ? 1 : STAMPS; } export function crossingLinks(placement) { // how many times the stamp boundary is crossed let n = 0; for (const path of Object.values(FLOW)) for (let i = 0; i + 1 < path.length; i += 1) if (scope(placement, path[i]) !== scope(placement, path[i + 1])) n += 1; return n; } export function blastRadius(placement, downComponent) { // requests and stamps affected when one component drops const d = stampCount(placement); const total = Object.values(LOAD).reduce((a, b) => a + b, 0) * d; let affected = 0; for (const [name, path] of Object.entries(FLOW)) { if (!path.includes(downComponent)) continue; affected += scope(placement, downComponent) === "shared" ? LOAD[name] * d : LOAD[name]; } const stamps = scope(placement, downComponent) === "shared" ? d : 1; return { affected, total, ratio: affected / total, stamps, d }; } ``` ```js // stamp/measure.mjs — crossing links in three placements and each component's blast radius import { FLOW, PLACEMENT, COMPONENTS, crossingLinks, blastRadius, stampCount, scope } from "./model.mjs"; const s = (x, n) => String(x).padStart(n); console.log(`${COMPONENTS.length} components, ${Object.keys(FLOW).length} flows, ` + `8 tracking + 2 event + 1 billing request per stamp.`); console.log(); console.log(`${"placement".padEnd(15)} | ${s("stamps", 6)} | ${s("shared components", 18)} | ${s("stamp-local", 11)} | ${s("crossing links", 14)}`); console.log(`${"-".repeat(15)}-|-${"-".repeat(6)}-|-${"-".repeat(18)}-|-${"-".repeat(11)}-|-${"-".repeat(14)}`); for (const p of Object.keys(PLACEMENT)) { const shared = COMPONENTS.filter((c) => scope(p, c) === "shared").length; console.log(`${p.padEnd(15)} | ${s(stampCount(p), 6)} | ${s(shared, 18)} | ` + `${s(COMPONENTS.length - shared, 11)} | ${s(crossingLinks(p), 14)}`); } console.log(); console.log(`${"component down".padEnd(16)} | ${s("single-piece", 15)} | ${s("shared-stamp", 16)} | ${s("self-contained", 14)}`); console.log(`${"-".repeat(16)}-|-${"-".repeat(15)}-|-${"-".repeat(16)}-|-${"-".repeat(14)}`); for (const c of COMPONENTS) { const cell = Object.keys(PLACEMENT).map((p) => { const r = blastRadius(p, c); return `${(100 * r.ratio).toFixed(1)}% (${r.stamps}/${r.d})`; }); console.log(`${c.padEnd(16)} | ${s(cell[0], 15)} | ${s(cell[1], 16)} | ${s(cell[2], 14)}`); } ``` ``` 9 components, 3 flows, 8 tracking + 2 event + 1 billing request per stamp. placement | stamps | shared components | stamp-local | crossing links ----------------|--------|--------------------|-------------|--------------- single-piece | 1 | 9 | 0 | 0 shared-stamp | 4 | 4 | 5 | 6 self-contained | 4 | 1 | 8 | 2 component down | single-piece | shared-stamp | self-contained -----------------|-----------------|------------------|--------------- routing | 90.9% (1/1) | 90.9% (4/4) | 90.9% (4/4) edge | 90.9% (1/1) | 22.7% (1/4) | 22.7% (1/4) tracking-service | 72.7% (1/1) | 18.2% (1/4) | 18.2% (1/4) cache | 72.7% (1/1) | 72.7% (4/4) | 18.2% (1/4) state-store | 100.0% (1/1) | 25.0% (1/4) | 25.0% (1/4) event-queue | 18.2% (1/1) | 4.5% (1/4) | 4.5% (1/4) number-dispenser | 18.2% (1/1) | 18.2% (4/4) | 4.5% (1/4) billing-job | 9.1% (1/1) | 2.3% (1/4) | 2.3% (1/4) tariff-store | 9.1% (1/1) | 9.1% (4/4) | 2.3% (1/4) ``` #### What the Blast Radius Is a Result Of The first table gives stamping's measure on paper. The single-piece arrangement has no stamp boundary, so it has no crossing link either — but it also has no independent unit. When four stamps are opened, the shared component count drops to four and the crossing link count becomes 6; once the shared components are moved into the stamp, the crossing link count drops to 2. **Stamp count does not give independence; shared component count does.** The second table counts the counterpart of this for each component separately. For stamp-local components, the scope is divided by four: the edge component's loss goes from 90.9 percent to 22.7 percent, the state store's loss goes from 100 percent to 25 percent, and the affected stamp count drops from 4 to 1. This column is the pattern's promise. For shared components, stamping does nothing. When the cache stays shared, its loss is the same as the single-piece arrangement: 72.7 percent and 4/4 stamps. Moved into the stamp, it becomes 18.2 percent and 1/4. The same pattern holds for the number dispenser (18.2 percent to 4.5 percent) and the tariff store (9.1 percent to 2.3 percent). **As long as a component stays outside the stamp, increasing stamp count does not change its blast radius.** The routing map row is the same across all three columns: 90.9 percent and 4/4. This is the single source that knows which shipment is in which stamp, and it cannot be moved into a stamp — if it were, there would be no way to know which stamp a request should go to. This is the pattern's limit: **stamping does not zero out the blast radius, it reduces it to the single point that remains shared.** That point's service availability is the upper bound for the whole system, and K01's weakest-link rule applies here without modification. #### What Stamp Count Changes ```js // stamp/load.mjs — converting the stamp count into K01 loads and the failure-free day's cost const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PEAK_WRITE = 97.22; // K01: peak write requests/s const STORE_REQ = 138.89; // K01: requests/s reaching the store const SHIPMENTS_DAY = 400_000; // K01: daily shipments const STAMP_LOCAL = 8; // from the model: stamp-local component count in the self-contained placement const b = (x, n = 2) => x.toFixed(n); console.log(`${"stamps".padStart(6)}${"edge per stamp".padStart(17)}${"store per stamp".padStart(19)}` + `${"writes per stamp".padStart(20)}${"blast radius".padStart(17)}${"daily shipments".padStart(16)}`); for (const n of [1, 2, 4, 8, 16]) { console.log(`${String(n).padStart(6)}${b(PEAK_EDGE / n).padStart(17)}${b(STORE_REQ / n).padStart(19)}` + `${b(PEAK_WRITE / n).padStart(20)}${(b(100 / n) + "%").padStart(17)}` + `${(SHIPMENTS_DAY / n).toLocaleString("en-US").padStart(16)}`); } console.log(`\n${"stamps".padStart(7)}${"total components".padStart(18)}${"with geo-replicas".padStart(19)}` + `${"stamps touched by billing".padStart(30)}`); for (const n of [1, 2, 4, 8, 16]) { const components = STAMP_LOCAL * n + 1; // +1: the shared routing map console.log(`${String(n).padStart(7)}${String(components).padStart(18)}` + `${String(STAMP_LOCAL * n * 2 + 1).padStart(19)}${String(n).padStart(30)}`); } console.log(`the shared single resource's blast radius is independent of stamp count: always 90.9%`); ``` ``` stamps edge per stamp store per stamp writes per stamp blast radius daily shipments 1 513.89 138.89 97.22 100.00% 400,000 2 256.94 69.44 48.61 50.00% 200,000 4 128.47 34.72 24.30 25.00% 100,000 8 64.24 17.36 12.15 12.50% 50,000 16 32.12 8.68 6.08 6.25% 25,000 stamps total components with geo-replicas stamps touched by billing 1 9 17 1 2 17 33 2 4 33 65 4 8 65 129 8 16 129 257 16 the shared single resource's blast radius is independent of stamp count: always 90.9% ``` The first table is stamping's scaling face: at four stamps, the edge load per stamp is 128.47 requests/s, the load reaching the store is 34.72 requests/s, and the daily shipment count is 100,000. All of K01's calculations become re-readable per stamp, and once a single stamp's capacity is known, the number of stamps needed follows by division. The blast radius column is the same division's resilience face, and the two are the same number — the scale unit and the failure isolation unit coincide here. The second table gives **the failure-free day's cost**. Total component count grows linearly with stamp count: 9 at one stamp, 129 at sixteen stamps. If a **geo-replica** of each stamp is kept in a second region, the count doubles: 257. A geo-replica is the whole stamp replicated in another region; the mechanics of replication were established in the Scaling the Data Layer course and are not repeated here — the novelty here is that what gets replicated is not a record but a **deployment unit**. The last column is stamping's least visible cost. End-of-day billing must touch every shipment, and shipments are split across stamps; at sixteen stamps, the job has to go to sixteen places and merge the results. While the load per stamp shrinks, distribution is a cost for work that **has to cross** the stamp. This job depends on K01's four-hour window, and the window does not change; what changes is how many pieces the job is assembled from. #### The Numbers for Two Days **The failure-free day's cost** is three line items. Component count climbs from 9 to 33 (four stamps), to 65 with geo-replicas — each one a unit set up, upgraded, and monitored separately. End-of-day billing goes to four places instead of one. And a query that concerns two stamps at once becomes an operation with no place in the design, because it crosses the stamp boundary; in the self-contained placement, only 2 links remained crossing the boundary, and both of them went to the routing map. **The failing day's gain** is in the second table's stamp-local rows. The same failure — the state store's loss — affects 100 percent of requests in the single-piece arrangement, 25 percent in the four-stamp arrangement, and 6.25 percent in the sixteen-stamp arrangement. The edge component's loss goes from 90.9 percent to 22.7 percent. A failure's duration does not change; what changes is how many users are affected during that duration, and K01's request-based availability measure counts exactly that. Comparing the two days also gives a limit. Increasing stamp count shrinks the blast radius by $1/n$ while growing component count by $n$. The single shared resource's scope, though, stays at 90.9 percent and never shrinks; the pattern's gain is limited to the share left above this floor. #### Summary - A deployment stamp is a self-contained, replicable copy of the service; it is called a scale unit when used as a scaling unit, and it is a separate concept from the version stamp. - The measure of independence is not stamp count but the number of links crossing the stamp: 6 in the four-stamp arrangement with shared components, 2 once the shared components are moved into the stamp. - For stamp-local components, blast radius is divided by four (the state store goes from 100 percent to 25 percent); for shared components, it does not change at all (the cache stays at 72.7 percent, 4/4 stamps). - The routing map cannot be moved into a stamp, and its scope is 90.9 percent across all three placements: stamping does not zero out the blast radius, it reduces it to the single point that remains shared. - Stamp count directly divides K01's loads: at four stamps, 128.47 requests/s edge load, 34.72 requests/s store load, and 100,000 daily shipments per stamp; blast radius is the same division. - The failure-free day's cost is component count: from 9 to 33 at four stamps, to 65 with geo-replicas; end-of-day billing also has to go to as many places as there are stamps. #### Next Step This topic built a recovery arrangement over five lessons: the failover threshold and failback policy, the health endpoint's content and interval, the recovery time and data loss objectives, zone and region placement, stamps and geo-replicas. Every lesson's numbers came from a model or arithmetic, and all of them assumed one thing: **the plan will work as written.** The failover threshold will trigger correctly, the backup replica will really be current, the second region will carry the load, every component in the stamp will really sit inside the stamp. None of these assumptions has been tested. The next lesson tests the plan with a controlled failure trial: how much room the drill itself takes from the outage budget, what the gap is between the targeted recovery time and what the drill measures, and which assumptions an untested arrangement's first real failure breaks all at once. ### Disaster Recovery Drill Source: https://academia.sh/en/courses/resilience-patterns/disaster-recovery-drill Testing the recovery plan with a controlled failure trial: comparing the plan's step durations against what the drill measures, surfacing a step the plan never wrote down, showing the recovery time objective holding by plan but breaking by drill, and counting the duplicated side effects and orphaned workflows that two delivery policies leave behind for workflows left half-done in the failover window. This topic built a recovery arrangement over five lessons: the failover threshold and failback policy, the health endpoint's content and interval, the recovery time and data loss objectives, zone and region placement, stamps and geo-replicas. Every lesson's numbers came from a model or arithmetic, and all of them assumed the same thing: **the plan will work as written.** This assumption is a guess until it is tested. A **disaster recovery drill** tests the plan with a controlled failure trial: a specific unit, at a chosen time, is deliberately taken offline, and how recovery actually works is measured. A drill's scope is defined by three things — which unit (replica, zone, region, stamp), which failure mode (stop, slowdown, partition), and how long. A trial run without a written scope cannot say what it measured either. #### The Gap Between Plan and Measurement A recovery plan is a list of steps, and each step is assigned a duration. The drill's first output is a comparison of these durations. The measured values in the calculation below are not made up: they come from previous lessons' parameters. The detection duration is the threshold the first lesson chose; the warming duration is the warming window in the second lesson's failure schedule. **KU8 — the routing record's lifetime is 3 rounds.** Rationale: routing to the new active replica does not complete until the old replica's record expires. **KU9 — the queue consumer's reconnection is 2 rounds.** Both are assumptions and are not added to K01's table. ```js // drill/plan.mjs — comparing the recovery plan's step durations against what the drill measured. // Measured values come from previous lessons' parameters; a round is an abstract step. const THRESHOLD = 3; // lesson 01: the selected trigger threshold const PROMOTION = 1; // lesson 01: promotion round const LIFETIME = 3; // KU8 (assumption): the routing record's lifetime (rounds) const WARMING = 8; // lesson 02 schedule: a replica's warming duration (rounds) const QUEUE = 2; // KU9 (assumption): the queue consumer's reconnection (rounds) const VALIDATION = 2; // the validation step written into the plan const ROUND_SEC = 2; // KU1 (assumption): one round is 2 seconds const PEAK_EDGE = 513.89; // K01: peak requests/s at the edge const PLANNED = 15.0; // K01: monthly budget's planned-work share (minutes) const b = (x, n = 2) => x.toFixed(n); const STEP = [ ["detection", 1, THRESHOLD], ["promotion", 1, PROMOTION], ["routing", 1, LIFETIME], ["cache-warming", 0, WARMING], ["queue-reconnect", 1, QUEUE], ["validation", 2, VALIDATION], ]; console.log(`${"step".padEnd(17)}${"planned".padStart(11)}${"measured".padStart(12)}${"deviation".padStart(11)}`); let p = 0, o = 0; for (const [name, plan, measured] of STEP) { p += plan; o += measured; const d = measured - plan; console.log(`${name.padEnd(17)}${(plan + " round(s)").padStart(11)}${(measured + " round(s)").padStart(12)}` + `${((d >= 0 ? "+" : "") + d).padStart(11)}`); } console.log(`${"total".padEnd(17)}${(p + " round(s)").padStart(11)}${(o + " round(s)").padStart(12)}` + `${("+" + (o - p)).padStart(11)}`); console.log(`measured/planned ratio ${b(o / p)}x`); console.log(`\nplan ${p * ROUND_SEC} sec, measured in drill ${o * ROUND_SEC} sec (KU1: round = ${ROUND_SEC} sec)`); for (const target of [30, 60]) { const pl = p * ROUND_SEC <= target ? "holds" : "breaks", ol = o * ROUND_SEC <= target ? "holds" : "breaks"; console.log(` recovery time objective ${target} sec: ${pl} by plan, ${ol} by drill`); } console.log(` edge requests dropped in the drill window = ${b(o * ROUND_SEC * PEAK_EDGE)} (at peak load)`); console.log(`\nthe drill's own cost (one drill = ${b((o * ROUND_SEC) / 60)} min):`); console.log(`${"frequency".padEnd(14)}${"drills a month".padStart(16)}${"minutes a month".padStart(17)}` + `${"of planned share".padStart(19)}${"of monthly budget".padStart(19)}`); for (const [name, perMonth] of [["weekly", 4], ["monthly", 1], ["quarterly", 1 / 3]]) { const min = (perMonth * o * ROUND_SEC) / 60; console.log(`${name.padEnd(14)}${b(perMonth).padStart(16)}${b(min).padStart(17)}` + `${(b((100 * min) / PLANNED) + "%").padStart(19)}${(b((100 * min) / 43.2) + "%").padStart(19)}`); } console.log(`${STEP.filter(([, pl, ol]) => ol !== pl).length} of the plan's 6 steps measured a deviation; ` + `one (cache-warming) was not in the plan at all`); ``` ``` step planned measured deviation detection 1 round(s) 3 round(s) +2 promotion 1 round(s) 1 round(s) +0 routing 1 round(s) 3 round(s) +2 cache-warming 0 round(s) 8 round(s) +8 queue-reconnect 1 round(s) 2 round(s) +1 validation 2 round(s) 2 round(s) +0 total 6 round(s) 19 round(s) +13 measured/planned ratio 3.17x plan 12 sec, measured in drill 38 sec (KU1: round = 2 sec) recovery time objective 30 sec: holds by plan, breaks by drill recovery time objective 60 sec: holds by plan, holds by drill edge requests dropped in the drill window = 19527.82 (at peak load) the drill's own cost (one drill = 0.63 min): frequency drills a month minutes a month of planned share of monthly budget weekly 4.00 2.53 16.89% 5.86% monthly 1.00 0.63 4.22% 1.47% quarterly 0.33 0.21 1.41% 0.49% 4 of the plan's 6 steps measured a deviation; one (cache-warming) was not in the plan at all ``` The total row is the lesson's first result: the plan said 6 rounds, the drill measured 19 — 3.17 times as much. This ratio is a quantity derived independently of the run; it stays the same even if rounds are never converted to seconds. The deviation does not come from a single place. The detection and routing steps were written into the plan as one round each, yet the first depends on the selected threshold and the second on the routing record's lifetime — both are decisions **already made** in other lessons, and the plan never read them. The largest deviation, though, comes from a step that is not in the plan at all: the cache warming up. The new active replica is up, accepts writes, and passes the health check, but serves reads from a cold cache. **The plan's most expensive mistake is not a wrong estimate but a missing step.** The result hits the threshold. A thirty-second recovery time objective holds comfortably when read from the plan (12 seconds), and breaks when read from the drill (38 seconds). The sixty-second objective passes on both readings. Whether an objective holds cannot be read from the plan; the drill is the one step that turns an objective from a claim into a measured number. #### Scope and Frequency The drill's scope is a ladder, and each rung tests something different. The bottom rung is a paper review of the plan's steps; this can find a missing step but cannot measure duration — in the table above, cache warming being missing from the plan could have been caught this way, but its eight-round duration could not have been. The middle rung is taking a single component offline; it tests the failover threshold and the health endpoint's content. The top rung is the loss of an entire zone or region, and it is the only trial that tests the previous two lessons' placement decisions; only here is it seen whether idle capacity actually carries the real load. The frequency table ties the cost to frequency. A weekly drill takes 2.53 minutes a month — 16.89 percent of the planned-work share and 5.86 percent of the monthly budget. A monthly drill takes 0.63 minutes, a quarterly one 0.21 minutes. An infrequent drill is cheap, but the changes that accumulate between two drills age the plan again; the measured deviation grows from zero each time. This lesson gives the number, not the choice: the choice depends on how often the plan's steps change. #### The Second Problem the Drill Surfaces The drill's last step is validation: checking the state of the data and the in-progress work once recovery is done. Recovery time has been measured, but what happens to **workflows left half-done** during the failover window has not been counted. The setup below is a **model**: the end-of-day billing flow has eight steps, four of them have side effects (writing an invoice, posting a ledger entry, sending a notification, notifying the carrier), and the failover window is the 19 rounds just measured. ```js // drill/work.mjs — the drill's data-validation step is a MODEL: what the workflows left half-done // in the failover window leave behind under two policies. A round is an abstract step. const STEP = [ // steps of the billing workflow ["read-state", false], ["apply-tariff", false], ["compute-amount", false], ["write-invoice", true], ["post-ledger-entry", true], ["send-notification", true], ["notify-carrier", true], ["close", false], ]; const ROUND = 70, WORKFLOWS_PER_ROUND = 4, START_UNTIL = 40; // 4 workflows start each round, through round 40 const FAILOVER = 20, WINDOW = 19; // 19 rounds: the total measured in plan.mjs function run(policy) { const workflows = []; const s = { completed: 0, duplicatedSideEffects: 0, orphaned: 0, restarted: 0, discarded: 0 }; for (let t = 1; t <= ROUND; t += 1) { const frozen = t >= FAILOVER && t < FAILOVER + WINDOW; if (t === FAILOVER) { for (const w of workflows) { if (w.step === 0 || w.step >= STEP.length) continue; // not yet started or already finished if (policy === "at-least-once") { s.restarted += 1; s.duplicatedSideEffects += STEP.slice(0, w.step).filter(([, y]) => y).length; w.step = 0; // the workflow restarts from the beginning } else { s.discarded += 1; if (STEP.slice(0, w.step).some(([, y]) => y)) s.orphaned += 1; w.dead = true; } } } if (frozen) continue; // the workflow makes no progress during the window for (const w of workflows) { if (w.dead || w.step >= STEP.length) continue; w.step += 1; if (w.step === STEP.length) s.completed += 1; } if (t <= START_UNTIL) for (let i = 0; i < WORKFLOWS_PER_ROUND; i += 1) workflows.push({ step: 0, dead: false }); } return { ...s, total: workflows.length }; } const s = (x, n) => String(x).padStart(n); console.log(`${ROUND} rounds, ${WORKFLOWS_PER_ROUND} workflows start per round, each workflow has ${STEP.length} steps ` + `(${STEP.filter(([, y]) => y).length} of them have side effects). Failover at round ${FAILOVER}, ` + `window ${WINDOW} rounds.`); console.log(); console.log(`${"policy".padEnd(14)} | ${s("total workflows", 15)} | ${s("completed", 9)} | ${s("restarted", 9)} | ${s("discarded", 9)} | ${s("duplicated side effects", 24)} | ${s("orphaned", 8)}`); console.log(`${"-".repeat(14)}-|-${"-".repeat(15)}-|-${"-".repeat(9)}-|-${"-".repeat(9)}-|-${"-".repeat(9)}-|-${"-".repeat(24)}-|-${"-".repeat(8)}`); for (const p of ["at-least-once", "at-most-once"]) { const r = run(p); console.log(`${p.padEnd(14)} | ${s(r.total, 15)} | ${s(r.completed, 9)} | ` + `${s(r.restarted, 9)} | ${s(r.discarded, 9)} | ${s(r.duplicatedSideEffects, 24)} | ${s(r.orphaned, 8)}`); } ``` ``` 70 rounds, 4 workflows start per round, each workflow has 8 steps (4 of them have side effects). Failover at round 20, window 19 rounds. policy | total workflows | completed | restarted | discarded | duplicated side effects | orphaned ---------------|-----------------|-----------|-----------|-----------|--------------------------|--------- at-least-once | 84 | 84 | 28 | 0 | 40 | 0 at-most-once | 84 | 56 | 0 | 28 | 0 | 16 ``` The two rows show two bad outcomes, and there is no third option in between. In **at-least-once** delivery, every workflow completes: 84/84. In exchange, the 28 workflows that were half-done at the moment of failover run from the beginning, and side-effect steps already executed run a **second time** — 40 duplicated side effects. That means 40 extra invoice lines, ledger entries, or carrier notifications. In **at-most-once** delivery there is no duplication, but completed work drops from 84 to 56: 28 workflows are discarded, and 16 of them have already executed at least one side-effect step. That means an invoice was written for 16 shipments but no ledger entry was posted, or a notification was sent but the workflow never closed. These workflows were neither completed nor rolled back; they left a trace in the system, and nothing collects them. The length of the recovery window directly grows these numbers. Because the window is 19 rounds, 28 workflows were left half-done; with the plan's 6-round window, this number would have been smaller. **Shortening recovery time does not only protect the outage budget, it also shrinks the correctness damage** — two separate quantities measured by two lessons meet in the same place. #### The Numbers for Two Days **The failure-free day's cost** is the drill itself. A drill run once a month takes 38 seconds, that is, 0.63 minutes, and spends 4.22 percent of the 15.0 minutes K01 sets aside for planned work. If run at peak load, it drops 19,527.82 edge requests; this is why a drill's scope includes the time window too. The cost is real and paid voluntarily. **The failing day's gain** comes from reading both tables together. In an untested arrangement, the plan promised a 12-second recovery time; the real failure would have measured 38 seconds, and the 30-second objective would have broken on the first failure. The drill made this gap visible before the objective broke. It also surfaced a step that was not in the plan at all, and measured that four of six steps had the wrong duration written down. The same drill also counted a damage that has nothing to do with recovery time: 40 duplicated side effects, or 16 orphaned workflows. This comparison is the clearest example of this course's rule. The drill pays only cost on a failure-free day — 0.63 minutes and 19,527.82 requests. Its gain becomes visible only when a failure happens, and on that day it takes the form of not running into an unknown deviation. #### Summary - A drill tests the recovery plan with a controlled failure trial; its scope is defined by the questions of which unit, which failure mode, and how long. - Recovery written into the plan as 6 rounds measured 19 rounds in the drill (3.17 times); four of six steps had the wrong duration written down, and the largest deviation came from cache warming, which was not in the plan at all. - An objective cannot be tested from the plan: the 30-second recovery time objective passes by plan, and breaks against the 38 seconds measured in the drill. - The drill's cost depends on frequency: a monthly drill costs 0.63 minutes (4.22 percent of the planned share), a weekly one 2.53 minutes (16.89 percent); at peak load, every drill drops 19,527.82 edge requests. - The validation step surfaced a second damage: 28 workflows were left half-done in the 19-round window; in at-least-once delivery all of them completed but 40 side effects ran a second time, in at-most-once delivery there was no duplication but 16 workflows were left orphaned. - Shortening the recovery window does not only protect the outage budget; it also reduces correctness damage by shrinking the number of workflows left half-done. #### Next Step The drill validated the recovery arrangement and, at the same time, showed its limit. Failover completed, objectives were measured, placement was tested — but the last table left something behind that none of this topic's patterns solves. **The same work ran twice:** 40 side-effect steps ran again, because the new active replica did not know where the work had left off and started it from the beginning. The same problem arises not only at failover but also in the previous topic's retries; every resent request may already have been processed. At the other end, **the orphaned workflows were never rolled back:** 16 workflows stopped after leaving one side effect, and no step was designed to collect them. The next topic takes on both questions: making a request's second processing harmless, and compensating for steps that cannot be rolled back. ## Distributed Correctness Getting the right result from work that is retried under failure, left half-finished, or moved in stages: neutralizing a duplicate request without producing side effects, balancing steps that cannot be rolled back with compensating transactions, moving a large payload outside the message, granting limited-time authorization for direct resource access, collecting runtime settings externally, and replacing a legacy system route by route. ### Idempotent Operations Source: https://academia.sh/en/courses/resilience-patterns/idempotent-operations Neutralizing a duplicate request during recovery: counting the three sources of duplication across a ninety-second failure window, an idempotency key's scope suppressing valid events when it is chosen too narrow, and the record, byte, and write-row cost the key ledger adds at K01 scale. The Recovery and Continuity topic closed with a drill, and it left two open problems. When failover and retry ran together, **the same work ran twice**: a carrier notification was both resent after a timeout and replayed from the log by the replica that took over. The second problem — half-finished workflows were never rolled back — is still open. This lesson takes on the first. **Idempotence** and the **idempotency key** were defined in the Web API Design and The Data Access Layer and Business Logic courses; those courses showed that a second request arriving with the same key gets served the first response, that the key is stored in the same transaction as the effect, and that delivery semantics duplicate delivery, not effect. The definition is not repeated here. This lesson has two questions: how many **side effects** a duplication produces in a failure window, and what the ledger that holds that duplication **costs on a failure-free day**. #### The Three Sources of Duplication The failure scenario is named directly. **The carrier notification endpoint slows down**: the endpoint keeps answering, but its response time crosses the caller's timeout threshold, so the caller counts the request as failed and retries. If a replica takes over during the slowdown, the replica that took over replays the log's last writes, and the queue side redelivers the unacknowledged message. Three sources run at the same time. **DD1 — the failure window is 90 seconds, twice a month.** Rationale: this duration sits below K01's ten-minute recovery-time assumption, so it does not register as an outage in the monthly outage budget, but it sits above the timeout threshold. Its sensitivity is given at 4 windows a month. **DD2 — the probability that a write is duplicated in the window is 0.35**, split across three sources: 0.20 for the carrier's retry, 0.10 for replay on failover, 0.05 for at-least-once redelivery. Both are assumptions and are not added to K01's table. The rig is a process-internal model: there is no queue, store, or network; the ledger is a set and side effects are counters. The only property the model carries is that the same event can reach the handler more than once. ```js // idempotent/stream.mjs — the in-process model of the state event stream and the idempotency ledger. // There is no real queue, store, or network: the ledger is a set, side effects are counters. export const SEQUENCE = ["accepted", "transferred", "transferred", "transferred", "out-for-delivery", "delivery-attempt", "delivered"]; // K01 V4 = 7 state events per shipment export function generator(seed) { // 32-bit linear congruential generator, seed exposed let s = seed >>> 0; return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; }; } export const events = (shipment) => SEQUENCE.map((state, i) => ({ tracking: `TR-${shipment}`, state, eventNo: i })); export const key = (o, mode) => mode === "tracking" ? o.tracking : mode === "tracking+state" ? `${o.tracking}|${o.state}` : `${o.tracking}|${o.state}|${o.eventNo}`; // Side effect: every event writes a state record; delivery also adds an invoice line and sends a notification. export const sideEffects = (o) => (o.state === "delivered" ? 3 : 1); export function processor({ ledgered, mode }) { const ledger = new Set(); const s = { processed: 0, suppressed: 0, sideEffects: 0 }; return { s, handle(o) { const k = key(o, mode); if (ledgered && ledger.has(k)) { s.suppressed += 1; return; } ledger.add(k); s.processed += 1; s.sideEffects += sideEffects(o); }, size: () => ledger.size, }; } // Delivery stream: every event arrives once; in the failure window, three sources produce extra deliveries. export function deliveries(shipmentCount, sources, seed) { const rnd = generator(seed); const stream = [], counts = new Map(sources.map(([name]) => [name, 0])); for (let g = 0; g < shipmentCount; g += 1) for (const o of events(g)) { stream.push(o); for (const [name, p] of sources) if (rnd() < p) { stream.push(o); counts.set(name, counts.get(name) + 1); } } return { stream, counts }; } ``` The same stream runs under three key modes and once without a ledger. The key mode is a scope decision: whether the key consists of only the tracking number, the tracking number and the state, or the triple that also includes the event number the carrier produces. ```js // idempotent/measure.mjs — the failure-free day's cost and the failure window's gain are counted separately import { SEQUENCE, deliveries, processor } from "./stream.mjs"; const PEAK_WRITE = 97.22, WINDOW = 90; // K01 computed value; DD1 failure window const SHIPMENTS = Math.round((PEAK_WRITE * WINDOW) / SEQUENCE.length); const SOURCES = [["carrier retry", 0.20], // DD2, split across three sources ["replay on failover", 0.10], ["at-least-once redelivery", 0.05]]; const MODE = ["tracking", "tracking+state", "tracking+state+event"]; const run = (sources, seed) => { const { stream, counts } = deliveries(SHIPMENTS, sources, seed); const rows = MODE.map((mode) => { const d = processor({ ledgered: true, mode }); for (const o of stream) d.handle(o); return [mode, d.s, d.size()]; }); const y = processor({ ledgered: false, mode: "tracking+state+event" }); for (const o of stream) y.handle(o); return { stream, counts, rows, y: y.s }; }; const print = (title, r) => { console.log(`\n${title}: ${r.stream.length} deliveries, side effects without a ledger ${r.y.sideEffects}`); for (const [name, s, size] of r.rows) console.log(`${name.padEnd(22)}${String(s.processed).padStart(9)}${String(s.suppressed).padStart(12)}` + `${String(s.sideEffects).padStart(13)}${String(r.y.sideEffects - s.sideEffects).padStart(11)}${String(size).padStart(11)}`); }; console.log(`model: ${SHIPMENTS} shipments x ${SEQUENCE.length} events = ${SHIPMENTS * SEQUENCE.length} distinct events`); console.log(`(K01 peak write ${PEAK_WRITE} events/s x DD1 window ${WINDOW} s)`); console.log(`\n${"key mode".padEnd(22)}${"processed".padStart(9)}${"suppressed".padStart(12)}` + `${"side effects".padStart(13)}${"prevented".padStart(11)}${"ledger".padStart(11)}`); print("failure-free day (no duplication)", run([], 20260730)); const r = run(SOURCES, 20260730); print("failure window (DD2)", r); console.log(`source split: ${[...r.counts].map(([a, n]) => `${a} ${n}`).join(", ")}`); const duplicates = r.stream.length - SHIPMENTS * SEQUENCE.length; const duplicateDeliveries = r.stream.filter((o) => o.state === "delivered").length - SHIPMENTS; console.log(`duplicate deliveries ${duplicates}, ${duplicateDeliveries} of them are "delivered" events; ` + `that means ${duplicateDeliveries} duplicate invoice lines and ${duplicateDeliveries} duplicate notifications`); ``` ``` model: 1250 shipments x 7 events = 8750 distinct events (K01 peak write 97.22 events/s x DD1 window 90 s) key mode processed suppressed side effects prevented ledger failure-free day (no duplication): 8750 deliveries, side effects without a ledger 11250 tracking 1250 7500 1250 10000 1250 tracking+state 6250 2500 8750 2500 6250 tracking+state+event 8750 0 11250 0 8750 failure window (DD2): 11849 deliveries, side effects without a ledger 15251 tracking 1250 10599 1250 14001 1250 tracking+state 6250 5599 8750 6501 6250 tracking+state+event 8750 3099 11250 4001 8750 source split: carrier retry 1789, replay on failover 876, at-least-once redelivery 434 duplicate deliveries 3099, 451 of them are "delivered" events; that means 451 duplicate invoice lines and 451 duplicate notifications ``` These numbers are in the **measurement** class: they come from a process-internal model run with a generator seeded at 20260730, and they reproduce under the same seed. Their inputs are K01's **computed value** (peak write 97.22 events/s) and this lesson's **assumptions** (DD1, DD2). #### The Key's Scope Is a Suppression Decision The block above counts the failure-free day, and the table's most expensive row sits there. With no duplication, the correct scope (`tracking+state+event`) suppresses nothing: 8750 events are processed, 11250 side effects are produced. The narrow `tracking` key, on the other hand, suppresses **7500 valid events**, because a shipment's seven state events all carry the same tracking number; only the first survives, and side effects drop from 11250 to 1250. The mid-scope `tracking+state` suppresses 2500 events, because the sequence's three `transferred` events all produce the same key. The wrong scope's large numbers in the "prevented" column cannot be read as a gain. In the failure window, the `tracking` mode looks like it prevented 14001 side effects; 10,000 of that is the same suppression that already happens on a failure-free day — that is, it deletes real side effects. The rule is this: **the idempotency key's scope must be wide enough to distinguish two events that are distinct in business terms**; narrowing the key destroys the system's own work, not the duplication. The block below counts the failure window. Under the correct scope, 8750 distinct events turn into 11,849 deliveries; of the 3099 extra deliveries, 1789 come from the carrier's retry, 876 from replay on failover, 434 from at-least-once redelivery. Without a ledger, side effects would be 15,251 instead of 11,250: **4001 extra side effects**. Of those, 451 are duplicate invoice lines and 451 are duplicate notifications — one is a delivery billed twice to the seller, the other a delivery notice sent twice to the recipient. #### The Ledger's Cost The ledger is not free. Every write first reads the ledger, then writes a key row alongside its own record; these rows are kept for a window. **DD3 — a key record is 72 bytes, the ledger window is 24 hours.** The record carries the key, a body digest, a timestamp, and an outcome marker. The window's rationale is that it needs to be long enough to cover a client's retry chain and the replay after failover. Its sensitivity is given at twice the record size, three times the window, and a one-hour window. ```js // idempotent/cost.mjs — the records the ledger holds, the bytes it occupies, and row rates at K01 scale const EVENTS = 2_800_000, SHIPMENTS = 400_000; // K01 computed value: daily state events and shipments const PEAK_WRITE = 97.22, BEHIND_CACHE_READ = 41.67; // K01 computed value: peak write, reads behind cache const DAILY_MB = 976, RATIO = 2.33; // K01 computed value: daily data growth, write/read const RECORD = 72, HOUR = 24; // DD3: key record bytes and ledger window const keyedWrites = EVENTS + SHIPMENTS; console.log(`daily writes needing a key ${keyedWrites.toLocaleString("en-US")} ` + `(${EVENTS.toLocaleString("en-US")} state events + ${SHIPMENTS.toLocaleString("en-US")} shipments)`); console.log(`\n${"window".padStart(9)}${"record bytes".padStart(14)}${"records held".padStart(15)}` + `${"ledger MB".padStart(12)}${"of daily growth".padStart(18)}`); for (const [s, b] of [[HOUR, RECORD], [HOUR, RECORD * 2], [HOUR * 3, RECORD], [1, RECORD]]) { const record = (keyedWrites * s) / 24, mb = (record * b) / 1e6; console.log(`${`${s} h`.padStart(9)}${String(b).padStart(14)}` + `${Math.round(record).toLocaleString("en-US").padStart(15)}${mb.toFixed(2).padStart(12)}` + `${`${((100 * mb) / DAILY_MB).toFixed(2)}%`.padStart(18)}`); } const write2 = PEAK_WRITE * 2, read2 = BEHIND_CACHE_READ + PEAK_WRITE; console.log(`\npeak write rows/s ${PEAK_WRITE} -> ${write2.toFixed(2)} (${(write2 / PEAK_WRITE).toFixed(2)}x)`); console.log(`reads behind cache/s ${BEHIND_CACHE_READ} -> ${read2.toFixed(2)} (${(read2 / BEHIND_CACHE_READ).toFixed(2)}x)`); console.log(`write/read ratio at the store ${RATIO} -> ${(write2 / read2).toFixed(2)}`); const PREVENTED = 4001, INVOICE = 451; // measure.mjs: prevented side effects and console.log(`\n${"DD1 windows/month".padStart(18)}${"side effects prevented".padStart(24)}` + // duplicate deliveries per window `${"duplicate invoice lines".padStart(25)}${"of daily lines".padStart(18)}`); for (const n of [2, 4]) console.log(`${n.toFixed(0).padStart(18)}${(n * PREVENTED).toLocaleString("en-US").padStart(24)}` + `${(n * INVOICE).toLocaleString("en-US").padStart(25)}` + `${`${((100 * n * INVOICE) / (SHIPMENTS * 30)).toFixed(5)}%`.padStart(18)}`); console.log(`the wrong scope's cost: the tracking+state mode suppresses ` + `${((EVENTS * 2) / 7).toLocaleString("en-US")} valid events a day, ` + `the tracking mode suppresses ${((EVENTS * 6) / 7).toLocaleString("en-US")}`); ``` ``` daily writes needing a key 3,200,000 (2,800,000 state events + 400,000 shipments) window record bytes records held ledger MB of daily growth 24 h 72 3,200,000 230.40 23.61% 24 h 144 3,200,000 460.80 47.21% 72 h 72 9,600,000 691.20 70.82% 1 h 72 133,333 9.60 0.98% peak write rows/s 97.22 -> 194.44 (2.00x) reads behind cache/s 41.67 -> 138.89 (3.33x) write/read ratio at the store 2.33 -> 1.40 DD1 windows/month side effects prevented duplicate invoice lines of daily lines 2 8,002 902 0.00752% 4 16,004 1,804 0.01503% the wrong scope's cost: the tracking+state mode suppresses 800,000 valid events a day, the tracking mode suppresses 2,400,000 ``` #### The Numbers for Two Days **The failure-free day's cost has three line items.** The ledger holds 3,200,000 records and takes up 230.40 MB — 23.61 percent of K01's 976 MB daily data growth. The peak write row rate climbs from 97.22 to 194.44, exactly double. A key lookup is added to the read side: reads behind cache climb from 41.67 to 138.89, 3.33 times. The write/read ratio at the store **drops** from 2.33 to 1.40 — the smaller number does not mean less load; the ledger grew both sides, the read side more. Sensitivity concentrates in the window. When the window climbs from 24 hours to 72 hours, the ledger reaches 691.20 MB, 70.82 percent of the daily growth; at a one-hour window it stays at 9.60 MB and 0.98 percent. Doubling the record size does not have the same effect: 460.80 MB. **The window, not the record size, sets the ledger's cost**, because the window multiplies both the record count and the retention time. **The failure day's gain has two line items.** At DD1's 2 windows a month, 4001 side effects prevented per window, 8002 a month. Of that, 902 are duplicate invoice lines, 0.00752 percent of K01's 400,000 daily lines. The percentage is small, and it is not the number that defends the pattern: 902 duplicate invoice lines are 902 wrongly billed deliveries — 902 reconciliation mismatches — and the same number of recipients get a second delivery notice. At twice DD1, it becomes 1804. The side effect count is read **by the unit**, not by the percentage, because each unit produces a correction job. #### Summary - Duplication does not come from one source: of the 3099 extra deliveries in the 90-second window, 1789 came from retry, 876 from replay on failover, 434 from at-least-once delivery. - Without a ledger, side effects would be 15,251 instead of 11,250; of the 4001 side effects prevented, 451 are duplicate invoice lines and 451 are duplicate notifications. - The idempotency key's scope is a suppression decision: on a failure-free day, the `tracking` mode suppresses 7500 valid events and `tracking+state` suppresses 2500; at K01 scale that is 2,400,000 and 800,000 events a day. - The failure-free day's cost: 3,200,000 ledger records, 230.40 MB (23.61 percent of daily growth), peak write rows at 2.00x, reads behind cache at 3.33x. - The window sets the cost: a 72-hour window pushes the ledger to 691.20 MB, a one-hour window holds it at 9.60 MB; doubling the record size gives 460.80 MB. - The failure day's gain is 8002 side effects prevented a month and 902 duplicate invoice lines — 0.00752 percent of the daily lines, but 902 separate correction jobs. #### Next Step The ledger stopped duplication, but only of the **same** work running a second time. The drill's second open problem still stands: a half-finished workflow's **completed steps** are still never rolled back. If a billing job fails at its third step, the first two steps' effects stay final, and those are not duplicated effects but **half-finished** ones; the idempotency key says nothing about this case. The next lesson takes on compensating for steps that cannot be rolled back: where an uncompensatable step belongs in the ordering, how many jobs are left stuck when the compensation call itself fails, and how long the intermediate state stays visible during the rollback window. ### Compensating Transactions Source: https://academia.sh/en/courses/resilience-patterns/compensating-transactions Balancing steps that cannot be rolled back: only two of twenty-four orderings satisfying the dependencies, the case where an unrollbackable step cannot be moved to the end, a failing compensation call leaving 2.979 jobs a day stuck, and the intermediate state window coming from stuck work rather than from rollback. The previous lesson stopped duplication: the idempotency key kept the same work from running a second time. The drill's second open problem still stands. When the end-of-day billing workflow fails at its third step, the first two steps' effects stay final; those are not duplicated effects but **half-finished** ones, and the idempotency key says nothing about them. A **compensating transaction** is a step that balances a completed step's effect with an inverse operation — the same concept as The Data Access Layer and Business Logic course's **compensating step**, which was built there inside a saga arrangement, where the possibility of the compensation itself failing was already measured; the mechanics are not repeated here. Do not let the similar name mislead: the Caching, Queues and Asynchronous Processing course's **catch-up policy** is a separate decision about how a lagging consumer catches up. This lesson has three questions: where an uncompensatable step can be placed in the ordering, how many jobs stay stuck when a compensation call itself fails, and how long the intermediate state stays visible in the rollback window. #### The Ordering Is Not Free End-of-day billing has four steps: closing the period, writing the invoice line, opening the collection entry, and sending the seller notification. Two of them cannot be rolled back. **Closing the period** cannot be rolled back, because reopening a closed period drops the carrier events that arrived in between out of scope and makes the same day billable twice. **The seller notification** cannot be rolled back, because a notification already sent can only be corrected by a second notification. The rule is well known: an uncompensatable step goes at the end of the ordering. The problem is that the steps carry **dependencies**. The invoice line cannot be written before the period closes, the collection entry cannot open without the invoice line, and the notification cannot be sent before the invoice line. How many orderings both satisfy the dependencies and honor the rule is a countable question. ```js // compensating/flow.mjs — the in-process model of the end-of-day billing workflow. // There is no service, network, or store: steps are plain objects, rollback is a loop. // A compensating transaction is the same thing as M16/K04's compensating step; its mechanics were built there. export const STEP = [ { name: "close-period", compensate: null, requires: [] }, // cannot be rolled back { name: "invoice-line", compensate: "cancel-line", requires: ["close-period"] }, { name: "collection-entry", compensate: "revert-entry", requires: ["invoice-line"] }, { name: "seller-notification", compensate: null, requires: ["invoice-line"] }, // cannot be rolled back ]; export function* permutations(steps) { // every ordering if (steps.length === 0) { yield []; return; } for (let i = 0; i < steps.length; i += 1) for (const k of permutations(steps.filter((_, j) => j !== i))) yield [steps[i], ...k]; } export const valid = (d) => d.every((a, i) => a.requires.every((o) => d.slice(0, i).some((x) => x.name === o))); // One run: the failure occurs at step k; completed steps are compensated in reverse order. // compensationFailure: the compensation call's failure probability; attempts: compensation attempt count. export function run(ordering, k, { compensationFailure, attempts, rnd }) { const completed = ordering.slice(0, k); const r = { uncompensated: 0, compensationCalls: 0, stuck: 0, round: 0 }; for (const a of [...completed].reverse()) { r.round += 1; if (a.compensate === null) { r.uncompensated += 1; continue; } let succeeded = false; for (let d = 0; d < attempts && !succeeded; d += 1) { r.compensationCalls += 1; if (d > 0) r.round += 1; succeeded = rnd() >= compensationFailure; } if (!succeeded) r.stuck += 1; } return r; } export function generator(seed) { // 32-bit linear congruential generator, seed exposed let s = seed >>> 0; return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; }; } ``` **DD5 — the compensation call's failure probability is 0.05.** Rationale: compensation is itself a remote call and runs right after a failure; the target service may still be broken at that moment, so it carries a higher error margin than an ordinary call. Its sensitivity is given by sweeping the attempt count. The number is an **assumption** and is not added to K01's table. ```js // compensating/measure.mjs — the uncompensated step and rollback rounds across valid orderings import { STEP, permutations, valid, run, generator } from "./flow.mjs"; const ALL = [...permutations(STEP)]; const VALID = ALL.filter(valid); console.log(`${ALL.length} orderings, ${VALID.length} of them satisfy the dependencies`); console.log(`cannot be rolled back: ${STEP.filter((a) => a.compensate === null).map((a) => a.name).join(", ")}`); const noFailure = () => 1; // failure-free compensation run: rnd() = 1 >= failure rate console.log(`\n${"ordering".padEnd(66)}${"uncompensated".padStart(15)}${"rounds".padStart(8)}`); const summary = []; for (const d of VALID) { let uncompensated = 0, rounds = 0; for (let k = 1; k <= STEP.length; k += 1) { // failure at step k const r = run(d, k - 1, { compensationFailure: 0, attempts: 1, rnd: noFailure }); uncompensated += r.uncompensated; rounds += r.round; } summary.push([d.map((a) => a.name).join(" > "), uncompensated, rounds]); console.log(`${summary.at(-1)[0].padEnd(66)}${String(uncompensated).padStart(15)}${String(rounds).padStart(8)}`); } const chosen = VALID[summary.findIndex((o) => o[1] === Math.min(...summary.map((x) => x[1])))]; console.log(`\nchosen ordering: ${chosen.map((a) => a.name).join(" > ")}`); console.log(`${"step that fails".padEnd(28)}${"completed".padStart(12)}${"compensated".padStart(14)}` + `${"uncompensated".padStart(16)}${"rollback rounds".padStart(18)}`); for (let k = 1; k <= STEP.length; k += 1) { const r = run(chosen, k - 1, { compensationFailure: 0, attempts: 1, rnd: noFailure }); console.log(`${`${k}. (${chosen[k - 1].name})`.padEnd(28)}${String(k - 1).padStart(12)}` + `${String(r.compensationCalls).padStart(14)}${String(r.uncompensated).padStart(16)}${String(r.round).padStart(18)}`); } // DD5: the compensation call's failure probability is 0.05. Attempt count is swept. const N = 100_000, FAILURE_RATE = 0.05; console.log(`\n${N} runs, DD5 compensation failure rate ${FAILURE_RATE}; the failing step is chosen with an even distribution`); console.log(`${"attempts".padStart(9)}${"stuck jobs".padStart(13)}${"ratio".padStart(10)}` + `${"calls/job".padStart(12)}${"avg rollback rounds".padStart(22)}`); for (const attempts of [1, 2, 3]) { const rnd = generator(20260730); let stuck = 0, calls = 0, rounds = 0; for (let i = 0; i < N; i += 1) { const k = i % STEP.length; // failing step, even distribution const r = run(chosen, k, { compensationFailure: FAILURE_RATE, attempts, rnd }); stuck += r.stuck > 0 ? 1 : 0; calls += r.compensationCalls; rounds += r.round; } console.log(`${String(attempts).padStart(9)}${String(stuck).padStart(13)}` + `${(stuck / N).toFixed(5).padStart(10)}${(calls / N).toFixed(3).padStart(12)}` + `${(rounds / N).toFixed(3).padStart(22)}`); } ``` ``` 24 orderings, 2 of them satisfy the dependencies cannot be rolled back: close-period, seller-notification ordering uncompensated rounds close-period > invoice-line > collection-entry > seller-notification 3 6 close-period > invoice-line > seller-notification > collection-entry 4 6 chosen ordering: close-period > invoice-line > collection-entry > seller-notification step that fails completed compensated uncompensated rollback rounds 1. (close-period) 0 0 0 0 2. (invoice-line) 1 0 1 1 3. (collection-entry) 2 1 1 2 4. (seller-notification) 3 2 1 3 100000 runs, DD5 compensation failure rate 0.05; the failing step is chosen with an even distribution attempts stuck jobs ratio calls/job avg rollback rounds 1 3724 0.03724 0.750 1.500 2 214 0.00214 0.788 1.538 3 8 0.00008 0.790 1.540 ``` These numbers are in the **measurement** class: they come from a process-internal model and a generator seeded at 20260730, and they reproduce under the same seed. #### The Rule Only Applies as Far as the Dependencies Allow Twenty-two of the twenty-four orderings violate a dependency; two orderings remain, and both start with `close-period`. This is the rule's limit: **one of the two uncompensatable steps cannot be moved to the end, because the other three steps depend on it.** The notification can be moved to the end; closing the period cannot. The table below shows the cost step by step. If the failure occurs at the first step, nothing has completed; the uncompensated step count is 0. If it occurs at the second, third, or fourth step, the period is closed and stays closed: the uncompensated step count is 1 in all three cases. Changing the ordering does not lower this number, it only moves the position of the second uncompensatable step — when the notification is moved to third place, the total uncompensated step count climbs from 3 to 4. The design rule that follows from this has nothing to do with ordering. If an uncompensatable step cannot be moved to the end, **making it compensatable** is the only way out: if closing the period is defined as writing a flag rather than a deletion, it has a compensation. The ordering rule is a placement tool; it does not substitute for step design. #### Stuck Work and the Intermediate State Window The block below counts compensation's own failure. At one attempt, 3.724 percent of jobs stay **stuck**: the compensation call failed, the step's effect stands, and the job is neither completed nor rolled back. A second attempt brings the ratio down to 0.00214, a third to 0.00008. In exchange, compensation calls per job climb from 0.750 to 0.790, and rollback rounds from 1.500 to 1.540 — a second attempt cuts stuck work **seventeenfold** for 5 percent more calls. The intermediate state window is the invoice line appearing "issued" while the rollback is still underway. This window's length is measured in rounds; giving a round a duration requires a conversion factor. **DD6 — one rollback round is 0.25 seconds.** Rationale: a round is a remote call plus its confirmation. Its sensitivity is given at four times that value. **DD7 — stuck work is closed within 4 hours**, the same as K01's V10 batch window; that is the billing job's window. ```js // compensating/cost.mjs — converting the measured ratios into daily numbers at K01 scale const INVOICE_LINES = 4000; // K01 computed value: daily invoice lines const KK2 = 0.02; // M19/K03 choreography lesson's assumption: workflow failure rate const ROUND_SEC = 0.25, STUCK_HOURS = 4; // DD6 round duration, DD7 time to close stuck work const failed = INVOICE_LINES * KK2; console.log(`daily workflows ${INVOICE_LINES}, KK2 = ${KK2} -> failed ${failed}/day`); console.log(`\n${"ordering".padEnd(22)}${"uncompensated steps/day".padStart(26)}${"a year".padStart(9)}`); for (const [name, ratio] of [["notification last", 3 / 4], ["notification third", 4 / 4]]) console.log(`${name.padEnd(22)}${(failed * ratio).toFixed(1).padStart(26)}` + `${(failed * ratio * 365).toFixed(0).padStart(9)}`); // measure.mjs measurements: stuck work ratio and average rollback rounds, by compensation attempts const MEASURED = [[1, 0.03724, 1.500], [2, 0.00214, 1.538], [3, 0.00008, 1.540]]; console.log(`\n${"attempts".padStart(9)}${"stuck work/day".padStart(16)}${"a year".padStart(8)}` + `${"rollback s".padStart(12)}${"intermediate s/day".padStart(20)}${"h/day from stuck".padStart(19)}`); for (const [attempts, ratio, rounds] of MEASURED) { const stuck = failed * ratio, rollback = rounds * ROUND_SEC; console.log(`${String(attempts).padStart(9)}${stuck.toFixed(3).padStart(16)}` + `${(stuck * 365).toFixed(0).padStart(8)}${rollback.toFixed(3).padStart(12)}` + `${(failed * rollback).toFixed(1).padStart(20)}${(stuck * STUCK_HOURS).toFixed(3).padStart(19)}`); } const [, ratio1, rounds1] = MEASURED[0]; const factor = (t) => (failed * ratio1 * STUCK_HOURS * 3600) / (failed * rounds1 * t); console.log(`\nat one attempt, from stuck work / from rollback = ${factor(ROUND_SEC).toFixed(0)}x`); console.log(`DD6 sensitivity: at a round duration of ${ROUND_SEC * 4} s instead of ${ROUND_SEC} s, intermediate state is ` + `${(failed * rounds1 * ROUND_SEC * 4).toFixed(1)} s/day, ratio ${factor(ROUND_SEC * 4).toFixed(0)}x`); console.log(`\nfailure-free day's cost: 2 of the 4 steps carry a compensation function, ` + `${(INVOICE_LINES * 2).toLocaleString("en-US")} written compensation paths a day; compensation calls run ` + `${(failed * 0.75).toFixed(0)}/day, ${(0.75 * KK2).toFixed(4)} per workflow`); ``` ``` daily workflows 4000, KK2 = 0.02 -> failed 80/day ordering uncompensated steps/day a year notification last 60.0 21900 notification third 80.0 29200 attempts stuck work/day a year rollback s intermediate s/day h/day from stuck 1 2.979 1087 0.375 30.0 11.917 2 0.171 62 0.385 30.8 0.685 3 0.006 2 0.385 30.8 0.026 at one attempt, from stuck work / from rollback = 1430x DD6 sensitivity: at a round duration of 1 s instead of 0.25 s, intermediate state is 120.0 s/day, ratio 358x failure-free day's cost: 2 of the 4 steps carry a compensation function, 8,000 written compensation paths a day; compensation calls run 60/day, 0.0150 per workflow ``` #### The Numbers for Two Days **The failure-free day's cost sits in the code path.** Two of the four steps must carry a compensation function, and those functions sit in every workflow definition: 8000 written compensation paths a day for 4000 workflows. How often they run is very small — 0.0150 per workflow, 60 calls a day. The cost is not a runtime load, it is **untested code**: alongside a path that runs 21,900 times a year, the codebase carries a path that runs with a probability of 1.5 percent. Closing the period being uncompensatable also produces a constraint on the failure-free day: no correction can be made on a closed period. **The failure day's gain is two numbers.** At KK2 = 0.02, 80 workflows fail a day. With the ordering rule applied, the uncompensated step count is 60 a day, 21,900 a year; if the notification is moved to third place, it is 80 and 29,200. Ordering alone makes a difference of 7300 uncompensated steps a year. Adding one compensation attempt brings stuck work down from 2.979 to 0.171 a day, from 1087 to 62 a year. The intermediate state window has two sources, and their sizes are not comparable. Rollback itself produces a total of **30.0 seconds** of intermediate state a day. Stuck work produces **11.917 hours** a day — 1430 times as much. Even at four times DD6, the ratio stays at 358x, so the result is not sensitive to the round-duration assumption. **The intermediate state window is set by compensation's own failure, not by rollback's speed**; a team that wants to shorten the window should tune the compensation attempt count, not the round duration. #### Summary - Only two of the twenty-four orderings satisfy the dependencies, and both start with the uncompensatable `close-period` step: the move-to-the-end rule cannot be applied to this step. - When the failure occurs at the second, third, or fourth step, the uncompensated step count is 1 in every case; what saves a step that cannot be moved to the end is not ordering but designing the step to be compensatable. - The notification's position makes a difference of 7300 uncompensated steps a year: 60/day and 21,900/year at the end, 80/day and 29,200/year at third place. - Compensation carries its own failure: at DD5 = 0.05, one attempt leaves 3.724 percent of jobs stuck; a second attempt brings the ratio down to 0.00214 and raises calls per job from 0.750 to 0.790. - The overwhelming share of the intermediate state window comes from stuck work, not rollback: 30.0 seconds a day against 11.917 hours, 1430 times; even at four times DD6, still 358 times. - The failure-free day's cost is 8000 written compensation paths, and only 60 of them run a day; what the codebase carries is not runtime load but rarely run code. #### Next Step Compensation kept the workflow's **steps** correct. The **data** the same workflow carries was never in question. When the carrier reports a delivery, it attaches proof of delivery alongside it: the recipient's signature or a photo of the delivery point. This addition turns the state event's record into something far larger than K01's 220 bytes, and that something passes through the queue, gets copied to every consumer, and is carried again on every redelivery. The next lesson takes on pulling the large payload out of the message: the message's size, the bytes crossing the queue, how long the payload lives in a separate store, and how many times the reference left behind after the payload is deleted dangles. ### Claim Check Pattern Source: https://academia.sh/en/courses/resilience-patterns/claim-check-pattern Moving a large payload outside the message: proof of delivery raising the message to 40,220 bytes, the claim check lowering message bytes by 141.6 times, the queue's backlog dropping from 3016.5 MB to 21.3 MB, the proof never shrinking on the storage side, and the orphaned payload and dangling claim check that splitting a single write into two brings. The previous two lessons kept the workflow's steps correct: duplication was suppressed, half-finished steps were balanced. The **data** the workflow carries was never in question. When the carrier reports a delivery, it attaches proof of delivery alongside it — the recipient's signature or a photo of the delivery point. This addition moves K01's 220-byte state event record to a different order of magnitude, and that byte count passes through the queue, gets copied to every consumer, and is carried again on every redelivery. The **claim check** is a layout that writes the large payload to a separate store and lets the message carry only a reference to that payload. This lesson's measures are four: the message's size, the bytes crossing the queue, how long the payload lives in a separate store, and the check dangling — the payload deleted, the check still in hand. #### Two Writes, No Shared Transaction Boundary **DD8 — proof of delivery is 40,000 bytes.** Rationale: the proof is a compressed signature image, two orders of magnitude above K01's 900-byte shipment record. Its sensitivity is already visible in the message byte ratio. **DD9 — a claim check record is 64 bytes**: the store name, the object key, and a digest. The pattern's hidden cost starts here. The attached layout has a single write: the message goes to the queue and the proof is inside it. The claim-check layout has **two writes** — the proof to the store first, then the message to the queue — and the two share no transaction boundary. This is exactly The Data Access Layer and Business Logic course's distributed transaction problem. Two new failure modes are born: the **orphaned payload** (the proof was written, the message was not) and the **dangling claim check** (the message is in hand, the proof is deleted). ```js // claim-check/queue.mjs — the in-process model of the event stream carrying proof of delivery. // There is no real queue, store, or network: the queue is an array, the payload store a set, bytes are counted. export const EVENT = 220, PROOF = 40_000, CHECK = 64; // K01 V6; DD8 proof; DD9 check record export function generator(seed) { // 32-bit linear congruential generator, seed exposed let s = seed >>> 0; return () => { s = (Math.imul(s, 1103515245) + 12345) >>> 0; return s / 4294967296; }; } // layout: "attached" = the proof travels inside the message; "check" = the proof sits in a store, the message carries a check // deletion: "immediate" = the consumer deletes on processing; "lifetime" = deletion waits out a lifetime export function run({ n, layout, deletion, producerCrash, unackedCrash, seed }) { const rnd = generator(seed); const usesCheck = layout === "check"; const message = usesCheck ? EVENT + CHECK : EVENT + PROOF; const store = new Set(), queue = []; const s = { message, bytesMoved: 0, deliveries: 0, lostEvents: 0, orphanedPayload: 0, danglingCheck: 0, processed: 0, remainingInStore: 0 }; for (let i = 0; i < n; i += 1) { if (usesCheck) store.add(i); // the payload is written first const crashed = rnd() < producerCrash; // then the message is written; a crash can land in between if (crashed) { if (usesCheck) s.orphanedPayload += 1; else s.lostEvents += 1; continue; } queue.push({ no: i, attempt: 0 }); } while (queue.length > 0) { const m = queue.shift(); s.bytesMoved += message; s.deliveries += 1; // every delivery carries the message from scratch if (usesCheck && !store.has(m.no)) { s.danglingCheck += 1; continue; } s.processed += 1; if (usesCheck && deletion === "immediate") store.delete(m.no); if (rnd() < unackedCrash && m.attempt === 0) queue.push({ no: m.no, attempt: 1 }); } s.remainingInStore = usesCheck ? store.size : 0; return s; } ``` The failure scenario is named directly. **DD10 — in the failure window, the probability that the producer crashes between its two writes is 0.005, the probability that the consumer crashes before acknowledgment is 0.01.** Rationale: the first window is the short gap between two remote writes, the second is the gap between processing and acknowledgment, and the second is longer. Both are assumptions, not added to K01's table. ```js // claim-check/measure.mjs — four layouts, without and with failure; message bytes and dangling counted import { run, EVENT, PROOF, CHECK } from "./queue.mjs"; const N = 100_000; const CONDITION = [["failure-free day", 0, 0], ["failure window (DD10)", 0.005, 0.01]]; const LAYOUT = [["attached", "-"], ["check", "immediate"], ["check", "lifetime"]]; console.log(`message bytes: attached ${EVENT + PROOF}, check ${EVENT + CHECK} ` + `(K01 V6 = ${EVENT} bytes event, DD8 = ${PROOF} bytes proof, DD9 = ${CHECK} bytes check)`); for (const [name, producerCrash, unackedCrash] of CONDITION) { console.log(`\n${name}: ${N} delivery events, producer crash ${producerCrash}, unacked crash ${unackedCrash}`); console.log(`${"layout".padEnd(18)}${"deliveries".padStart(11)}${"MB moved".padStart(11)}` + `${"lost events".padStart(13)}${"orphaned".padStart(11)}${"dangling".padStart(11)}` + `${"processed".padStart(11)}${"in store".padStart(10)}`); for (const [layout, deletion] of LAYOUT) { const r = run({ n: N, layout, deletion, producerCrash, unackedCrash, seed: 20260730 }); console.log(`${`${layout} ${deletion}`.padEnd(18)}${String(r.deliveries).padStart(11)}` + `${(r.bytesMoved / 1e6).toFixed(2).padStart(11)}${String(r.lostEvents).padStart(13)}` + `${String(r.orphanedPayload).padStart(11)}${String(r.danglingCheck).padStart(11)}` + `${String(r.processed).padStart(11)}${String(r.remainingInStore).padStart(10)}`); } } ``` ``` message bytes: attached 40220, check 284 (K01 V6 = 220 bytes event, DD8 = 40000 bytes proof, DD9 = 64 bytes check) failure-free day: 100000 delivery events, producer crash 0, unacked crash 0 layout deliveries MB moved lost events orphaned dangling processed in store attached - 100000 4022.00 0 0 0 100000 0 check immediate 100000 28.40 0 0 0 100000 0 check lifetime 100000 28.40 0 0 0 100000 100000 failure window (DD10): 100000 delivery events, producer crash 0.005, unacked crash 0.01 layout deliveries MB moved lost events orphaned dangling processed in store attached - 100495 4041.91 492 0 0 100495 0 check immediate 100495 28.54 0 492 987 99508 492 check lifetime 100495 28.54 0 492 0 100495 100000 ``` These numbers are in the **measurement** class: they come from a process-internal model and a generator seeded at 20260730. #### The Deletion Policy Is a Correctness Decision The top table gives the message bytes: the attached layout moves 4022.00 MB for a hundred thousand events, the claim-check layout 28.40 MB. The ratio is the message byte ratio, and with no failure there is no other difference. The bottom table makes the real distinction, and the column that deserves attention is `processed`. In the attached layout, the producer's crash **drops 492 events entirely** — the proof and the event were in a single write, and both were gone together. In the claim-check layout, the same crash still loses the events, but it leaves 492 orphaned payloads in the store; the loss is equal, and there is now garbage on top of it. The two deletion policies behave differently under the same failure. Under **immediate deletion**, 987 redeliveries cannot find the proof and turn into dangling checks; the processed count sits at 99,508, meaning every event is processed **exactly once**. Under **lifetime-based deletion**, dangling checks are zero, but processed is 100,495: all 987 redeliveries find the proof and redo the work. This is the pattern's least-noticed consequence. The moment the payload is deleted turns, without anyone intending it, into an idempotency check: immediate deletion blocks duplicate processing but turns redelivery into an error; lifetime-based deletion runs the redelivery and hands the duplicated side effect off to **the first lesson's ledger**. The correct setup keeps the two separate — the idempotency decision is made in the ledger, the deletion decision in the store; the payload's lifetime is not used as a correctness mechanism. #### Back to the Numbers **DD11 — proof lifetime is 7 days.** Rationale: a delivery dispute arrives within the business day following the delivery day, and the proof is requested inside that window; proof is not kept without a dispute. Its sensitivity is given at 1 and 30 days. ```js // claim-check/cost.mjs — converting the measured message bytes into queue, storage, and dangling numbers at K01 scale const V3 = 400_000, V8 = 3, DAY_SEC = 86_400; // K01: daily shipments, peak factor const WRITE_MBIT = 0.17, GROWTH_MB = 976, STORAGE_GB = 712.48; // K01 computed values const BACKLOG = 525_000, EVENT_SHARE = 7; // M19/K03 message queues lesson: highest backlog const ATTACHED = 40_220, CHECKED = 284, PROOF = 40_000; // measure.mjs measurement; DD8 const peakDeliveries = (V3 / DAY_SEC) * V8; // every shipment produces one delivery event console.log(`peak delivery events ${peakDeliveries.toFixed(4)} events/s (K01: ${V3} shipments/day, peak factor ${V8})`); console.log(`\n${"layout".padEnd(12)}${"message bytes".padStart(14)}${"queue Mbit/s".padStart(15)}` + `${"of K01 write entry".padStart(21)}${"backlog MB".padStart(13)}`); for (const [name, b] of [["attached", ATTACHED], ["check", CHECKED]]) { const mbit = (peakDeliveries * b * 8) / 1e6, backlog = ((BACKLOG / EVENT_SHARE) * b) / 1e6; console.log(`${name.padEnd(12)}${String(b).padStart(14)}${mbit.toFixed(4).padStart(15)}` + `${`${(mbit / WRITE_MBIT).toFixed(2)}x`.padStart(21)}${backlog.toFixed(1).padStart(13)}`); } console.log(`queue byte ratio ${(ATTACHED / CHECKED).toFixed(1)}x; ` + `backlog ${(BACKLOG / EVENT_SHARE).toLocaleString("en-US")} messages with proof (M19/K03: ${BACKLOG.toLocaleString("en-US")} events / ${EVENT_SHARE})`); console.log(`\n${"DD11 lifetime".padStart(15)}${"proof in store".padStart(16)}${"store GB".padStart(11)}` + `${"of K01 storage".padStart(16)}${"of daily growth".padStart(17)}`); for (const days of [1, 7, 30]) { const count = V3 * days, gb = (count * PROOF) / 1e9; console.log(`${`${days} d`.padStart(15)}${count.toLocaleString("en-US").padStart(16)}` + `${gb.toFixed(2).padStart(11)}${`${((100 * gb) / STORAGE_GB).toFixed(2)}%`.padStart(16)}` + `${`${((gb * 1000) / GROWTH_MB).toFixed(2)}x`.padStart(17)}`); } const DANGLING = 987 / 100_000, ORPHANED = 492 / 100_000; // measure.mjs measurement console.log(`\nfailure window at K01 scale (${V3.toLocaleString("en-US")} delivery events/day):`); console.log(`dangling checks ${(V3 * DANGLING).toFixed(0)}/day, orphaned payloads ${(V3 * ORPHANED).toFixed(0)}/day`); console.log(`orphaned payload bytes ${((V3 * ORPHANED * PROOF) / 1e6).toFixed(1)} MB/day, ` + `without a lifetime, a year gives ${((V3 * ORPHANED * PROOF * 365) / 1e9).toFixed(2)} GB`); console.log(`in the attached layout, the same crash drops ${(V3 * ORPHANED).toFixed(0)} events entirely`); ``` ``` peak delivery events 13.8889 events/s (K01: 400000 shipments/day, peak factor 3) layout message bytes queue Mbit/s of K01 write entry backlog MB attached 40220 4.4689 26.29x 3016.5 check 284 0.0316 0.19x 21.3 queue byte ratio 141.6x; backlog 75,000 messages with proof (M19/K03: 525,000 events / 7) DD11 lifetime proof in store store GB of K01 storage of daily growth 1 d 400,000 16.00 2.25% 16.39x 7 d 2,800,000 112.00 15.72% 114.75x 30 d 12,000,000 480.00 67.37% 491.80x failure window at K01 scale (400,000 delivery events/day): dangling checks 3948/day, orphaned payloads 1968/day orphaned payload bytes 78.7 MB/day, without a lifetime, a year gives 28.73 GB in the attached layout, the same crash drops 1968 events entirely ``` #### The Numbers for Two Days **The failure-free day's cost has three line items.** Every delivery event now carries two writes and one extra read on the consumer side; the single write's atomicity is gone. At DD11 = 7 days, the proof sits in the store at 112.00 GB, 15.72 percent of K01's 712.48 GB storage line. And on the storage side, the pattern has **no gain at all**: the proof occupies the same 40,000 bytes whether it is attached or checked. The claim check is a **transport** decision, not a storage decision. **The failure day's gain sits in the queue.** In the attached layout, the peak delivery flow puts 4.4689 Mbit/s on the queue — 26.29 times K01's 0.17 Mbit/s write entry. In the claim-check layout, it is 0.0316 Mbit/s, 0.19 times the write entry. The real difference shows up in the backlog: given that one in seven of the 525,000 events The Application Layer and Service Interaction course measured is a message carrying proof, the queue has to hold **3016.5 MB** in the attached layout and **21.3 MB** in the claim-check layout. In a design where the queue is a bounded buffer, this 141.6-fold difference is the difference in how many events the same bound can hold. The gain's counterpart is new failure modes. At K01 scale, the failure window produces 1968 orphaned payloads a day (78.7 MB) and, under the immediate deletion policy, 3948 dangling checks. Without a lifetime, orphaned payloads climb to 28.73 GB a year; **DD11 is not a storage parameter, it is a garbage collection mechanism**. Dangling checks, on the other hand, are not data loss — all 3948 are redeliveries of work already processed — but they show up as errors on the consumer side, and if they are not routed off the dead-letter path, they hide real errors. #### Summary - Proof of delivery raises the message from 220 bytes to 40,220 bytes; the claim check brings it down to 284 bytes, a ratio of 141.6x. - Queue load: the peak delivery flow is 4.4689 Mbit/s in the attached layout (26.29 times K01's write entry), 0.0316 Mbit/s in the claim-check layout (0.19 times). - The difference grows in the backlog: a backlog of 75,000 messages with proof holds 21.3 MB instead of 3016.5 MB. - The pattern has no gain on the storage side: at DD11 = 7 days the proof sits at 112.00 GB, 15.72 percent of K01's storage; the claim check is a transport decision, not a storage decision. - The pattern splits a single write into two and introduces two new failure modes: 1968 orphaned payloads a day (78.7 MB, 28.73 GB a year without a lifetime) and 3948 dangling checks. - The deletion policy is silently a correctness decision: under immediate deletion, processed is 99,508 (exactly once); under lifetime-based deletion, it is 100,495 (987 duplicate processing runs); the latter depends on the idempotency ledger. #### Next Step The proof left the queue, but the whole path still runs through the service. The carrier uploads the proof to the service, the service writes it to the store; when a recipient wants to see the proof, the service reads it from the store and passes it to the client. The 40,000 bytes removed from the queue still pass through the service's request path twice, and when the store slows down, they tie up the service's own workers. The next lesson takes on removing this byte count from the service as well: the limited-time authorization granted for direct client access to the resource, the byte count that never passes through the service, the authorization window it opens, and what narrowing that scope costs. ### Valet Key Pattern Source: https://academia.sh/en/courses/resilience-patterns/valet-key-pattern Time-limited authorization for direct resource access: taking proof bytes off the service's path drops per-transfer bytes from 80,000 to 512, the held-slot pool saturating at 64 and dropping 28 transfers when the store slows down, the open authorization window growing linearly with key duration, and scope narrowing's payoff being non-linear. The claim check pattern took the proof out of the queue, but the whole path still runs through the service: the carrier uploads the proof to the service, the service writes it to the store, and when the recipient requests the proof the service reads it back and transfers it to the client. The 40,000 bytes removed from the queue still pass through twice on the on-demand path. The **valet key** pattern grants the client authorization narrow and short-lived enough to reach the resource directly, letting data flow around the service. The Authentication and Authorization course established the signed link, scope, and time-limited token; the verification mechanics are not repeated here. This lesson measures three things: the bytes that **do not pass** through the service, the **authorization window** it opens, and the **cost** of narrowing scope. #### Getting the Byte Off the Service **DD12 — delivery proof view rate 0.02.** Rationale: proof is read only on dispute or audit, not on every shipment; its sensitivity already enters the measure as a rate. **DD13 — proof transfer pool 64 slots**, with a failure scenario: **the proof store slows down**, response time rising from 1 round to 20 rounds and lasting 80 rounds. Round duration is this course's DD6 assumption (0.25 seconds), so the slowdown lasts 20 seconds, taken twice a month. The rig is an in-process model: there is no store, network, or client; a round is an abstract step, a slot is a counter, and bytes are a counted quantity. ```js // valet/path.mjs — in-process model of the two paths for proof transfer. // There is no real store, network, or client: a round is an abstract step, a slot is a counter, bytes are counted. export const PROOF = 40_000, KEY = 512; // DD8 proof bytes; DD14 valet key response // path: "proxy" = bytes pass through the service; "valet" = the service only issues keys. // Each round: incoming transfers request a slot, the store holds the slot for its duration, then releases it. export function run({ rounds, arrivals, pool, normalDuration, failureDuration, failureStart, failureEnd, path }) { const held = []; const s = { accepted: 0, rejected: 0, peakSlots: 0, serviceBytes: 0, slotTurns: 0, keys: 0 }; let remainder = 0; for (let t = 0; t < rounds; t += 1) { for (let i = held.length - 1; i >= 0; i -= 1) if (held[i] <= t) held.splice(i, 1); const duration = t >= failureStart && t < failureEnd ? failureDuration : normalDuration; remainder += arrivals; const n = Math.floor(remainder); remainder -= n; for (let i = 0; i < n; i += 1) { if (path === "valet") { s.accepted += 1; s.keys += 1; s.serviceBytes += KEY; continue; } if (held.length >= pool) { s.rejected += 1; continue; } held.push(t + duration); s.accepted += 1; s.serviceBytes += PROOF * 2; // received from the client, written to the store } s.peakSlots = Math.max(s.peakSlots, held.length); s.slotTurns += held.length; } return s; } ``` ```js // valet/measure.mjs — held slots and dropped transfers for both paths when the proof store slows down import { run, PROOF, KEY } from "./path.mjs"; const V3 = 400_000, V8 = 3, DAY = 86_400; // K01: daily shipments, peak multiplier const DD12 = 0.02; // proof view rate const ROUND_S = 0.25, ROUNDS = 320, POOL = 64; // DD6 round duration; DD13 pool const uploadRate = (V3 / DAY) * V8, readRate = ((V3 * DD12) / DAY) * V8; const rate = uploadRate + readRate; const arrivals = rate * ROUND_S; console.log(`peak proof transfer ${rate.toFixed(4)}/s (upload ${uploadRate.toFixed(4)}, ` + `read ${readRate.toFixed(4)}; DD12 = ${DD12})`); console.log(`${ROUNDS} rounds x ${ROUND_S} s = ${(ROUNDS * ROUND_S).toFixed(0)} s; ${arrivals.toFixed(4)} transfers per round`); console.log(`DD13: store response rises from 1 round to 20 rounds, lasting 80 rounds (${(80 * ROUND_S).toFixed(0)} s)`); console.log(`\n${"path".padEnd(8)}${"state".padEnd(10)}${"accepted".padStart(10)}${"rejected".padStart(10)}` + `${"peak slots".padStart(12)}${"avg. slots".padStart(12)}${"service MB".padStart(12)}`); for (const path of ["proxy", "valet"]) for (const [label, failureDuration, failureStart, failureEnd] of [["healthy", 1, ROUNDS, ROUNDS], ["slowdown", 20, 80, 160]]) { const r = run({ rounds: ROUNDS, arrivals, pool: POOL, normalDuration: 1, failureDuration, failureStart, failureEnd, path }); console.log(`${path.padEnd(8)}${label.padEnd(10)}${String(r.accepted).padStart(10)}` + `${String(r.rejected).padStart(10)}${String(r.peakSlots).padStart(12)}` + `${(r.slotTurns / ROUNDS).toFixed(2).padStart(12)}${(r.serviceBytes / 1e6).toFixed(3).padStart(12)}`); } console.log(`\nbytes passing through the service per transfer: proxy ${PROOF * 2}, valet ${KEY} ` + `(ratio ${((PROOF * 2) / KEY).toFixed(1)}x)`); ``` ``` peak proof transfer 14.1667/s (upload 13.8889, read 0.2778; DD12 = 0.02) 320 rounds x 0.25 s = 80 s; 3.5417 transfers per round DD13: store response rises from 1 round to 20 rounds, lasting 80 rounds (20 s) path state accepted rejected peak slots avg. slots service MB proxy healthy 1133 0 4 3.54 90.640 proxy slowdown 1105 28 64 18.65 88.400 valet healthy 1133 0 0 0.00 0.580 valet slowdown 1133 0 0 0.00 0.580 bytes passing through the service per transfer: proxy 80000, valet 512 (ratio 156.3x) ``` These numbers belong to the **measurement** class and come from an in-process model; their inputs are K01's computed values and this lesson's assumptions. #### The Slowing Store's Spread Into the Service The healthy rows give the quiet difference between the two paths: in eighty seconds, the proxy path passes 90.640 MB through the service, the valet path 0.580 MB. That is 80,000 bytes against 512 bytes per transfer, a 156.3x ratio. The proxy path's average held slots is 3.54 — the product of the arrival rate and the service duration — small next to the 64-slot pool on a healthy day. The slowdown row makes the real distinction. When the store's response time stretches to twenty times its normal length, average held slots rise to 18.65, the peak to 64 — the pool **saturates** — and 28 transfers are rejected. Accepted transfers drop from 1133 to 1105. On the valet path, under the same slowdown, held slots are 0, rejected 0, accepted 1133: because the service never touches the proof's bytes, the store's slowdown does not show up in the service's pool. The client still talks to a slow store, but it is the one waiting; **the blast radius stays outside the service.** A saturated pool affects not only the proof transfer but every other job sharing that pool; separating pools is the bulkhead pattern's question, from this course's Fault Isolation topic. #### Scope and Duration The pattern's counterpart is an authorization, measured on two axes: how long it stays open and how many objects it opens. **DD14 — valet key duration 300 seconds.** Rationale: a proof upload or view is a single user action and does not take minutes; the duration is kept long enough to leave room for a retry. Its sensitivity is given at 60 and 3600 seconds. **DD15 — a key's leak probability is 10⁻⁶**; a key can end up in a log file, browser history, or a shared link. Both are assumptions. ```js // valet/scope.mjs — narrowing scope's effect on key issuance, authorization window, and exposure const V3 = 400_000, V8 = 3, V12 = 730, DAY = 86_400; // K01: shipments, peak multiplier, retention const SELLERS = 4000, EDGE_PEAK = 513.89; // K01 computed value: seller count, edge peak const DD14 = 300, DD15 = 1e-6; // key duration (s), key leak probability const DAILY = V3 / SELLERS; // daily shipments per seller = 100 // Scope: [label, daily key issuance, objects one key can reach] const SCOPE = [["single object", V3, 1], ["seller-day prefix", SELLERS, DAILY], ["seller bucket", SELLERS, DAILY * V12]]; console.log(`DD14 key duration ${DD14} s, DD15 leak probability ${DD15}; ` + `${V3.toLocaleString("en-US")} proofs a day, ${SELLERS} sellers, ${DAILY} shipments per seller`); console.log(`\n${"scope".padEnd(18)}${"keys/day".padStart(13)}${"peak keys/s".padStart(16)}` + `${"of K01 edge".padStart(12)}${"open window".padStart(14)}${"surface".padStart(9)}` + `${"object-sec/day".padStart(18)}`); for (const [label, issuance, surface] of SCOPE) { const peak = (issuance / DAY) * V8, openWindow = peak * DD14; const exposure = issuance * DD15 * surface * DD14; console.log(`${label.padEnd(18)}${issuance.toLocaleString("en-US").padStart(13)}` + `${peak.toFixed(4).padStart(16)}${`${((100 * peak) / EDGE_PEAK).toFixed(3)}%`.padStart(12)}` + `${openWindow.toFixed(1).padStart(14)}${surface.toLocaleString("en-US").padStart(9)}` + `${exposure.toFixed(1).padStart(18)}`); } console.log(`\n${"DD14 sensitivity".padStart(17)}${"single-object open".padStart(20)}` + `${"seller-day open".padStart(17)}${"bucket object-sec".padStart(19)}`); for (const s of [60, 300, 3600]) { const t = (V3 / DAY) * V8 * s, g = (SELLERS / DAY) * V8 * s; console.log(`${`${s} s`.padStart(17)}${t.toFixed(1).padStart(20)}${g.toFixed(1).padStart(17)}` + `${(SELLERS * DD15 * DAILY * V12 * s).toFixed(0).padStart(19)}`); } const [t1, g1, k1] = SCOPE.map(([, u, y]) => u * DD15 * y * DD14); console.log(`\nbucket -> seller-day: exposure falls ${(k1 / g1).toFixed(0)}x, ` + `key issuance unchanged`); console.log(`seller-day -> single object: exposure changes ${(g1 / t1).toFixed(2)}x, ` + `key issuance rises ${(V3 / SELLERS).toFixed(0)}x`); console.log(`worst case is not the same: objects opened by one leaked key ` + `${SCOPE.map(([label, , y]) => `${label} ${y.toLocaleString("en-US")}`).join(", ")}`); ``` ``` DD14 key duration 300 s, DD15 leak probability 0.000001; 400,000 proofs a day, 4000 sellers, 100 shipments per seller scope keys/day peak keys/s of K01 edge open window surface object-sec/day single object 400,000 13.8889 2.703% 4166.7 1 120.0 seller-day prefix 4,000 0.1389 0.027% 41.7 100 120.0 seller bucket 4,000 0.1389 0.027% 41.7 73,000 87600.0 DD14 sensitivity single-object open seller-day open bucket object-sec 60 s 833.3 8.3 17520 300 s 4166.7 41.7 87600 3600 s 50000.0 500.0 1051200 bucket -> seller-day: exposure falls 730x, key issuance unchanged seller-day -> single object: exposure changes 1.00x, key issuance rises 100x worst case is not the same: objects opened by one leaked key single object 1, seller-day prefix 100, seller bucket 73,000 ``` #### Narrowing Scope's Payoff Is Not Linear The table separates narrowing scope into two steps that do not resemble each other. **Stepping down from the bucket scope to the seller-day prefix gains 730x** and costs nothing: key issuance is 4000 a day in both, the open window 41.7 keys in both. Expected exposure drops from 87,600 object-seconds to 120.0. The narrowing is free, with no reason to skip it. **Stepping down from the seller-day prefix to the single object does not change the expected exposure at all** — both rows show 120.0 object-seconds — but it raises key issuance 100x: 400,000 a day instead of 4000, 13.8889 keys/s at peak instead of 0.1389. That is 2.703 percent of K01's 513.89 requests/s at the edge. The expected value holds because the arithmetic cancels: scope narrows a hundredfold, issued keys rise a hundredfold, and the product stays constant. The decision can still favor the single object on other grounds: the **worst case** is not equal. A single leaked key opens 1 object under the single-object scope, 100 under the seller-day prefix, 73,000 under the bucket scope. Expected exposure is the same, but the tail is not, and the largest damage a single event can carry spans three orders of magnitude. On the duration axis, the result is simpler: the open window grows linearly with DD14. Shortening the key duration to 60 seconds drops the single-object scope's concurrently valid keys from 4166.7 to 833.3; stretching it to 3600 seconds raises it to 50,000. Under the bucket scope, an hour-long key produces 1,051,200 object-seconds a day. **Shortening the duration pays off linearly at every scope; narrowing the scope pays off only at the first step.** #### The Numbers for Two Days **The failure-free day's cost has two items.** The service issues a key for every byte it does not carry: 400,000 signings a day under the single-object scope, 13.8889 requests/s at peak, 2.703 percent of the edge rate. And a permanently open authorization window is carried — under the single-object scope, 4166.7 keys are concurrently valid for the 300-second duration. The proxy path has no such window; access is bounded by the request itself. **The failing day's gain is isolation.** When the proof store slows down twentyfold, the proxy path saturates its pool at 64 slots and drops 28 transfers in eighty seconds; on the valet path, dropped transfers are 0, held slots 0. The difference is not a speed difference, it is a spread difference: on the proxy path, the store's failure turns into a resource problem for the service; on the valet path, it stays with the client. The bytes that do not pass through the service total 90.060 MB in eighty seconds (90.640 − 0.580), and those bytes are the source of the held slots during the failure. #### Summary - Bytes passing through the service per transfer are 80,000 on the proxy path and 512 on the valet path; a 156.3x ratio, 90.640 MB against 0.580 MB over an eighty-second run. - When the store slows down twentyfold, the proxy path's average held slots reach 18.65, the peak 64 (the pool saturates), and 28 transfers are dropped; on the valet path, held slots are 0, dropped 0. - The failure-free day's cost is key issuance: 400,000 signings a day under the single-object scope, 13.8889 requests/s at peak, 2.703 percent of K01's edge rate. - The open authorization window is linear with duration: at DD14 = 60/300/3600 seconds, 833.3 / 4166.7 / 50,000 keys are concurrently valid under the single-object scope. - Narrowing scope's first step is free: bucket → seller-day drops expected exposure from 87,600 to 120.0 object-seconds and does not change key issuance. - The second step does not change the expected value (120.0 → 120.0) but raises key issuance 100x; the justification is the worst case — objects opened by one leaked key are 73,000 / 100 / 1. #### Next Step The proof path now rests on three decisions: where the load stops, when the claim check is deleted, and what scope and duration the key is valid for. All three numbers are embedded in code as constants: lifetime, key duration, pool size. If a failure calls for shortening the key duration or closing the proof path entirely, the only way is a redeploy, and a redeploy draws from the outage budget's planned share. The next lesson takes these settings out of the application: how long a change takes to reach every copy, the cost of a cached read, the behavior when the store itself goes down, and the blast radius of a value set wrong. ### External Configuration Store Source: https://academia.sh/en/courses/resilience-patterns/external-configuration-store Centralizing runtime settings: a change propagating to three replicas in 25 seconds, caching cutting store reads 5139x, a wrong value under staged rollout producing 20,556 faulty responses instead of 61,668, and running on the last known good value zeroing out a 40-second equivalent outage when the store goes down. The previous three lessons' decisions sit as numbers embedded in code: the claim check's lifetime is a constant, the key duration is a constant, the transfer pool's size is a constant, the idempotency ledger's window is a constant. If a failure calls for shortening the key duration or closing the proof path entirely, the only way is a redeploy, and a redeploy draws from the 15.0-minute planned share within K01's monthly 43.2-minute outage budget. The **external configuration store** pattern keeps runtime settings in a single place outside the application unit, with the replicas reading from there. The Application Scaffolding course established defining configuration by schema, validating it at startup, and separating out secrets; those rules are not repeated here. This lesson's question is **runtime**: how long a change takes to reach the replicas, how much caching cuts the store's load, what happens when the store goes down, and how many replicas a wrongly set value spreads to. #### Propagation and the Mixed-Value Window The rig is an in-process model: there is no real store, network, or replica process; time advances in modeled seconds, and no number depends on this machine's speed. Replica count and per-replica request rate are taken from the Stateless Services lesson in the Application Layer and Service Interaction course: 3 replicas, 171.30 requests/s per replica. **DD16 — refresh interval 30 seconds.** Rationale: the longest acceptable wait for a setting to take effect is kept under a minute. Its sensitivity is given at 5 and 300 seconds. **DD17 — a wrong value's detection time is 120 seconds.** Rationale: a wrong value shows up as a spike in the error rate, and confirming it takes time; how the spike is detected is not this course's subject — it belongs to the Performance Anti-Patterns and Monitoring course. ```js // config/store.mjs — in-process model of the external configuration store and the replica caches. // There is no real store, network, or replica process; time advances in modeled seconds, counters are explicit. // rollout: "all-at-once" = the value is valid for every replica; "staged" = replica 0 first, then the rest once confirmed export function target(replicaIndex, t, { rollout, wrongAt, detection }) { const inWindow = t >= wrongAt && t < wrongAt + detection; if (rollout === "staged") return replicaIndex === 0 && inWindow ? "wrong" : "correct"; return inWindow ? "wrong" : "correct"; } export function run(a) { const { duration, replicas, interval, stagger, replicaRate, downAt, upAt, restartAt, diskCache } = a; const value = Array.from({ length: replicas }, () => "correct"); const up = Array.from({ length: replicas }, () => true); const s = { faultyResponses: 0, droppedRequests: 0, storeReads: 0, failedReads: 0, firstSeen: -1, lastSeen: -1, mixed: 0, propagation: -1 }; for (let t = 0; t < duration; t += 1) { if (t === restartAt) // a replica restarts after failover up[replicas - 1] = diskCache || !(t >= downAt && t < upAt); if (t >= upAt && !up[replicas - 1]) up[replicas - 1] = true; for (let k = 0; k < replicas; k += 1) { if (t >= k * stagger && (t - k * stagger) % interval === 0) { s.storeReads += 1; if (t >= downAt && t < upAt) s.failedReads += 1; // last known good value is kept else value[k] = target(k, t, a); } if (!up[k]) { s.droppedRequests += replicaRate; continue; } if (value[k] === "wrong") s.faultyResponses += replicaRate; } const n = value.filter((d) => d === "wrong").length; const expected = value.map((_, k) => target(k, t, a)).filter((d) => d === "wrong").length; if (n > 0 && s.firstSeen < 0) s.firstSeen = t; if (n > 0) s.lastSeen = t; if (n > 0 && n < replicas) s.mixed += 1; if (s.propagation < 0 && expected > 0 && n === expected) s.propagation = t - a.wrongAt; } return s; } ``` ```js // config/measure.mjs — blast radius of a wrong value, and behavior when the store goes down import { run } from "./store.mjs"; const REPLICAS = 3, REPLICA_RATE = 171.30; // M19/K03: 3 replicas, 171.30 requests/s per replica const EDGE_PEAK = 513.89, BUDGET = 28.2 * 60; // K01: peak requests/s at the edge, monthly failure share const INTERVAL = 30, DETECTION = 120, DURATION = 600; // DD16 refresh interval, DD17 detection time const BASE = { duration: DURATION, replicas: REPLICAS, interval: INTERVAL, replicaRate: REPLICA_RATE, wrongAt: 65, detection: DETECTION, downAt: DURATION, upAt: DURATION, restartAt: -1, diskCache: true }; console.log(`${REPLICAS} replicas, ${REPLICA_RATE} requests/s per replica (M19/K03), edge peak ${EDGE_PEAK}`); console.log(`DD16 refresh interval ${INTERVAL} s, DD17 detection time ${DETECTION} s; ` + `wrong value written at second 65`); console.log(`\n${"rollout".padEnd(13)}${"stagger".padStart(9)}${"propagation".padStart(13)}` + `${"first seen".padStart(12)}${"last seen".padStart(11)}${"mixed s".padStart(9)}` + `${"faulty replies".padStart(16)}${"affected replicas".padStart(20)}`); for (const rollout of ["all-at-once", "staged"]) for (const stagger of [0, 10]) { const r = run({ ...BASE, rollout, stagger }); console.log(`${rollout.padEnd(13)}${String(stagger).padStart(9)}${String(r.propagation).padStart(13)}` + `${String(r.firstSeen).padStart(12)}${String(r.lastSeen).padStart(11)}` + `${String(r.mixed).padStart(9)}${r.faultyResponses.toFixed(0).padStart(16)}` + `${String(rollout === "all-at-once" ? REPLICAS : 1).padStart(20)}`); } // Store outage: goes down at second 200, 180 s down; a replica restarts after failover at second 260. console.log(`\nstore outage: down for 180 s; a replica restarts after failover at second 260`); console.log(`${"behavior".padEnd(24)}${"failed reads".padStart(17)}${"dropped requests".padStart(19)}` + `${"equiv. outage s".padStart(19)}${"of failure share".padStart(19)}`); for (const [label, diskCache] of [["last known good value", true], ["store required", false]]) { const r = run({ ...BASE, rollout: "all-at-once", stagger: 10, wrongAt: DURATION, downAt: 200, upAt: 380, restartAt: 260, diskCache }); const outage = r.droppedRequests / EDGE_PEAK; // equivalent outage seconds at the edge peak rate console.log(`${label.padEnd(24)}${String(r.failedReads).padStart(17)}` + `${r.droppedRequests.toFixed(0).padStart(19)}${outage.toFixed(2).padStart(19)}` + `${`${((100 * outage) / BUDGET).toFixed(2)}%`.padStart(19)}`); } console.log(`\n${"DD16".padStart(8)}${"store reads/s".padStart(16)}${"worst-case propagation".padStart(25)}` + `${"faulty reply ceiling".padStart(25)}`); for (const a of [5, 30, 300]) console.log(`${`${a} s`.padStart(8)}${(REPLICAS / a).toFixed(4).padStart(16)}` + `${`${a} s`.padStart(25)}${(EDGE_PEAK * (DETECTION + a)).toFixed(0).padStart(25)}`); console.log(`if every request read the store, store reads would be ${EDGE_PEAK}/s; at DD16 = ${INTERVAL} s that is ` + `${(EDGE_PEAK / (REPLICAS / INTERVAL)).toFixed(0)}x fewer`); ``` ``` 3 replicas, 171.3 requests/s per replica (M19/K03), edge peak 513.89 DD16 refresh interval 30 s, DD17 detection time 120 s; wrong value written at second 65 rollout stagger propagation first seen last seen mixed s faulty replies affected replicas all-at-once 0 25 90 209 0 61668 3 all-at-once 10 25 70 209 40 61668 3 staged 0 25 90 209 120 20556 1 staged 10 25 90 209 120 20556 1 store outage: down for 180 s; a replica restarts after failover at second 260 behavior failed reads dropped requests equiv. outage s of failure share last known good value 18 0 0.00 0.00% store required 18 20556 40.00 2.36% DD16 store reads/s worst-case propagation faulty reply ceiling 5 s 0.6000 5 s 64236 30 s 0.1000 30 s 77084 300 s 0.0100 300 s 215834 if every request read the store, store reads would be 513.89/s; at DD16 = 30 s that is 5139x fewer ``` These numbers belong to the **measurement** class, but the time axis is modeled seconds; the run is arithmetic and does not depend on the environment. #### Propagation Opens a Window The change is written to the store at second 65, and the replicas finish converging 25 seconds later. Propagation time can run shorter than the refresh interval — it depends on the distance between the write instant and the next refresh — but its upper bound is the interval. The bottom table's last column confirms this: at DD16 = 5, 30, 300 seconds, worst-case propagation is 5, 30, 300 seconds respectively. The second row shows what staggering does and does not do. When the replicas' refresh is spread out with a 10-second stagger, the first replica to see the value sees it at second 70 instead of 90, and a **40-second mixed-value window** opens: some replicas answer with the new value, some with the old. Total faulty responses stay at 61,668 in both rows; staggering does not speed up propagation, it only spreads it out over time. The mixed-value window is not a defect. One client request can be served with the new value and the next with the old; a configuration change is **not atomic**, and values must make sense side by side. Writing two dependent settings one at a time produces an invalid combination inside the mixed window; that is why dependent settings are written as a single record. #### Blast Radius and the Store Going Down The third and fourth rows measure a wrong value's scope. When the value is written valid for every replica, it spreads to all three at once and produces **61,668 faulty responses** within the DD17 = 120-second detection window. When the same value is first marked on a single replica and handed to the others only after confirmation, the blast radius stays at 1 replica and faulty responses come to **20,556** — exactly a third. The staged rollout's mixed-value window is 120 seconds, and that is not a defect but the pattern itself: the time the two values coexist is time set aside for detection. The bottom table has the store going down: it stays down for 180 seconds, and during that time a replica restarts after failover. Two behaviors split apart. Choosing **the last known good value** gives 18 failed reads, the replicas keep running on their cached value, the restarting replica loads the value from its disk copy, and dropped requests are 0. Choosing **the store required** gives the same 18 failed reads, but the restarting replica cannot come up: it drops 20,556 requests over 120 seconds. That is a 40.00-second equivalent outage at the edge peak rate, spending 2.36 percent of K01's monthly 28.2-minute failure share in a single event. The conclusion is a design rule: **the external configuration store must be read in a way that does not bring the application down when it goes down itself.** Otherwise a single helper component caps the service availability of every service depending on it. The startup validation rule does not conflict with this; validation runs against the last known good value — the value is missing, not the store. #### The Numbers for Two Days **The failure-free day's cost has three items.** The first is store reads, and caching keeps it small: at DD16 = 30 seconds, the three replicas together make 0.1000 reads a second. If every request read the store, that would be 513.89 reads/s — 5139 times more. The second is delay: a setting change does not take effect instantly; in the worst case it waits up to DD16. The third is the mixed-value window: replicas run on different values for a while, which is why dependent settings must be kept in a single record. DD16 ties these three items together and pulls in both directions. Shortening the interval to 5 seconds raises store reads to 0.6000/s (sixfold) but lowers the faulty-reply ceiling from 77,084 to 64,236. Stretching the interval to 300 seconds drops reads to 0.0100/s and raises the ceiling to 215,834. **The refresh interval is not a performance parameter, it is a rollback time**: it sets how long a wrong value stays in effect. **The failing day's gain has two items.** When the store goes down, running on the last known good value drops dropped requests from 20,556 to 0 and the equivalent outage from 40.00 seconds to zero, preserving 2.36 percent of the monthly failure share. And a wrong value's blast radius under staged rollout falls from three replicas to one, faulty responses from 61,668 to 20,556. Neither improves the failure-free day; the first adds a read path, the second a rollout procedure. #### Summary - A change propagated to three replicas in 25 seconds; the upper bound is the refresh interval, and at DD16 = 5/30/300 seconds it is 5, 30, 300 seconds respectively. - Staggering the refresh does not speed up propagation, it opens a 40-second mixed-value window; total faulty responses stay at 61,668 in both cases. That is why dependent settings are written as a single record. - Cached reads keep store load at 0.1000 reads/s; if every request read the store it would be 513.89, 5139 times more. - Staged rollout drops the affected replicas from 3 to 1, faulty responses from 61,668 to 20,556; in exchange it makes the 120-second mixed-value window permanent. - When the store goes down, running on the last known good value drops dropped requests from 20,556 to 0: a 40.00-second equivalent outage, 2.36 percent of K01's monthly 28.2-minute failure share. - DD16 is a rollback time, not a performance parameter: at 5 seconds the faulty-reply ceiling is 64,236, at 300 seconds it is 215,834. #### Next Step The five patterns in this topic ran inside the same system. The idempotency ledger, the compensation chain, the claim check, the valet key, and the external configuration store — all parts of the **new** arrangement, all sitting on K01's numbers. But the shipment tracking and billing service was not born from scratch: the old system is still standing, still answering the same tracking numbers, still producing the same invoice lines, next to the new arrangement. Nothing these five lessons designed said how that system would be taken over — which route moves when, where the unmoved routes go, how state held in both systems stays consistent, and how long the migration takes were never asked. The next lesson takes on migrating the old system in stages. ### Strangler Fig Pattern Source: https://academia.sh/en/courses/resilience-patterns/strangler-fig-pattern Migrating the old system one route at a time: moving eight routes in two different orders, the volume order carrying traffic 3.89 times faster while growing the dual-write load 1.52 times, the migration facade drawing a share from the timeout budget, and reversibility rising from one round to 3360 seconds the moment dual writes stop. The previous five lessons built the parts of the new arrangement: the idempotency ledger, the compensation chain, the claim check, the valet key, and the external configuration store. All of them stand on a single assumption — that the shipment tracking and billing service consists of nothing but this design. But the service was not born from scratch: the old system is still standing, next to the new arrangement. The **strangler fig** pattern is a migration style that moves the old system's routes one at a time instead of replacing it in one shot, placing a routing layer in front of both systems. That layer is the **migration facade**: it takes every request, checks whether that route has been moved, and sends it to the old or new system. The measures are four: the number and order of moved routes, the size of state held in both systems at once, the facade's cost on a failure-free day, and the round it takes to revert a route. #### Order Is a State Decision The service has eight routes and five state entities. When a route is moved, if a state entity it touches is also used by routes that have not moved, that entity **splits**: it must be held in two stores at once and written to both on every write. Split state is the migration's real cost, and the migration order determines it directly. **DD18 — one invoice read per seller a day**, so 4000 a day; the rationale is K01's 4000 sellers. Tariff volume comes from the Scaling the Data Layer course's D5 assumption, proof volumes from this topic's DD8 and DD12 assumptions; the rest is K01's computed value. ```js // fig/migration.mjs — in-process model of moving routes one at a time. There is no real system, // network, or store: routes and state entities are objects, split state is a set operation. // [route name, daily request volume (K01 computed values and DD18), state read, state written] export const ROUTES = [ ["tracking-query", 12_000_000, "shipment event", ""], ["status-event", 2_800_000, "shipment", "event"], ["shipment-create", 400_000, "", "shipment"], ["proof-upload", 400_000, "event", "proof"], ["proof-read", 8_000, "proof", ""], ["billing-period", 4_000, "shipment event tariff", "invoice"], ["invoice-read", 4_000, "invoice", ""], ["tariff-admin", 1, "", "tariff"], ].map(([name, volume, r, w]) => ({ name, volume, touches: `${r} ${w}`.split(" ").filter(Boolean) })); // Daily write volume of each state entity (K01 computed values; tariff is M19/K04's D5 assumption) export const WRITES = { shipment: 400_000, event: 2_800_000, proof: 400_000, invoice: 4_000, tariff: 1 }; export const DAY = 86_400; // Given a set of moved routes, the split state entities and the dual-write rate export function split(moved) { const names = new Set(moved.map((r) => r.name)), remaining = ROUTES.filter((r) => !names.has(r.name)); const b = Object.keys(WRITES).filter((d) => moved.some((r) => r.touches.includes(d)) && remaining.some((r) => r.touches.includes(d))); return { entities: b, dualWrite: b.reduce((t, d) => t + WRITES[d], 0) / DAY }; } // Order: "volume" = the route with the most requests first; "state" = the route minimizing split writes first export function sortRoutes(mode) { if (mode === "volume") return [...ROUTES].sort((a, b) => b.volume - a.volume); const remaining = [...ROUTES], order = []; while (remaining.length > 0) { let best = 0; for (let i = 1; i < remaining.length; i += 1) { const a = split([...order, remaining[i]]).dualWrite, e = split([...order, remaining[best]]).dualWrite; if (a < e || (a === e && remaining[i].volume > remaining[best].volume)) best = i; } order.push(...remaining.splice(best, 1)); } return order; } ``` ```js // fig/measure.mjs — dual-write load, moved traffic, revert, and the failing day for two orders import { ROUTES, WRITES, DAY, split, sortRoutes } from "./migration.mjs"; const TOTAL = ROUTES.reduce((t, r) => t + r.volume, 0); const EDGE_PEAK = 513.89, SCAN_RATE = 833.33, BUDGET = 28.2 * 60; // K01 computed values; monthly failure share console.log(`${ROUTES.length} routes, ${Object.keys(WRITES).length} state entities, ` + `${TOTAL.toLocaleString("en-US")} requests a day`); const track = (mode) => { const moved = [], rows = []; let dualWriteArea = 0, trafficArea = 0; for (const r of sortRoutes(mode)) { moved.push(r); const b = split(moved), share = moved.reduce((t, x) => t + x.volume, 0) / TOTAL; dualWriteArea += b.dualWrite; trafficArea += share; rows.push([r.name, share, b.entities.length, b.dualWrite]); } return { rows, dualWriteArea, trafficArea }; }; const R = {}; for (const mode of ["volume", "state"]) { R[mode] = track(mode); console.log(`\n${mode} order`); console.log(`${"step".padStart(5)}${"moved route".padStart(18)}${"moved traffic".padStart(16)}${"split entities".padStart(17)}${"dual writes/s".padStart(16)}`); R[mode].rows.forEach(([name, share, n, c], i) => console.log(`${String(i + 1).padStart(5)}${name.padStart(18)}${`${(100 * share).toFixed(2)}%`.padStart(16)}${String(n).padStart(17)}${c.toFixed(2).padStart(16)}`)); console.log(`dual-write area ${R[mode].dualWriteArea.toFixed(2)} writes/s-step, moved-traffic area ${R[mode].trafficArea.toFixed(4)} steps`); } console.log(`\nvolume / state: dual writes ${(R.volume.dualWriteArea / R.state.dualWriteArea).toFixed(2)}x, moved traffic ${(R.volume.trafficArea / R.state.trafficArea).toFixed(2)}x`); // Revert: while dual writes are running, switching routing is 1 round; once they stop, backfill is needed. console.log(`\n${"after dual writes stop".padEnd(31)}${"events to backfill".padStart(21)}${"revert s".padStart(11)}${"of failure share".padStart(19)}`); for (const [label, afterSeconds] of [["0 (dual writes still running)", 0], ["1 hour", 3600], ["1 day", 86_400]]) { const records = (WRITES.event / DAY) * afterSeconds, seconds = records / SCAN_RATE; console.log(`${label.padEnd(31)}${Math.round(records).toLocaleString("en-US").padStart(21)}${seconds.toFixed(2).padStart(11)}${`${((100 * seconds) / BUDGET).toFixed(2)}%`.padStart(19)}`); } // Failing day: the new system stops the first step each order's moved traffic crosses 50 percent. const DD20 = 60; // routing entry lifetime in the facade-less arrangement, s console.log(`\n${"order".padEnd(8)}${"step".padStart(6)}${"moved traffic".padStart(16)}${"facade: dropped".padStart(17)}${"DD20: dropped".padStart(15)}${"equiv. outage s".padStart(17)}${"of failure share".padStart(19)}`); for (const mode of ["volume", "state"]) { const i = R[mode].rows.findIndex(([, p]) => p >= 0.5), share = R[mode].rows[i][1]; const dropped = EDGE_PEAK * share * DD20; console.log(`${mode.padEnd(8)}${String(i + 1).padStart(6)}${`${(100 * share).toFixed(2)}%`.padStart(16)}${(EDGE_PEAK * share).toFixed(0).padStart(17)}${dropped.toFixed(0).padStart(15)}${(dropped / EDGE_PEAK).toFixed(2).padStart(17)}${`${((100 * dropped) / EDGE_PEAK / BUDGET).toFixed(2)}%`.padStart(19)}`); } ``` ``` 8 routes, 5 state entities, 15,616,001 requests a day volume order step moved route moved traffic split entities dual writes/s 1 tracking-query 76.84% 2 37.04 2 status-event 94.77% 2 37.04 3 shipment-create 97.34% 2 37.04 4 proof-upload 99.90% 3 41.67 5 proof-read 99.95% 2 37.04 6 billing-period 99.97% 2 0.05 7 invoice-read 100.00% 1 0.00 8 tariff-admin 100.00% 0 0.00 dual-write area 189.86 writes/s-step, moved-traffic area 7.6878 steps state order step moved route moved traffic split entities dual writes/s 1 tariff-admin 0.00% 1 0.00 2 invoice-read 0.03% 2 0.05 3 shipment-create 2.59% 3 4.68 4 proof-read 2.64% 4 9.31 5 proof-upload 5.20% 4 37.08 6 billing-period 5.23% 2 37.04 7 tracking-query 82.07% 2 37.04 8 status-event 100.00% 0 0.00 dual-write area 125.19 writes/s-step, moved-traffic area 1.9775 steps volume / state: dual writes 1.52x, moved traffic 3.89x after dual writes stop events to backfill revert s of failure share 0 (dual writes still running) 0 0.00 0.00% 1 hour 116,667 140.00 8.27% 1 day 2,800,000 3360.01 198.58% order step moved traffic facade: dropped DD20: dropped equiv. outage s of failure share volume 1 76.84% 395 23694 46.11 2.72% state 7 82.07% 422 25305 49.24 2.91% ``` These numbers belong to the **measurement** class; they come from an in-process model, and their inputs are K01's computed values and this topic's assumptions. #### Two Orders, Two Different Things Improve The volume order carries 76.84 percent of traffic in the first step. Its cost starts immediately: `shipment` and `event` split from the first step onward, and dual writes hold at 37.04 writes/s through the migration. That is an average rate, and because each split entity's write is applied **again** to the second store, it doubles the write path exactly, to 74.08 writes/s. The state order does the opposite: dual writes are near zero for the first six steps, but moved traffic stays at 5.23 percent, and the old system carries its full load through most of the migration. Two integrals give the trade-off in a single line — the volume order carries traffic 3.89 times faster, growing the dual-write load 1.52 times. The split-entity count opens a third distinction: the state order lowers the dual-write **rate** while raising the split-entity **count** to 4 (at most 3 under the volume order) — more code paths, fewer bytes. **There is no single best migration order**; the choice is the answer to whether the old system's load or the dual-write risk costs more. #### The Facade's Cost and Reversibility The facade sits in front of all traffic, and its failure-free-day cost is three items. The first is an **extra stop**: all 15,616,001 requests a day, 513.89 requests/s at peak edge, pass through one more routing decision. The second is **a share of the timeout budget**. The Timeout Budgets lesson split the 200-millisecond threshold: gateway share 20 ms, remaining arm 180 ms, distribution 90/90. If the facade also takes a gateway-sized share (**DD19**), the remaining arm drops to 160 ms and the distribution becomes 80/80 — each step's share narrows 11.1 percent. The third is dual writes. The facade is also a new single point: the Deployment Stamps and Geo-Replicas lesson measured the routing map's blast radius at 90.9 percent; the facade's is 100 percent. What it buys in return is **reversibility**. The bottom table counts the case where the new system stops mid-migration. With the facade in place, reverting a route is a single routing decision change: 395 requests drop under the volume order, 422 under the state order. In an arrangement where clients are routed directly, reverting takes as long as the routing entry's lifetime (**DD20 — 60 seconds**), and dropped requests climb to 23,694 and 25,305: a 46–49-second equivalent outage, 2.72–2.91 percent of K01's monthly failure share. Reversibility has an expiration date. While dual writes run, the old system's data stays current, and reverting is one round. An hour after they stop, 116,667 events must be backfilled into the old system: 140.00 seconds at K01's 833.33 records/s scan rate, 8.27 percent of the failure share. A day later it is 2,800,000 events and 3360.01 seconds — 198.58 percent of the share. **Stopping dual writes is not a savings decision, it is closing off the way back.** #### Summary - If a moved route's touched state entity is also used by an unmoved route, it splits and is written to two stores at once; dual writes raise the write path from 37.04 to 74.08 writes/s. - The volume order carries 76.84 percent of traffic in the first step but pushes dual writes to 37.04 writes/s immediately; the state order gets through its first six steps with no dual writes, keeping traffic at 5.23 percent. - Two integrals give the trade-off: the volume order carries traffic 3.89 times faster, growing the dual-write load 1.52 times. There is no single best order. - The facade's failure-free-day cost: an extra stop on 15,616,001 requests a day, a gateway-sized share of the budget (arm share 180 → 160 ms, distribution 90/90 → 80/80), and a single point with 100 percent blast radius. - The facade's failing-day gain is revert speed: 422 dropped requests instead of 25,305. - Reverting is one round only while dual writes run; a day after they stop, it rises to 3360.01 seconds (198.58 percent of the failure share). #### Course Wrap-Up This course took the system four earlier courses designed and put it under failure, asking the same two questions in every lesson: what does this pattern cost on a failure-free day, and how many units does it rescue on a failing day. | Lesson | Failure mode | Failure-free day's cost | Failing day's gain | |---|---|---|---| | Failure Modes | down, slowdown, partial failure, stale content, network partition | 8-byte stamp on every response (4111.12 bytes/s); 20-call window for naming | down 97.22 req/s and 1 component, slowdown 513.89 req/s and 2 components — 5.29x | | Circuit Breaker | dependency slowdown and down | queried on all 1166 calls, 20-record window per dependency; ratio-0.10 threshold rejects 386 valid events on a healthy day | tracking requests finding no slot 671 → 0; event leg's slot-rounds 19,312 → 1646 (91.48% less) | | Bulkhead Pattern | slowdown of one dependency | dropped requests 105 → 578 (5.5x), utilization 0.541 → 0.524; 4239 times rejected while an idle slot sits in the neighboring pool | neighboring flow's dropped requests 2654 → 172 (93.5%), independent of detection | | Retries and Storm Risk | full outage and partial degradation | call count unchanged from allowance 1 to 27, multiplier 1.03 | retry budget 0.30 cuts the outage storm from 6.24 to 2.88 calls/round; cost 103 dropped requests | | Timeout Budgets | dependency slowdown (AY10) | deadline field 40 bytes (8.3% of the response); 2 of 48 requests cut off | weighted split brings median 200 → 93 ms, held slots 82.4 → 40.2 (with cancellation signal) | | Backpressure | overload (AY11) | 4164 valid events rejected (3.57%), 291.66 signal checks/s | 44,399 events shift from lost to rejected; at the narrow gap, loss falls 34,064 → 8579 (ratio 3.971) | | Throttling and Load Shedding | batch job moves into peak hour (AY13a); edge node goes down (AY13b) | throttling rejects 15,630 valid queries (6.25%); shedding costs nothing | shedding protects 0.9348 of the peak under AY13b; the gateless arrangement would have eaten 56.7% of the failure share | | Graceful Degradation | optional dependency (AY14a); core dependency (AY14b) goes down | 24-byte freshness marker (5.0%), 2.29 MiB fallback store, 1666.68 writes/s | AY14a cuts downtime 15.00 → 0.00 minutes; AY14b only 15.00 → 14.02 | | Failover Design | active replica stops | 120 s/month false-failover outage at threshold 1; zero at threshold 3, real-stop window rising 4 → 8 s | outage 20.0 minutes → 16 seconds (75x); 0.95% of the failure share instead of 70.9% | | Health Endpoint Monitoring | half-failed replica; shared dependency goes down | deep check adds 1.08% load to the store (17.28% at 48 replicas); false eviction raises load to 1.50x, 13.3 times a month | silent lost updates 17 → 0; readiness check saved 40 reads and 10 writes | | Recovery Objectives | data-loss and recovery-time thresholds exceeded | zeroing out data loss costs 2,800,000 acknowledgment rounds a day, 1,555.52 rejected writes a month | 194.44 permanent losses a month disappear (ratio 8.00); the 8-second window fits 211.50 failures into the failure share | | Redundancy Zones | machine, zone, and region loss | idle capacity $1/(n-1)$: 100% with two zones, 20% with six zones; the two-region arrangement loses 1 write even at a single machine's loss | zone loss: dropped writes 126 → 0, lost 114 → 2 | | Deployment Stamps and Geo-Replicas | in-stamp component failure | components 9 → 33 (65 with geo-replicas); end-of-day billing must reach as many places as there are stamps | state store blast radius 100% → 25%; links crossing the stamp 6 → 2 | | Disaster Recovery Drill | controlled failure trial | a monthly drill costs 0.63 minutes (4.22% of the planned share); drops 19,527.82 edge requests at peak load | plan's 6-round recovery measured 19 rounds in the drill (3.17x); the 30 s objective breaks at 38 s | | Idempotent Operations | duplicate request under failover and retry | ledger 3,200,000 records / 230.40 MB (23.61% of daily growth); write rows ×2.00, reads ×3.33 | 8002 prevented side effects a month: 902 duplicate invoice lines, 902 duplicate notifications | | Compensating Transactions | workflow interrupted midway | 8000 written compensation paths, 60 calls run a day; uncompensatable step cannot be moved to the end | uncompensated step 80 → 60/day; second attempt drops stuck work 2.979 → 0.171/day | | Claim Check Pattern | producer and consumer crashes; queue backlog | 112.00 GB in the store (15.72% of retention); 1968 orphaned payloads a day, 3948 dangling claim checks | queue 4.4689 → 0.0316 Mbit/s; backlog 3016.5 → 21.3 MB (141.6x) | | Valet Key Pattern | proof store slowdown | 400,000 signings a day (2.703% of K01's edge peak); 4166.7-key open window | dropped transfers 28 → 0, held slots 64 → 0; scope narrowing 730x | | External Configuration Store | store goes down; wrong value written | store reads 0.1000/s; propagation 25 s, mixed-value window 40 s | dropped requests 20,556 → 0 (2.36% of the failure share); staged rollout drops faulty responses 61,668 → 20,556 | | Strangler Fig Pattern | new system stops mid-migration | dual writes push the write path 37.04 → 74.08 writes/s; the facade takes 20 ms from the budget (180 → 160 ms) | dropped requests 25,305 → 422; revert is 1 round with dual writes, 3360.01 s a day after they stop | The table's rule deserves its own name: **a pattern cannot be defended without two numbers.** The sentence "it improves resilience" carried no decision in this course. The two columns are not interchangeable either — the failure-free day's cost is paid every day, the failing day's gain is collected a few times a year; choosing a pattern means comparing the product of these two frequencies. The course's own assumptions (the AY, KU, and DD series) never mixed into K01's table; each was written under its own name and sensitivity. The question the course leaves behind sits underneath the whole table. The circuit breaker works once a threshold is **noticed** as crossed, failover once a replica is noticed to have stopped, backpressure once a queue is noticed to be filling, the external configuration store once a value is noticed to be wrong. How the noticing happens was never designed: which metric gets collected, which threshold raises an alert, which measurement makes a slowdown visible, and how the load a design can carry gets tested were all assumed. The next course, **Performance Anti-Patterns and Monitoring**, takes on that layer.