--- title: 'Circuit Breaker' source: 'https://academia.sh/en/courses/resilience-patterns/circuit-breaker' course: 'Resilience and Reliability' language: en updated: '2026-08-23T07:01:30+00:00' license: 'CC BY-SA 4.0' --- # Circuit Breaker Taking a broken dependency out of the circuit: the breaker protecting the caller's slot pool shared between services, the threshold being chosen by failure mode, comparing the consecutive-count and window-ratio rules across four modes, counting the calls saved and the neighboring requests rescued while open, and the cost of a false open that cuts off a healthy dependency, measured in rejected valid requests. The previous lesson left a direction: stopping calls to a dependency that is slowing down rounds the slowdown into a down, cutting the blast radius from 513.89 to 97.22 req/s. This lesson's job is to round that sentence into a component and measure, in the same failure scenario, whether it actually holds. A **circuit breaker** is a component that looks at the outcome of a dependency's recent calls and stops making calls to it for a while. It has three states, established in the Application Architecture course: **closed** lets the call through; **open** makes no call and fails the caller immediately; **half-open** lets a single probe call through and closes again or reopens by its result. The transition mechanics are not retold here. Three things change across services. First, the resource the breaker protects is the caller's **slot pool**, shared with other flows — the breaker protects not the broken dependency but the flows sharing its pool. Second, the threshold is not a preference but **a choice bound to the failure mode**: the same threshold is cheap under down, expensive under slowdown, wrong under partial failure. Third, requests rejected in the open state are real requests that have to go somewhere. ## The Same Scenario Without and With the Breaker The run below expands lesson 1's model into rounds. The slot pool sits in the gateway and is shared between the tracking query and the state event; the dependency's mode is a parameter. It is again a **model** — no real cluster, container, or chaos tool is set up, randomness comes from a seeded generator. **AY5 — the breaker's rule: open if 0.50 of the last 20 calls fail, stay open for 100 rounds (1 second).** Rationale: 20 calls is roughly a 20-round window at the carrier's event rate, that is, K01's read threshold; the 0.50 threshold sits between partial failure's 0.125 failure ratio and down/slowdown's 1.000. Its sensitivity is printed below at an open duration of 200 rounds. ```js // breaker/run.mjs — the same failure scenario without and with the breaker. MODEL: a round is an // abstract step, the gateway's slot pool is shared between two flows, the dependency's mode is a parameter. const SLOTS = 32, CYCLE = 100, T = 1200, START = 200; // AY2, AY1; 12 s run, failure at second 2 const TRACKING = 416.67, EVENT = 97.22, TRACKING_HOLD = 4; // K01 peak rates; lesson 01: tracking leg 4 rounds const MODES = { // p: the call's failure probability healthy: { p: 0.025, ok: 6, fail: 22 }, // AY4: rare timeout, holds the slot 22 rounds partial: { p: 0.125, ok: 6, fail: 3 }, // one of eight shards: fast error down: { p: 1, ok: 6, fail: 3 }, slowdown: { p: 1, ok: 6, fail: 22 }, }; function run({ mode, rule }) { let s = 20260731 % 2147483647; const random = () => (s = (s * 48271) % 2147483647) / 2147483647; const free = new Array(SLOTS).fill(0); const c = { trackingResponses: 0, trackingNoSlot: 0, eventResponses: 0, eventRejected: 0, eventErrors: 0, eventNoSlot: 0, calls: 0, eventSlotRounds: 0, opens: 0, detectRound: null, detectSlotRounds: null }; let trackingAccum = 0, eventAccum = 0, state = "closed", consecutive = 0, openUntil = 0, window = [], startSlotRounds = 0; const acquireSlot = (t, duration) => { const i = free.findIndex((x) => x <= t); if (i < 0) return false; free[i] = t + duration; return true; }; const opened = (t) => { // when the breaker moves from closed to open c.opens += 1; state = "open"; openUntil = t + rule.openFor; consecutive = 0; window = []; if (c.detectRound === null && t >= START && mode !== "healthy") { c.detectRound = t - START; c.detectSlotRounds = c.eventSlotRounds - startSlotRounds; } }; for (let t = 0; t < T; t += 1) { if (t === START) startSlotRounds = c.eventSlotRounds; if (state === "open" && t >= openUntil) state = "halfOpen"; trackingAccum += TRACKING / CYCLE; eventAccum += EVENT / CYCLE; const arrivals = []; // arrivals within the round are shuffled (seeded) for (let n = Math.floor(trackingAccum); n > 0; n -= 1, trackingAccum -= 1) arrivals.push("tracking"); for (let n = Math.floor(eventAccum); n > 0; n -= 1, eventAccum -= 1) arrivals.push("event"); for (let i = arrivals.length - 1; i > 0; i -= 1) { const j = Math.floor(random() * (i + 1)); [arrivals[i], arrivals[j]] = [arrivals[j], arrivals[i]]; } for (const flow of arrivals) { if (flow === "tracking") { if (acquireSlot(t, TRACKING_HOLD)) c.trackingResponses += 1; else c.trackingNoSlot += 1; continue; } if (state === "open") { c.eventRejected += 1; acquireSlot(t, 1); continue; } // open: the call is never made const cfg = MODES[mode === "healthy" || t < START ? "healthy" : mode]; const failed = random() < cfg.p; const duration = failed ? cfg.fail : cfg.ok; if (acquireSlot(t, duration) === false) { c.eventNoSlot += 1; continue; } c.calls += 1; c.eventSlotRounds += duration; if (failed) c.eventErrors += 1; else c.eventResponses += 1; if (rule === null) continue; // ruleless run: state is not tracked if (state === "halfOpen") { if (failed) opened(t); else { state = "closed"; consecutive = 0; window = []; } continue; } if (rule.kind === "consecutive") { consecutive = failed ? consecutive + 1 : 0; if (consecutive >= rule.threshold) opened(t); } else { window.push(failed ? 1 : 0); if (window.length > rule.windowSize) window.shift(); if (window.length === rule.windowSize && window.reduce((a, x) => a + x, 0) / rule.windowSize >= rule.threshold) opened(t); } } } return c; } const RATIO_RULE = { kind: "ratio", windowSize: 20, threshold: 0.5, openFor: 100 }; // AY5 const b = (x) => x.toFixed(2); const METRICS = ["trackingResponses", "trackingNoSlot", "eventResponses", "eventRejected", "eventErrors", "calls", "eventSlotRounds"]; const HEADERS = ["tracking responses", "tracking no slot", "event responses", "event rejected", "event errors", "calls made", "event slot-rounds"]; console.log(`model: ${T} rounds (${T / CYCLE} s), failure at round ${START}, ${SLOTS} slots; chosen rule: open if ` + `${RATIO_RULE.threshold} of the last ${RATIO_RULE.windowSize} calls fail, stay open ${RATIO_RULE.openFor} rounds`); console.log(`\n${"run".padEnd(24)}` + HEADERS.map((x) => x.padStart(20)).join("")); const r = {}; for (const mode of ["healthy", "partial", "slowdown", "down"]) for (const [suffix, rule] of [["no breaker", null], ["with breaker", RATIO_RULE]]) { const name = `${mode}, ${suffix}`; r[name] = run({ mode, rule }); console.log(`${name.padEnd(24)}` + METRICS.map((k) => String(r[name][k]).padStart(20)).join("")); } const h1 = r["healthy, with breaker"]; const [p0, p1] = [r["partial, no breaker"], r["partial, with breaker"]]; const [w0, w1] = [r["slowdown, no breaker"], r["slowdown, with breaker"]]; const [d0, d1] = [r["down, no breaker"], r["down, with breaker"]]; const scale = (600 * CYCLE) / (T - START); // AY3: scaling to a 10-minute failure console.log(`\nfailure-free day's cost: the breaker is queried on all ${h1.calls} calls and a ` + `${RATIO_RULE.windowSize}-record window is kept; ${h1.opens} false opens, ${h1.eventRejected} rejected valid events; ` + `partial failure opens ${p1.opens} times and event responses ${p0.eventResponses} -> ${p1.eventResponses} ` + `(${p0.eventResponses - p1.eventResponses} events that could have succeeded were rejected)`); console.log(`failing day's gain (slowdown): tracking no slot ${w0.trackingNoSlot} -> ${w1.trackingNoSlot}, ` + `event slot-rounds ${w0.eventSlotRounds} -> ${w1.eventSlotRounds} (${b((100 * (w0.eventSlotRounds - w1.eventSlotRounds)) / w0.eventSlotRounds)}% less), ` + `calls to the broken dependency ${w0.calls} -> ${w1.calls}`); console.log(`the same gain under down: tracking no slot ${d0.trackingNoSlot} -> ${d1.trackingNoSlot}, ` + `event slot-rounds ${d0.eventSlotRounds} -> ${d1.eventSlotRounds}, calls ${d0.calls} -> ${d1.calls}`); console.log(`scaled to AY3's 10 minutes: calls saved under slowdown ` + `${((w0.calls - w1.calls) * scale).toFixed(0)}, tracking requests rescued ` + `${((w0.trackingNoSlot - w1.trackingNoSlot) * scale).toFixed(0)}`); console.log(`\n${"rule".padEnd(18)}${"mode".padEnd(11)}${"detection round".padStart(15)}` + `${"slot-rounds to detection".padStart(26)}${"opens".padStart(8)}${"rejected valid events".padStart(25)}`); const RULES = [ ["consecutive 5", { kind: "consecutive", threshold: 5, openFor: 100 }], ["ratio 0.50 / 20", { kind: "ratio", windowSize: 20, threshold: 0.5, openFor: 100 }], ["ratio 0.10 / 20", { kind: "ratio", windowSize: 20, threshold: 0.1, openFor: 100 }], ]; for (const [name, rule] of RULES) for (const mode of ["healthy", "partial", "slowdown", "down"]) { const k = run({ mode, rule }); const lost = mode === "partial" ? p0.eventResponses - k.eventResponses : k.eventRejected; console.log(`${name.padEnd(18)}${mode.padEnd(11)}${String(k.detectRound ?? "-").padStart(15)}` + `${String(k.detectSlotRounds ?? "-").padStart(26)}${String(k.opens).padStart(8)}` + `${String(mode === "healthy" || mode === "partial" ? lost : "-").padStart(25)}`); } const longer = { ...RATIO_RULE, openFor: 200 }, narrower = { kind: "ratio", windowSize: 20, threshold: 0.1, openFor: 200 }; console.log(`sensitivity: if open duration were ${longer.openFor} rounds instead of ${RATIO_RULE.openFor}, calls made under slowdown ` + `would go ${w1.calls} -> ${run({ mode: "slowdown", rule: longer }).calls}; under the ratio 0.10 rule, rejected valid events ` + `on a failure-free day would go ${run({ mode: "healthy", rule: RULES[2][1] }).eventRejected} -> ` + `${run({ mode: "healthy", rule: narrower }).eventRejected}`); ``` ``` model: 1200 rounds (12 s), failure at round 200, 32 slots; chosen rule: open if 0.5 of the last 20 calls fail, stay open 100 rounds run tracking responses tracking no slot event responses event rejected event errors calls made event slot-rounds healthy, no breaker 5000 0 1135 0 31 1166 7492 healthy, with breaker 5000 0 1135 0 31 1166 7492 partial, no breaker 5000 0 1021 0 145 1166 6637 partial, with breaker 5000 0 1021 0 145 1166 6637 slowdown, no breaker 4329 671 190 0 826 1016 19312 slowdown, with breaker 5000 0 190 953 23 213 1646 down, no breaker 5000 0 190 0 976 1166 4144 down, with breaker 5000 0 190 953 23 213 1285 failure-free day's cost: the breaker is queried on all 1166 calls and a 20-record window is kept; 0 false opens, 0 rejected valid events; partial failure opens 0 times and event responses 1021 -> 1021 (0 events that could have succeeded were rejected) failing day's gain (slowdown): tracking no slot 671 -> 0, event slot-rounds 19312 -> 1646 (91.48% less), calls to the broken dependency 1016 -> 213 the same gain under down: tracking no slot 0 -> 0, event slot-rounds 4144 -> 1285, calls 1166 -> 213 scaled to AY3's 10 minutes: calls saved under slowdown 48180, tracking requests rescued 40260 rule mode detection round slot-rounds to detection opens rejected valid events consecutive 5 healthy - - 0 0 consecutive 5 partial - - 0 0 consecutive 5 slowdown 4 110 10 - consecutive 5 down 4 15 10 - ratio 0.50 / 20 healthy - - 0 0 ratio 0.50 / 20 partial - - 0 0 ratio 0.50 / 20 slowdown 9 220 10 - ratio 0.50 / 20 down 9 30 10 - ratio 0.10 / 20 healthy - - 4 386 ratio 0.10 / 20 partial 88 117 9 698 ratio 0.10 / 20 slowdown 67 22 11 - ratio 0.10 / 20 down 67 3 11 - sensitivity: if open duration were 200 rounds instead of 100, calls made under slowdown would go 213 -> 208; under the ratio 0.10 rule, rejected valid events on a failure-free day would go 386 -> 741 ``` These numbers belong to the measurement class and depend on the seed; the same seed gives the same table. The first two rows give the pattern's failure-free-day cost, and the cost is surprisingly small. On a healthy day the runs with and without the breaker are **identical**: 5000 tracking responses, 1135 event responses, 0 false opens, 0 rejected valid events. The cost is countable but small — the breaker is queried on all 1166 calls and keeps a 20-record window per dependency. **A correctly sized threshold becomes invisible on a failure-free day.** The third and fourth rows give the gain. Under slowdown, without a breaker, 671 of 5000 tracking requests find no slot — the round-level counterpart of lesson 1's blast-radius table. With the breaker, that number becomes **0**: the tracking query is no longer affected by the slowdown of a dependency it has nothing to do with. The event leg's slot-rounds fall from 19,312 to 1646, 91.48 percent less, and calls to the broken dependency drop from 1016 to 213. Scaled to AY3's 10-minute slowdown, that is 48,180 calls saved and 40,260 tracking requests rescued. The fifth and sixth rows show what the breaker **does not** rescue. Under down, tracking requests with no slot is already 0 without a breaker; a fast failure holds no slot, so there is no spread and nothing for the breaker to protect. The gain sits elsewhere: calls made fall from 1166 to 213, slot-rounds from 4144 to 1285. For a dependency that is down, the breaker cuts **not the spread, but the wasted work**. One row also confesses the pattern's own cost: 953 events are rejected in the with-breaker slowdown run. The breaker does not rescue the event flow, it **fences it off**. Those 953 rejected state events are still real events and have to go somewhere; where is not this pattern's question. ## The Threshold Is Chosen by Failure Mode The second table runs three threshold rules across four modes, and shows that no single threshold can serve all four. **Consecutive failure count is the cheapest rule, but it never sees partial failure.** The consecutive-5 rule catches both slowdown and down at round 4, and never opens falsely on a healthy day — but never opens under partial failure either. The reason is arithmetic: at a one-in-eight failure ratio, five consecutive failures is rare. Under partial failure that is correct; seven-eighths of calls still succeed, and cutting off would cost more than it gains. **It is not detection's delay that changes by mode, but its cost.** In the consecutive-5 rule, both down and slowdown are detected at round 4; concurrent calls accumulate five failures at the same rate. But slot-rounds held until detection are 15 under down and 110 under slowdown — **7.3 times.** Under ratio-0.50 the gap widens further: 30 against 220. A threshold should be chosen not by call count but by how many slot-rounds that count costs; waiting twenty calls on a slow dependency equals waiting a hundred and twenty. ## The Cost of a False Open The third rule shows the reverse. The ratio-0.10 threshold catches slowdown and down far more cheaply — only 22 and 3 slot-rounds until detection — and even sees partial failure at round 88. Its cost sits in the other two columns. On a healthy day this rule opens 4 times and **rejects 386 valid events** — while nothing is wrong with the dependency. Under partial failure it opens 9 times and rejects 698 valid events, even though those events had a seven-in-eight chance of succeeding. **A false open is cutting off a healthy dependency, and its cost is rejected valid requests.** The sensitivity row says this cost depends on open duration: raising it from 100 to 200 rounds would drop calls made under slowdown from 213 to 208 — almost no gain — but raise rejected valid events on a healthy day from 386 to 741. **Extending open duration buys little on a failing day and costs a lot on a failure-free one**, because the gain depends on the breaker opening once, not on how long the failure lasts, while the loss is multiplied directly by the open duration. ## Summary - A circuit breaker takes a dependency out of the circuit; its closed, open, and half-open states were established in the Application Architecture course and are not repeated here. Across services, it protects the caller's shared slot pool. - A correctly sized threshold is invisible on a failure-free day: the runs with and without a breaker give the same numbers, 0 false opens. The cost is one check per call and a 20-record window per dependency. - The gain under slowdown is large: tracking requests finding no slot go 671 → 0, the event leg's slot-rounds 19,312 → 1646 (91.48 percent less), calls to the broken dependency 1016 → 213. AY3's 10 minutes save 48,180 calls and rescue 40,260 tracking requests. - Under down there is no spread to begin with (tracking no slot 0 → 0); the gain is wasted work: calls 1166 → 213, slot-rounds 4144 → 1285. - Detection's cost changes by mode, not its delay: the consecutive-5 rule catches both modes at round 4, but slot-rounds held until detection are 15 under down and 110 under slowdown. - A false open cuts off a healthy dependency: ratio-0.10 rejects 386 valid events on a healthy day, 698 under partial failure; raising open duration to 200 rounds pushes the healthy-day loss to 741, while the failing-day gain only drops from 213 to 208. ## Next Step The breaker drew a boundary, but where it sits must be read carefully: protection starts **after detection**. Under ratio-0.50, slowdown is detected at round 9, and during those nine rounds the broken dependency draws 220 slot-rounds from the shared pool. Lowering the threshold shrinks that share but grows false opens; the two cannot shrink together. And a breaker sits on each dependency separately, while the pool is single: a slowdown with no breaker, or one under its threshold, still consumes the pool and drags down the neighboring flow. One question remains — can the spread be cut without depending on detection at all. The next lesson splits the pool itself: each flow gets its own slots, the resource a flow can consume is bounded from the start, and what that costs on a failure-free day is counted.