Circuit Breaker

The previous lesson left a direction: stopping calls to a dependency that is slowing down rounds the slowdown into a down, cutting the blast radius from 513.89 to 97.22 req/s. This lesson’s job is to round that sentence into a component and measure, in the same failure scenario, whether it actually holds.

A circuit breaker is a component that looks at the outcome of a dependency’s recent calls and stops making calls to it for a while. It has three states, established in the Application Architecture course: closed lets the call through; open makes no call and fails the caller immediately; half-open lets a single probe call through and closes again or reopens by its result. The transition mechanics are not retold here.

Three things change across services. First, the resource the breaker protects is the caller’s slot pool, shared with other flows — the breaker protects not the broken dependency but the flows sharing its pool. Second, the threshold is not a preference but a choice bound to the failure mode: the same threshold is cheap under down, expensive under slowdown, wrong under partial failure. Third, requests rejected in the open state are real requests that have to go somewhere.

The Same Scenario Without and With the Breaker

The run below expands lesson 1’s model into rounds. The slot pool sits in the gateway and is shared between the tracking query and the state event; the dependency’s mode is a parameter. It is again a model — no real cluster, container, or chaos tool is set up, randomness comes from a seeded generator.

AY5 — the breaker’s rule: open if 0.50 of the last 20 calls fail, stay open for 100 rounds (1 second). Rationale: 20 calls is roughly a 20-round window at the carrier’s event rate, that is, K01’s read threshold; the 0.50 threshold sits between partial failure’s 0.125 failure ratio and down/slowdown’s 1.000. Its sensitivity is printed below at an open duration of 200 rounds.

// breaker/run.mjs — the same failure scenario without and with the breaker. MODEL: a round is an
// abstract step, the gateway's slot pool is shared between two flows, the dependency's mode is a parameter.
const SLOTS = 32, CYCLE = 100, T = 1200, START = 200;   // AY2, AY1; 12 s run, failure at second 2
const TRACKING = 416.67, EVENT = 97.22, TRACKING_HOLD = 4;  // K01 peak rates; lesson 01: tracking leg 4 rounds
const MODES = {                                       // p: the call's failure probability
  healthy: { p: 0.025, ok: 6, fail: 22 },             // AY4: rare timeout, holds the slot 22 rounds
  partial: { p: 0.125, ok: 6, fail: 3 },              // one of eight shards: fast error
  down: { p: 1, ok: 6, fail: 3 },
  slowdown: { p: 1, ok: 6, fail: 22 },
};

function run({ mode, rule }) {
  let s = 20260731 % 2147483647;
  const random = () => (s = (s * 48271) % 2147483647) / 2147483647;
  const free = new Array(SLOTS).fill(0);
  const c = { trackingResponses: 0, trackingNoSlot: 0, eventResponses: 0, eventRejected: 0, eventErrors: 0, eventNoSlot: 0,
    calls: 0, eventSlotRounds: 0, opens: 0, detectRound: null, detectSlotRounds: null };
  let trackingAccum = 0, eventAccum = 0, state = "closed", consecutive = 0, openUntil = 0, window = [], startSlotRounds = 0;

  const acquireSlot = (t, duration) => {
    const i = free.findIndex((x) => x <= t);
    if (i < 0) return false;
    free[i] = t + duration; return true;
  };
  const opened = (t) => {                      // when the breaker moves from closed to open
    c.opens += 1; state = "open"; openUntil = t + rule.openFor; consecutive = 0; window = [];
    if (c.detectRound === null && t >= START && mode !== "healthy") { c.detectRound = t - START; c.detectSlotRounds = c.eventSlotRounds - startSlotRounds; }
  };

  for (let t = 0; t < T; t += 1) {
    if (t === START) startSlotRounds = c.eventSlotRounds;
    if (state === "open" && t >= openUntil) state = "halfOpen";
    trackingAccum += TRACKING / CYCLE; eventAccum += EVENT / CYCLE;
    const arrivals = [];                        // arrivals within the round are shuffled (seeded)
    for (let n = Math.floor(trackingAccum); n > 0; n -= 1, trackingAccum -= 1) arrivals.push("tracking");
    for (let n = Math.floor(eventAccum); n > 0; n -= 1, eventAccum -= 1) arrivals.push("event");
    for (let i = arrivals.length - 1; i > 0; i -= 1) {
      const j = Math.floor(random() * (i + 1));
      [arrivals[i], arrivals[j]] = [arrivals[j], arrivals[i]];
    }

    for (const flow of arrivals) {
      if (flow === "tracking") { if (acquireSlot(t, TRACKING_HOLD)) c.trackingResponses += 1; else c.trackingNoSlot += 1; continue; }
      if (state === "open") { c.eventRejected += 1; acquireSlot(t, 1); continue; }   // open: the call is never made
      const cfg = MODES[mode === "healthy" || t < START ? "healthy" : mode];
      const failed = random() < cfg.p;
      const duration = failed ? cfg.fail : cfg.ok;
      if (acquireSlot(t, duration) === false) { c.eventNoSlot += 1; continue; }
      c.calls += 1; c.eventSlotRounds += duration;
      if (failed) c.eventErrors += 1; else c.eventResponses += 1;
      if (rule === null) continue;                     // ruleless run: state is not tracked
      if (state === "halfOpen") { if (failed) opened(t); else { state = "closed"; consecutive = 0; window = []; } continue; }
      if (rule.kind === "consecutive") {
        consecutive = failed ? consecutive + 1 : 0;
        if (consecutive >= rule.threshold) opened(t);
      } else {
        window.push(failed ? 1 : 0);
        if (window.length > rule.windowSize) window.shift();
        if (window.length === rule.windowSize &&
          window.reduce((a, x) => a + x, 0) / rule.windowSize >= rule.threshold) opened(t);
      }
    }
  }
  return c;
}

const RATIO_RULE = { kind: "ratio", windowSize: 20, threshold: 0.5, openFor: 100 };   // AY5
const b = (x) => x.toFixed(2);
const METRICS = ["trackingResponses", "trackingNoSlot", "eventResponses", "eventRejected", "eventErrors", "calls", "eventSlotRounds"];
const HEADERS = ["tracking responses", "tracking no slot", "event responses", "event rejected", "event errors", "calls made", "event slot-rounds"];

console.log(`model: ${T} rounds (${T / CYCLE} s), failure at round ${START}, ${SLOTS} slots; chosen rule: open if ` +
  `${RATIO_RULE.threshold} of the last ${RATIO_RULE.windowSize} calls fail, stay open ${RATIO_RULE.openFor} rounds`);
console.log(`\n${"run".padEnd(24)}` + HEADERS.map((x) => x.padStart(20)).join(""));
const r = {};
for (const mode of ["healthy", "partial", "slowdown", "down"])
  for (const [suffix, rule] of [["no breaker", null], ["with breaker", RATIO_RULE]]) {
    const name = `${mode}, ${suffix}`;
    r[name] = run({ mode, rule });
    console.log(`${name.padEnd(24)}` + METRICS.map((k) => String(r[name][k]).padStart(20)).join(""));
  }

const h1 = r["healthy, with breaker"];
const [p0, p1] = [r["partial, no breaker"], r["partial, with breaker"]];
const [w0, w1] = [r["slowdown, no breaker"], r["slowdown, with breaker"]];
const [d0, d1] = [r["down, no breaker"], r["down, with breaker"]];
const scale = (600 * CYCLE) / (T - START);          // AY3: scaling to a 10-minute failure
console.log(`\nfailure-free day's cost: the breaker is queried on all ${h1.calls} calls and a ` +
  `${RATIO_RULE.windowSize}-record window is kept; ${h1.opens} false opens, ${h1.eventRejected} rejected valid events; ` +
  `partial failure opens ${p1.opens} times and event responses ${p0.eventResponses} -> ${p1.eventResponses} ` +
  `(${p0.eventResponses - p1.eventResponses} events that could have succeeded were rejected)`);
console.log(`failing day's gain (slowdown): tracking no slot ${w0.trackingNoSlot} -> ${w1.trackingNoSlot}, ` +
  `event slot-rounds ${w0.eventSlotRounds} -> ${w1.eventSlotRounds} (${b((100 * (w0.eventSlotRounds - w1.eventSlotRounds)) / w0.eventSlotRounds)}% less), ` +
  `calls to the broken dependency ${w0.calls} -> ${w1.calls}`);
console.log(`the same gain under down: tracking no slot ${d0.trackingNoSlot} -> ${d1.trackingNoSlot}, ` +
  `event slot-rounds ${d0.eventSlotRounds} -> ${d1.eventSlotRounds}, calls ${d0.calls} -> ${d1.calls}`);
console.log(`scaled to AY3's 10 minutes: calls saved under slowdown ` +
  `${((w0.calls - w1.calls) * scale).toFixed(0)}, tracking requests rescued ` +
  `${((w0.trackingNoSlot - w1.trackingNoSlot) * scale).toFixed(0)}`);

console.log(`\n${"rule".padEnd(18)}${"mode".padEnd(11)}${"detection round".padStart(15)}` +
  `${"slot-rounds to detection".padStart(26)}${"opens".padStart(8)}${"rejected valid events".padStart(25)}`);
const RULES = [
  ["consecutive 5", { kind: "consecutive", threshold: 5, openFor: 100 }],
  ["ratio 0.50 / 20", { kind: "ratio", windowSize: 20, threshold: 0.5, openFor: 100 }],
  ["ratio 0.10 / 20", { kind: "ratio", windowSize: 20, threshold: 0.1, openFor: 100 }],
];
for (const [name, rule] of RULES)
  for (const mode of ["healthy", "partial", "slowdown", "down"]) {
    const k = run({ mode, rule });
    const lost = mode === "partial" ? p0.eventResponses - k.eventResponses : k.eventRejected;
    console.log(`${name.padEnd(18)}${mode.padEnd(11)}${String(k.detectRound ?? "-").padStart(15)}` +
      `${String(k.detectSlotRounds ?? "-").padStart(26)}${String(k.opens).padStart(8)}` +
      `${String(mode === "healthy" || mode === "partial" ? lost : "-").padStart(25)}`);
  }
const longer = { ...RATIO_RULE, openFor: 200 }, narrower = { kind: "ratio", windowSize: 20, threshold: 0.1, openFor: 200 };
console.log(`sensitivity: if open duration were ${longer.openFor} rounds instead of ${RATIO_RULE.openFor}, calls made under slowdown ` +
  `would go ${w1.calls} -> ${run({ mode: "slowdown", rule: longer }).calls}; under the ratio 0.10 rule, rejected valid events ` +
  `on a failure-free day would go ${run({ mode: "healthy", rule: RULES[2][1] }).eventRejected} -> ` +
  `${run({ mode: "healthy", rule: narrower }).eventRejected}`);

model: 1200 rounds (12 s), failure at round 200, 32 slots; chosen rule: open if 0.5 of the last 20 calls fail, stay open 100 rounds

run                       tracking responses    tracking no slot     event responses      event rejected        event errors          calls made   event slot-rounds
healthy, no breaker                     5000                   0                1135                   0                  31                1166                7492
healthy, with breaker                   5000                   0                1135                   0                  31                1166                7492
partial, no breaker                     5000                   0                1021                   0                 145                1166                6637
partial, with breaker                   5000                   0                1021                   0                 145                1166                6637
slowdown, no breaker                    4329                 671                 190                   0                 826                1016               19312
slowdown, with breaker                  5000                   0                 190                 953                  23                 213                1646
down, no breaker                        5000                   0                 190                   0                 976                1166                4144
down, with breaker                      5000                   0                 190                 953                  23                 213                1285

failure-free day's cost: the breaker is queried on all 1166 calls and a 20-record window is kept; 0 false opens, 0 rejected valid events; partial failure opens 0 times and event responses 1021 -> 1021 (0 events that could have succeeded were rejected)
failing day's gain (slowdown): tracking no slot 671 -> 0, event slot-rounds 19312 -> 1646 (91.48% less), calls to the broken dependency 1016 -> 213
the same gain under down: tracking no slot 0 -> 0, event slot-rounds 4144 -> 1285, calls 1166 -> 213
scaled to AY3's 10 minutes: calls saved under slowdown 48180, tracking requests rescued 40260

rule              mode       detection round  slot-rounds to detection   opens    rejected valid events
consecutive 5     healthy                  -                         -       0                        0
consecutive 5     partial                  -                         -       0                        0
consecutive 5     slowdown                 4                       110      10                        -
consecutive 5     down                     4                        15      10                        -
ratio 0.50 / 20   healthy                  -                         -       0                        0
ratio 0.50 / 20   partial                  -                         -       0                        0
ratio 0.50 / 20   slowdown                 9                       220      10                        -
ratio 0.50 / 20   down                     9                        30      10                        -
ratio 0.10 / 20   healthy                  -                         -       4                      386
ratio 0.10 / 20   partial                 88                       117       9                      698
ratio 0.10 / 20   slowdown                67                        22      11                        -
ratio 0.10 / 20   down                    67                         3      11                        -
sensitivity: if open duration were 200 rounds instead of 100, calls made under slowdown would go 213 -> 208; under the ratio 0.10 rule, rejected valid events on a failure-free day would go 386 -> 741

These numbers belong to the measurement class and depend on the seed; the same seed gives the same table.

The first two rows give the pattern’s failure-free-day cost, and the cost is surprisingly small. On a healthy day the runs with and without the breaker are identical: 5000 tracking responses, 1135 event responses, 0 false opens, 0 rejected valid events. The cost is countable but small — the breaker is queried on all 1166 calls and keeps a 20-record window per dependency. A correctly sized threshold becomes invisible on a failure-free day.

The third and fourth rows give the gain. Under slowdown, without a breaker, 671 of 5000 tracking requests find no slot — the round-level counterpart of lesson 1’s blast-radius table. With the breaker, that number becomes 0: the tracking query is no longer affected by the slowdown of a dependency it has nothing to do with. The event leg’s slot-rounds fall from 19,312 to 1646, 91.48 percent less, and calls to the broken dependency drop from 1016 to 213. Scaled to AY3’s 10-minute slowdown, that is 48,180 calls saved and 40,260 tracking requests rescued.

The fifth and sixth rows show what the breaker does not rescue. Under down, tracking requests with no slot is already 0 without a breaker; a fast failure holds no slot, so there is no spread and nothing for the breaker to protect. The gain sits elsewhere: calls made fall from 1166 to 213, slot-rounds from 4144 to 1285. For a dependency that is down, the breaker cuts not the spread, but the wasted work.

One row also confesses the pattern’s own cost: 953 events are rejected in the with-breaker slowdown run. The breaker does not rescue the event flow, it fences it off. Those 953 rejected state events are still real events and have to go somewhere; where is not this pattern’s question.

The Threshold Is Chosen by Failure Mode

The second table runs three threshold rules across four modes, and shows that no single threshold can serve all four.

Consecutive failure count is the cheapest rule, but it never sees partial failure. The consecutive-5 rule catches both slowdown and down at round 4, and never opens falsely on a healthy day — but never opens under partial failure either. The reason is arithmetic: at a one-in-eight failure ratio, five consecutive failures is rare. Under partial failure that is correct; seven-eighths of calls still succeed, and cutting off would cost more than it gains.

It is not detection’s delay that changes by mode, but its cost. In the consecutive-5 rule, both down and slowdown are detected at round 4; concurrent calls accumulate five failures at the same rate. But slot-rounds held until detection are 15 under down and 110 under slowdown — 7.3 times. Under ratio-0.50 the gap widens further: 30 against 220. A threshold should be chosen not by call count but by how many slot-rounds that count costs; waiting twenty calls on a slow dependency equals waiting a hundred and twenty.

The Cost of a False Open

The third rule shows the reverse. The ratio-0.10 threshold catches slowdown and down far more cheaply — only 22 and 3 slot-rounds until detection — and even sees partial failure at round 88. Its cost sits in the other two columns.

On a healthy day this rule opens 4 times and rejects 386 valid events — while nothing is wrong with the dependency. Under partial failure it opens 9 times and rejects 698 valid events, even though those events had a seven-in-eight chance of succeeding. A false open is cutting off a healthy dependency, and its cost is rejected valid requests.

The sensitivity row says this cost depends on open duration: raising it from 100 to 200 rounds would drop calls made under slowdown from 213 to 208 — almost no gain — but raise rejected valid events on a healthy day from 386 to 741. Extending open duration buys little on a failing day and costs a lot on a failure-free one, because the gain depends on the breaker opening once, not on how long the failure lasts, while the loss is multiplied directly by the open duration.

Summary

• A circuit breaker takes a dependency out of the circuit; its closed, open, and half-open states were established in the Application Architecture course and are not repeated here. Across services, it protects the caller’s shared slot pool.
• A correctly sized threshold is invisible on a failure-free day: the runs with and without a breaker give the same numbers, 0 false opens. The cost is one check per call and a 20-record window per dependency.
• The gain under slowdown is large: tracking requests finding no slot go 671 → 0, the event leg’s slot-rounds 19,312 → 1646 (91.48 percent less), calls to the broken dependency 1016 → 213. AY3’s 10 minutes save 48,180 calls and rescue 40,260 tracking requests.
• Under down there is no spread to begin with (tracking no slot 0 → 0); the gain is wasted work: calls 1166 → 213, slot-rounds 4144 → 1285.
• Detection’s cost changes by mode, not its delay: the consecutive-5 rule catches both modes at round 4, but slot-rounds held until detection are 15 under down and 110 under slowdown.
• A false open cuts off a healthy dependency: ratio-0.10 rejects 386 valid events on a healthy day, 698 under partial failure; raising open duration to 200 rounds pushes the healthy-day loss to 741, while the failing-day gain only drops from 213 to 208.

Next Step

The breaker drew a boundary, but where it sits must be read carefully: protection starts after detection. Under ratio-0.50, slowdown is detected at round 9, and during those nine rounds the broken dependency draws 220 slot-rounds from the shared pool. Lowering the threshold shrinks that share but grows false opens; the two cannot shrink together. And a breaker sits on each dependency separately, while the pool is single: a slowdown with no breaker, or one under its threshold, still consumes the pool and drags down the neighboring flow. One question remains — can the spread be cut without depending on detection at all. The next lesson splits the pool itself: each flow gets its own slots, the resource a flow can consume is bounded from the start, and what that costs on a failure-free day is counted.