Bulkhead Pattern

The circuit breaker cut the spread, but its protection started only after detection: until the threshold filled, the broken dependency drew slots from the shared pool — 220 slot-rounds under ratio-0.50. Lowering the threshold shrank that share but grew false opens. This lesson’s question: can the spread be cut without detection at all.

It can, and the way is to split the pool. The bulkhead pattern splits the caller’s resources into separate pools by flow or by dependency; a flow can only exhaust its own pool, and when it does, the neighboring flow feels nothing. The name comes from a ship hull’s watertight compartments: when one compartment floods, the ship does not sink, that compartment does. Bulkhead shares its root with the file splitting in the Shell Programming course and the remainder in modulo hashing from the Traffic Layer course, but is a separate concept: here, what gets split is the resource pool.

The difference from the breaker fits in one sentence. The breaker understands the dependency is broken and stops the call; the bulkhead understands nothing, it only bounds from the start the resource a broken flow can consume. One decides, the other sets a limit.

Same Pool, Two Layouts

The run below is again a model. It takes up lesson 1’s gateway slot pool; the only difference is that arrivals are no longer smooth but Poisson, drawn from a seeded generator. The bulkhead’s cost shows up only under bursty arrival: under smooth flow, each flow fills its own share exactly, and sharing gains nothing.

The failure scenario is lesson 1’s: carrier validation slows down, the event leg’s hold climbs from 6 to 22 rounds, and by AY3 this happens three times a month, ten minutes each.

// bulkhead/run.mjs — comparing a single pool against a bulkhead layout in the same failure scenario.
// MODEL: a round is an abstract step, arrivals are Poisson from a seeded generator, the mode is a parameter.
const TOTAL = 32, CYCLE = 100, T = 4000, START = 1000;   // AY2, AY1; 40 s run, failure at second 10
const TRACKING = 416.67, EVENT = 97.22;                   // K01: peak read and peak write req/s
const HOLD = { tracking: 4, event: 6 }, SLOW = 22;        // lesson 01: gateway leg durations

function run({ split, mode }) {                           // split: single pool if null
  let s = 20260801 % 2147483647;
  const random = () => (s = (s * 48271) % 2147483647) / 2147483647;
  const poisson = (lam) => { const L = Math.exp(-lam); let k = 0, p = 1;
    do { k += 1; p *= random(); } while (p > L); return k - 1; };
  const pool = split === null ? { shared: TOTAL } : { tracking: split, event: TOTAL - split };
  const busy = Object.fromEntries(Object.keys(pool).map((h) => [h, []]));   // busy slots' end round
  const poolFor = (flow) => (split === null ? "shared" : flow);
  const c = { trackingResponses: 0, trackingDropped: 0, eventResponses: 0, eventDropped: 0, idle: 0, busyRounds: 0 };

  for (let t = 0; t < T; t += 1) {
    for (const h of Object.keys(pool)) busy[h] = busy[h].filter((x) => x > t);
    c.busyRounds += Object.values(busy).reduce((a, x) => a + x.length, 0);
    const arrivals = [];
    for (let n = poisson(TRACKING / CYCLE); n > 0; n -= 1) arrivals.push("tracking");
    for (let n = poisson(EVENT / CYCLE); n > 0; n -= 1) arrivals.push("event");
    for (let i = arrivals.length - 1; i > 0; i -= 1) {
      const j = Math.floor(random() * (i + 1));
      [arrivals[i], arrivals[j]] = [arrivals[j], arrivals[i]];
    }
    for (const flow of arrivals) {
      const h = poolFor(flow);
      const duration = flow === "event" && mode === "slowdown" && t >= START ? SLOW : HOLD[flow];
      if (busy[h].length < pool[h]) { busy[h].push(t + duration); c[`${flow}Responses`] += 1; continue; }
      c[`${flow}Dropped`] += 1;
      for (const o of Object.keys(pool)) if (o !== h) c.idle += pool[o] - busy[o].length;
    }
  }
  return c;
}

const b = (x, n = 2) => x.toFixed(n);
const METRICS = ["trackingResponses", "trackingDropped", "eventResponses", "eventDropped", "idle"];
const HEADERS = ["tracking responses", "tracking dropped", "event responses", "event dropped", "idle slots"];
console.log(`model: ${T} rounds (${T / CYCLE} s), failure at round ${START}, ${TOTAL} slots total; ` +
  `Poisson arrivals (seed 20260801), tracking ${(TRACKING / CYCLE).toFixed(4)} and event ${(EVENT / CYCLE).toFixed(4)} arrivals/round`);

console.log(`\n${"layout".padEnd(22)}${"day".padEnd(12)}` + HEADERS.map((x) => x.padStart(20)).join("") +
  `${"average busy slots".padStart(20)}`);
const r = {};
for (const [name, split] of [["single pool (32)", null], ["bulkhead (24/8)", 24]])
  for (const mode of ["healthy", "slowdown"]) {
    const k = run({ split, mode });
    r[`${name}|${mode}`] = k;
    console.log(`${name.padEnd(22)}${mode.padEnd(12)}` + METRICS.map((x) => String(k[x]).padStart(20)).join("") +
      `${b(k.busyRounds / T).padStart(20)}`);
  }

const single0 = r["single pool (32)|healthy"], bulk0 = r["bulkhead (24/8)|healthy"];
const single1 = r["single pool (32)|slowdown"], bulk1 = r["bulkhead (24/8)|slowdown"];
const scale = (600 * CYCLE) / (T - START);          // AY3: scaling to a 10-minute failure
console.log(`\nfailure-free day's cost: tracking dropped ${single0.trackingDropped} -> ${bulk0.trackingDropped}, event dropped ` +
  `${single0.eventDropped} -> ${bulk0.eventDropped}; average busy slots ${b(single0.busyRounds / T)} -> ${b(bulk0.busyRounds / T)} ` +
  `(utilization ${b(single0.busyRounds / T / TOTAL, 3)} -> ${b(bulk0.busyRounds / T / TOTAL, 3)}); idle slots ${single0.idle} -> ${bulk0.idle}`);
console.log(`failing day's gain: tracking dropped ${single1.trackingDropped} -> ${bulk1.trackingDropped} ` +
  `(${b((100 * (single1.trackingDropped - bulk1.trackingDropped)) / single1.trackingDropped, 1)}% less), event dropped ` +
  `${single1.eventDropped} -> ${bulk1.eventDropped}; tracking requests rescued in AY3's 10 minutes ` +
  `${((single1.trackingDropped - bulk1.trackingDropped) * scale).toFixed(0)}`);

console.log(`\n${"pool split".padEnd(20)}${"healthy tracking dropped".padStart(26)}` +
  `${"healthy event dropped".padStart(23)}${"slowdown tracking dropped".padStart(27)}` +
  `${"slowdown event dropped".padStart(24)}`);
for (const split of [null, 28, 24, 20, 16]) {
  const s = run({ split, mode: "healthy" }), y = run({ split, mode: "slowdown" });
  const label = split === null ? `single pool (${TOTAL})` : `${split}/${TOTAL - split}`;
  console.log(`${label.padEnd(20)}${String(s.trackingDropped).padStart(26)}${String(s.eventDropped).padStart(23)}` +
    `${String(y.trackingDropped).padStart(27)}${String(y.eventDropped).padStart(24)}`);
}
const sensitivity = run({ split: 27, mode: "healthy" });
console.log(`sensitivity: with 27/5 instead of 24/8, failure-free tracking dropped would go ${bulk0.trackingDropped} -> ` +
  `${sensitivity.trackingDropped}, event dropped ${bulk0.eventDropped} -> ${sensitivity.eventDropped}; the event pool ` +
  `runs at ${b(((EVENT / CYCLE) * HOLD.event) / 8, 3)} utilization at 8 slots, ` +
  `${b(((EVENT / CYCLE) * HOLD.event) / 5, 3)} at 5`);

model: 4000 rounds (40 s), failure at round 1000, 32 slots total; Poisson arrivals (seed 20260801), tracking 4.1667 and event 0.9722 arrivals/round

layout                day           tracking responses    tracking dropped     event responses       event dropped          idle slots  average busy slots
single pool (32)      healthy                    16542                  78                3931                  27                   0               17.31
single pool (32)      slowdown                   13966                2654                3358                 600                   0               24.00
bulkhead (24/8)       healthy                    16448                 172                3552                 406                4239               16.77
bulkhead (24/8)       slowdown                   16448                 172                1947                2011               19196               18.90

failure-free day's cost: tracking dropped 78 -> 172, event dropped 27 -> 406; average busy slots 17.31 -> 16.77 (utilization 0.541 -> 0.524); idle slots 0 -> 4239
failing day's gain: tracking dropped 2654 -> 172 (93.5% less), event dropped 600 -> 2011; tracking requests rescued in AY3's 10 minutes 49640

pool split            healthy tracking dropped  healthy event dropped  slowdown tracking dropped  slowdown event dropped
single pool (32)                            78                     27                       2654                     600
28/4                                        11                   1754                         11                    2876
24/8                                       172                    406                        172                    2011
20/12                                      868                     37                        868                    1416
16/16                                     2591                      0                       2591                     942
sensitivity: with 27/5 instead of 24/8, failure-free tracking dropped would go 172 -> 26, event dropped 406 -> 1303; the event pool runs at 0.729 utilization at 8 slots, 1.167 at 5

These numbers belong to the measurement class and depend on the seed.

The Bulkhead Does Not Change the Failing Day

Under the bulkhead layout, the tracking query’s dropped-request count is 172 on a healthy day and 172 on a slowdown day. The same number. Whether the dependency slows down or not changes nothing for the tracking flow — because the event flow’s slots are capped at eight, and those eight can never touch the tracking flow’s twenty-four.

In a single pool, the same failure drops 2654 tracking requests. The bulkhead cuts that to 172: 93.5 percent less. Scaled to AY3’s 10-minute slowdown, that is 49,640 tracking requests rescued. And this protection depends on no detection at all; the breaker’s nine-round detection window does not exist here, because no decision is made.

What the pattern does not do sits in the same row. Under the bulkhead, the event flow’s dropped requests rise from 600 to 2011. The bulkhead does not rescue the event flow, it chokes it earlier instead; what it rescues is the neighbor. The pattern’s name already says so — the flooding compartment keeps filling.

The Failure-Free Day’s Cost: Lost Sharing

The first two rows give the cost, and it is not small. On a healthy day, the single pool drops 78 tracking and 27 event requests; 105 total. The same day, the bulkhead layout drops 172 tracking and 406 event requests; 578 total. With no failure at all, the bulkhead multiplies dropped requests by 5.5.

The reason sits in the idle slots column. Whenever a request drops under the bulkhead, the slots sitting idle in the other pool are added up, and on a healthy day this total comes to 4239: 4239 times, a request was rejected while an idle slot sat in the neighboring pool. In a single pool this number is zero by definition, because an idle slot does not care whose it is. Average busy slots fall from 17.31 to 16.77, utilization from 0.541 to 0.524 — not doing the same work with fewer resources, but doing it with less of the same resource.

What is lost is called sharing: when two bursty flows share a single pool, one’s peak overlaps the other’s trough, and the combined peak comes out smaller than the sum of the peaks. The bulkhead forbids that overlap. This is the failure-free day’s cost, and it is directly measurable: 94 extra dropped tracking requests, 379 extra dropped event requests, 0.017 less utilization.

Bulkhead Size Trades the Two Days Against Each Other

The second table shows the sizing choice, and that no single split is right.

A 28/4 split protects the tracking flow almost perfectly: under slowdown, only 11 requests drop, against the single pool’s 2654. Its cost sits on the failure-free day — the event flow’s four slots are not enough, and 1754 events drop on a healthy day. A 16/16 split does the opposite: the event flow never drops on a failure-free day, but the tracking flow loses 2591 requests — as much as the single pool loses on the failing day. A wrongly sized bulkhead does on a failure-free day as much damage as a failing one.

24/8 sits between the two, and its choice is measurable: the event flow’s average load is 0.9722 arrivals/round times 6 rounds, that is, 5.83 slots; eight slots is 1.37 times that. The sensitivity row gives the cost of narrowing the share — at 27/5, tracking dropped falls from 172 to 26, but event dropped climbs from 406 to 1303, because the event pool’s utilization goes from 0.729 to 1.167, past capacity. A bulkhead shrunk below its own flow’s usual load fails without waiting for a failure.

The bulkhead and the breaker do not replace each other. The bulkhead cuts the spread without detection, but does not cut calls to the broken dependency: in the bulkhead slowdown run, the event flow still calls the broken dependency and still exhausts its own eight slots. The breaker stops those calls, but has to understand first. Each moves a different number — the bulkhead moves the neighbor’s dropped requests, the breaker moves calls to the broken dependency.

Summary

• A bulkhead pattern splits the caller’s resource pool by flow; a flow can only exhaust its own pool. Its difference from the breaker is that it decides nothing — the boundary is set from the start, no detection needed.
• The bulkhead makes the failing day invisible to the neighboring flow: under the bulkhead layout, the tracking query’s dropped requests are 172 on both the healthy and the slowdown day. In a single pool, the same failure drops 2654 requests; the gain is 93.5 percent, 49,640 requests in AY3’s 10 minutes.
• The bulkhead does not rescue the broken flow, it chokes it earlier: the event flow’s dropped requests rise from 600 to 2011.
• The failure-free day’s cost is lost sharing: dropped requests go from 105 to 578 (5.5 times), utilization from 0.541 to 0.524; 4239 times a request is rejected while an idle slot sits in the neighboring pool.
• Size trades the two days against each other: 28/4 drops only 11 tracking requests under slowdown but 1754 event requests on a failure-free day; 16/16 never drops the event flow but loses 2591 tracking requests on a failure-free day.
• A bulkhead must carry its own flow’s usual load: the event flow needs 5.83 slots, 8 slots is 1.37 times that; at 5 slots utilization becomes 1.167 and the bulkhead fails without a failure.

Next Step

Together, the bulkhead and the breaker draw one boundary: the broken dependency stays in its own bulkhead and, after a while, is not called at all. Both stand on the same assumption — a failed call is made once. A real caller does not meet failure that way; it retries, since most errors are transient and a second attempt usually succeeds. Retrying changes everything these two patterns measure: bulkhead slots run out faster, more failures land in the breaker’s window, and, most importantly, the broken dependency receives more requests from the system already straining it. The next lesson measures that multiplier: how far the request rate reaching the dependency multiplies if every chain layer retries, how the storm stretches the failure out, and which layer should hold the retry budget.