Bulkhead Pattern

The previous lesson placed the breaker’s state in the code and showed that protection only starts after enough failures accumulate, and never starts on rarely called paths. The breaker understands that a dependency is broken; it does not understand that the work waiting on it consumes the caller’s own resources. All three call sites run in the same process, from the same slot pool.

The bulkhead pattern and its effect on the system were built and measured in M19/K05 Resilience and Reliability. None of it repeats here. This lesson’s question is: where the pool is defined in the code, how many call sites using it changes, and which paths stay outside the limit.

Mechanism

Two dependencies: notification takes 300 ms, catalog takes 5 ms. Forty concurrent calls are made to each. Every dependency records its own peak concurrent request count — this is what the pool limit looks like from the dependency’s side. RS8 — the total slot count is equal under both placements: eight in the single pool, four plus four in the pool-per-dependency case. The comparison measures placement only while the total stays fixed.

// services.mjs — 8402 notification (slow), 8403 catalog (fast), 8401 loan: pools are defined here.
import { createServer, request, Agent } from "node:http";
const N = 40;                                       // 40 concurrent calls per stream
const s = { notification: { open: 0, peak: 0 }, catalog: { open: 0, peak: 0 } };
const service = (name, delay, port) => createServer((req, res) => {
  const d = s[name]; d.open += 1; d.peak = Math.max(d.peak, d.open);
  setTimeout(() => { d.open -= 1; res.end("{}"); }, delay);
}).listen(port);
service("notification", 300, 8402);
service("catalog", 5, 8403);

// >>> pool definition: two placements, same total slots
const pools = (placement, notif, cat) => {
  if (placement === "single") { const a = new Agent({ maxTotalSockets: notif + cat }); return () => a; }
  const t = { 8402: new Agent({ maxTotalSockets: notif }), 8403: new Agent({ maxTotalSockets: cat }) };
  return (port) => t[port];
};
// <<<
const call = (port, path, pool) => new Promise((resolve) => {
  const r = request({ port, path, agent: pool }, (y) => { y.resume(); y.on("end", resolve); });
  r.on("error", resolve); r.end();
});

createServer(async (req, res) => {                  // 8401 loan: runs one round start to finish
  const q = new URL(req.url, "http://x").searchParams;
  const poolFor = pools(q.get("placement"), Number(q.get("notif")), Number(q.get("cat")));
  const skip = Number(q.get("skip"));               // call sites that never use the pool at all
  for (const d of Object.values(s)) { d.open = 0; d.peak = 0; }
  const t0 = Date.now();
  let done = 0, catalogBefore = null;
  const slow = Array.from({ length: N }, (_, j) => call(8402, "/send", j < skip ? false : poolFor(8402)));
  const fast = Array.from({ length: N }, () => call(8403, "/book", poolFor(8403)).then(() => { done += 1; }));
  Promise.race(slow).then(() => { catalogBefore = done; });
  await Promise.all([...slow, ...fast]);
  res.end(JSON.stringify({ n: N, catalogBefore, notification: s.notification.peak, catalog: s.catalog.peak,
    duration: Math.round((Date.now() - t0) / 500) * 500 }));
}).listen(8401);

catalogBefore is this lesson’s main measure: how many catalog calls finished by the time the first notification call completes. Since catalog answers in five milliseconds, a stream with its own resources should finish all forty of its calls before notification gives even its first response. If it cannot, that means it found no slot.

The skip parameter produces a call site that never uses the pool at all: a request made with agent: false opens its own socket and never looks at the pool table. In the source this looks like no shortcoming at all; it is just another option.

// measure.mjs — where the pool is defined, the skipped path, and wrong sizing.
import { readFileSync } from "node:fs";
import { request } from "node:http";
const call = (path) => new Promise((c) => request({ port: 8401, path, agent: false }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(JSON.parse(g)));
}).end());
const print = (a, ...r) => console.log(String(a).padEnd(34) + r.map((x) => String(x).padStart(20)).join(""));

console.log("40 concurrent calls per stream: notification 300 ms, catalog 5 ms; total slots 8 under every placement");
console.log("the duration column is this run's measurement, rounded to 500 ms; the expected value is ceil(40/slots) x 300 ms");
print("placement", "catalog done before", "notification peak", "catalog peak", "duration ms");
for (const [name, q] of [["single shared pool", "placement=single&notif=4&cat=4&skip=0"],
  ["pool per dependency", "placement=separate&notif=4&cat=4&skip=0"],
  ["pool per dependency + 8 skipped", "placement=separate&notif=4&cat=4&skip=8"]]) {
  const r = await call(`/run?${q}`);
  print(name, `${r.catalogBefore} / ${r.n}`, r.notification, r.catalog, r.duration);
}

console.log("\nwrong sizing: slot counts under pool-per-dependency placement");
print("notification slots / catalog slots", "notification peak", "catalog peak", "duration ms");
for (const [notif, cat] of [[1, 4], [4, 4], [40, 4], [4, 1]]) {
  const r = await call(`/run?placement=separate&notif=${notif}&cat=${cat}&skip=0`);
  print(`${notif} / ${cat}`, r.notification, r.catalog, r.duration);
}

const source = readFileSync("services.mjs", "utf8");
const block = source.match(/\/\/ >>> pool definition[^\n]*\n([\s\S]*?)\/\/ <<</)[1].trim().split("\n").length;
const sites = (source.match(/call\(\d+, "/g) || []).length;
console.log(`\npool definition is ${block} lines, in one place; outbound call sites ${sites} and none of them changed — ` +
  `what moves the pool to the call site is ${(source.match(/poolFor\(\d+\)/g) || []).length} poolFor(port) expressions.`);
process.exit(0);

node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1; node measure.mjs; kill $SP

40 concurrent calls per stream: notification 300 ms, catalog 5 ms; total slots 8 under every placement
the duration column is this run's measurement, rounded to 500 ms; the expected value is ceil(40/slots) x 300 ms
placement                          catalog done before   notification peak        catalog peak         duration ms
single shared pool                              0 / 40                   8                   1                2000
pool per dependency                            40 / 40                   4                   4                3000
pool per dependency + 8 skipped                40 / 40                  12                   4                2500

wrong sizing: slot counts under pool-per-dependency placement
notification slots / catalog slots   notification peak        catalog peak         duration ms
1 / 4                                                1                   4               12000
4 / 4                                                4                   4                3000
40 / 4                                              40                   4                 500
4 / 1                                                4                   1                3000

pool definition is 5 lines, in one place; outbound call sites 2 and none of them changed — what moves the pool to the call site is 2 poolFor(port) expressions.

Where the Pool Sits in the Code

The last line gives the placement’s code cost: five lines, in one place. Neither outbound call site changed at all; what moves the pool to the call site is two expressions of the form poolFor(port). This is where the bulkhead parts ways with the previous three patterns. A timeout adds a behavior, a retry adds a loop, a breaker holds state — all change what the call site does. A pool only says which resource the call uses; the call site’s code stays the same.

The result: pool placement is nearly invisible in the source. Whether a call site uses the right pool cannot be told by looking at the call site itself; you must check which expression was passed in. Writing false on one line instead of poolFor(8402) is syntactically flawless.

Two Placements, the Same Eight Slots

The first two rows of the first table split the same eight slots two different ways.

Under the single shared pool, catalog done before is 0 / 40 — not one catalog call finished before the first notification response arrived. The reason is in the notification-peak column: the slow dependency holds all eight slots, and catalog can find at most one. Nothing is wrong with catalog, it answers in five milliseconds; even so, all forty of its requests wait. When the pool is shared, one dependency’s slowness becomes latency for a stream that has nothing to do with it.

Under pool per dependency, the same column is 40 / 40. Catalog finds its own four slots and finishes all forty calls before notification gives even its first response. Notification’s concurrency drops from eight to four, and duration rises from 2000 ms to 3000 ms — the bulkhead’s cost, shown directly in the table. The pattern cuts off the contagion without ever depending on detecting it: no threshold, no counter, no learning period. The limit was set from the start.

The Skipped Path

The third row is this lesson’s most valuable measure. The pool configuration did not change — it still declares four slots for notification. Eight of the forty call sites do not use the pool at all; they open their own sockets. The result: notification concurrency is 12 instead of 4.

The pool’s limit says four, the dependency sees twelve. The excess is exactly the skipped-call-site count, appearing nowhere in the configuration: someone reading the slot count sees four, someone watching the dependency sees twelve. The previous two lessons’ coverage problem is at its sneakiest here, because pool placement does not change the call site’s code, so it does not change the gap either — a skipped call site looks nearly identical to a wrapped one.

Notably, the neighboring stream still comes out fine: catalog done before is still 40 / 40. The skipped path breaches the notification pool’s limit but not the catalog pool. The bulkhead’s protection is that the leak stays inside its own bulkhead — the leak still voids that bulkhead’s promise regardless.

Wrong Sizing

The second table shows both ends of pool sizing, and both leave the pattern useless, but in different directions.

One slot queues the notification stream: concurrency 1, duration 12000 ms. Forty calls run back to back while nothing is wrong with the dependency. Here the bulkhead produces not protection but a bottleneck; the neighboring stream comes out fine, but the protected stream itself becomes the slowest part of the system.

Forty slots is the opposite end: concurrency 40, duration 500 ms. The pool exists in the code, is defined, is named, and binds nothing. This is the most dangerous configuration of all — it is indistinguishable from a system with no pattern at all, yet anyone reading the source sees the bulkhead set up.

The last row says the limit applies just as much on the neighboring side: when the catalog pool drops to one, catalog concurrency becomes 1 too. Every pool is a ceiling on its own stream — the bulkhead limits not only the slow dependency but the fast one too, and wrong sizing pays its cost in both directions the same way.

Testability

// test/pool.test.mjs — is the pool limit actually binding.
import { test } from "node:test";
import assert from "node:assert/strict";
import { request } from "node:http";
const run = (path) => new Promise((c) => request({ port: 8401, path, agent: false }, (r) => {
  let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(JSON.parse(g)));
}).end());

test("pool per dependency saves the neighboring stream", async () => {
  assert.equal((await run("/run?placement=single&notif=4&cat=4&skip=0")).catalogBefore, 0);
  assert.equal((await run("/run?placement=separate&notif=4&cat=4&skip=0")).catalogBefore, 40);
});
test("a skipped call site exceeds the pool limit", async () => {
  assert.equal((await run("/run?placement=separate&notif=4&cat=4&skip=0")).notification, 4);
  assert.equal((await run("/run?placement=separate&notif=4&cat=4&skip=8")).notification, 12);   // 4 + 8
});
test("the pool limit binds only its own dependency", async () => {
  for (const notif of [1, 4, 40]) assert.equal((await run(`/run?placement=separate&notif=${notif}&cat=4&skip=0`)).catalog, 4);
});

node services.mjs > /dev/null 2>&1 & SP=$!
sleep 1
node --test --test-force-exit --test-reporter=tap test/pool.test.mjs |
  grep -E "^(ok|not ok|# (tests|pass|fail) )"
kill $SP

ok 1 - pool per dependency saves the neighboring stream
ok 2 - a skipped call site exceeds the pool limit
ok 3 - the pool limit binds only its own dependency
# tests 3
# pass 3
# fail 0

The bulkhead’s testability differs from the previous three patterns: its behavior shows up not in the caller’s code but in the concurrency the dependency observes. A timeout or breaker test can be written from the error the caller receives; a pool test can only be written by counting how many connections the other side sees. The second test uses this directly: the pool says four, the assertion is built on twelve, and the skipped call site becomes a number this way. The third pins the limit’s scope — whatever the notification pool is, catalog stays at four.

Summary

• The bulkhead’s effect was measured in M19/K05 Resilience and Reliability; what is measured here is where the pool is defined. The definition is five lines, one place; neither call site changes, and what moves the pool is the poolFor(port) expression passed into the call.
• When the same eight slots are pooled together, the slow dependency holds all of them and catalog calls finished before the first notification response are 0 / 40; catalog concurrency drops to 1.
• Under pool per dependency, the same number is 40 / 40. The cost: notification concurrency from 8 to 4, duration from 2000 ms to 3000 ms. Protection does not depend on a threshold or learning — it was set from the start.
• Eight call sites bypassing the pool deliver twelve concurrent connections while the configuration says four slots. The excess is exactly the skipped-call-site count, invisible in the configuration.
• Both ends of pool sizing leave the pattern useless: one slot drops concurrency to 1 and pushes duration to 12000 ms, forty slots binds nothing (concurrency 40, duration 500 ms) and looks like a bulkhead was set up.
• Every pool is a ceiling on its own stream: the catalog pool at 1 makes catalog concurrency 1 too. A pool test can only be written by counting the concurrency the dependency observes.

Next Step

All four patterns did the same thing: placed a limit and refused the request beyond it. The timeout left the waiter waiting, the retry multiplied and gave up, the breaker never made the call, the bulkhead handed out no slot. What was left behind every time was the same thing — an unmet request. Where that rejected request goes was never a question for any of these four patterns; each contained the failure, none met it. The next lesson fills this gap: a request that hits the limit gets an incomplete but usable response instead of a full one, so the system returns something even while degrading.