Retries and Storm Risk

The bulkhead and the circuit breaker fenced off the spread, but both stood on the same assumption: a failed call is made once. A real caller does not behave that way. Most errors are transient, a second attempt usually succeeds, and retries get added to nearly every layer. This lesson’s question is what that decision does to the broken dependency.

The decision to retry, exponential backoff, a cap, full jitter, and the retry budget were established and measured in the Application Architecture course; the Caching, Queues and Asynchronous Processing course measured backoff schedules. In every run below, backoff and full jitter are explicit and the seed is fixed — so only the multiplier decides the comparison. Jitter’s arithmetic is not recomputed here.

What gets measured is something else. The chain is three layers deep: the gateway calls the event endpoint, the event endpoint calls the state store. Each layer makes a reasonable decision on its own — “retry three times on failure” — and none knows the others retry too. The result multiplies: three layers retrying three times each can put twenty-seven calls on the bottom dependency for one external request. This is called a retry storm, and by definition it happens at the worst moment.

Two Failure Profiles

The run below is a model: a round is an abstract step, arrivals are Poisson from a seeded generator, and the dependency’s capacity is calls per round. No real cluster, queue, or chaos tool is set up.

AY6 — the dependency’s usual capacity is 6 calls per round (600 req/s). Rationale: K01’s peak write rate is 97.22 req/s, that is, 0.9722 calls per round; a sixfold margin makes overflow rare under bursty arrival. Its sensitivity is given below together with the retry budget.

The failure runs under two profiles, because the multiplier behaves differently in each. Outage: the dependency answers no call for one second (capacity 0) — lesson 1’s down mode. Degradation: its capacity drops to a sixth for ten seconds (capacity 1) — a portion of calls still succeed.

// storm/run.mjs — the retry storm under the call amplification. MODEL: a round is an abstract step,
// arrivals are Poisson, backoff is exponential with full jitter (its arithmetic measured in M14/K06).
const CYCLE = 100, T = 4000, START = 500;       // AY1: 1 round = 10 ms; 40 s run, failure at second 5
const EVENT = 97.22, USUAL = 6;                 // K01 peak write req/s; AY6: usual capacity calls/round
const BASE = 2, CAP = 32, RARE = 0.025;         // backoff base/cap (rounds); AY4
const SCENARIOS = {                             // window duration and capacity kept during the window
  healthy: { duration: 0, capacity: USUAL },
  "outage (1 s)": { duration: 100, capacity: 0 },
  "degradation (10 s)": { duration: 1000, capacity: 1 },
};

function run({ tries, budget, scenario }) {
  let s = 20260802 % 2147483647;
  const random = () => (s = (s * 48271) % 2147483647) / 2147483647;
  const poisson = (lam) => { const L = Math.exp(-lam); let k = 0, p = 1;
    do { k += 1; p *= random(); } while (p > L); return k - 1; };
  const { duration, capacity: broken } = SCENARIOS[scenario];
  const allowed = tries.reduce((a, x) => a * x, 1);   // allowed retries = product across layers
  const schedule = new Map();
  const enqueue = (t, i) => { if (t < T) (schedule.get(t) ?? schedule.set(t, []).get(t)).push(i); };
  const c = { requests: 0, calls: 0, primary: 0, extra: 0, successful: 0, dropped: 0, peak: 0, windowCalls: 0 };

  for (let t = 0; t < T; t += 1) {
    const inWindow = duration > 0 && t >= START && t < START + duration;
    const capacity = inWindow ? broken : USUAL;
    for (let n = poisson(EVENT / CYCLE); n > 0; n -= 1) { c.requests += 1; enqueue(t, { n: 0 }); }
    const thisRound = schedule.get(t) ?? [];
    c.calls += thisRound.length; c.peak = Math.max(c.peak, thisRound.length);
    if (inWindow) c.windowCalls += thisRound.length;
    thisRound.forEach((item, i) => {
      if (item.n === 0) c.primary += 1; else c.extra += 1;
      if (i < capacity && (duration > 0 || random() >= RARE)) { c.successful += 1; return; }
      const budgetOk = budget === null || c.extra <= budget * c.primary;
      if (item.n + 1 >= allowed || budgetOk === false) { c.dropped += 1; return; }
      const backoff = Math.min(CAP, BASE * 2 ** item.n);   // exponential backoff + full jitter
      enqueue(t + 1 + Math.floor(random() * backoff), { n: item.n + 1 });
    });
    schedule.delete(t);
  }
  return { ...c, allowed, multiplier: c.calls / c.requests, windowRate: duration === 0 ? 0 : c.windowCalls / duration };
}

const b = (x, n = 2) => x.toFixed(n);
const CONFIGS = [                                // chain: gateway -> event endpoint -> state store
  ["no retries", [1, 1, 1], null],
  ["gateway only (3)", [3, 1, 1], null],
  ["two layers (3x3)", [3, 3, 1], null],
  ["three layers (3x3x3)", [3, 3, 3], null],
  ["three layers + budget 0.30", [3, 3, 3], 0.3],
];
console.log(`model: ${T} rounds (${T / CYCLE} s), failure at round ${START}; usual capacity ${USUAL} calls/round ` +
  `(${USUAL * CYCLE} req/s), event arrivals ${(EVENT / CYCLE).toFixed(4)}/round; seed 20260802`);

const r = {};
for (const scenario of Object.keys(SCENARIOS)) {
  const p = SCENARIOS[scenario];
  console.log(`
-- ${scenario}${p.duration === 0 ? " (AY4: 0.025 rare failure)"
    : `: for ${p.duration} rounds, capacity ${p.capacity}`} --`);
  console.log(`${"configuration".padEnd(25)}${"allowed".padStart(8)}${"calls".padStart(8)}${"multiplier".padStart(11)}` +
    `${"peak calls/round".padStart(18)}${"successful".padStart(11)}${"dropped".padStart(9)}` +
    (p.duration === 0 ? "" : `${"window calls/round".padStart(20)}${"times usual rate".padStart(19)}`));
  for (const [name, tries, budget] of CONFIGS) {
    const k = run({ tries, budget, scenario });
    r[`${name}|${scenario}`] = k;
    console.log(`${name.padEnd(25)}${String(k.allowed).padStart(8)}${String(k.calls).padStart(8)}` +
      `${`x${b(k.multiplier)}`.padStart(11)}${String(k.peak).padStart(18)}${String(k.successful).padStart(11)}` +
      `${String(k.dropped).padStart(9)}` + (p.duration === 0 ? ""
        : `${b(k.windowRate).padStart(20)}${`x${b(k.windowRate / (EVENT / CYCLE), 1)}`.padStart(19)}`));
  }
}

const at = (name, scenario) => r[`${name}|${scenario}`];
const OUTAGE = "outage (1 s)", DEGRADATION = "degradation (10 s)";
const [s0, s27] = [at("no retries", "healthy"), at("three layers (3x3x3)", "healthy")];
const [k0, k3, k9, k27, kb] = CONFIGS.map(([name]) => at(name, OUTAGE));
const [z0, z9, z27, zb] = [at("no retries", DEGRADATION), at("two layers (3x3)", DEGRADATION),
  at("three layers (3x3x3)", DEGRADATION), at("three layers + budget 0.30", DEGRADATION)];
console.log(`
failure-free day's cost: calls ${s0.calls} -> ${s27.calls} (multiplier x${b(s27.multiplier)}), ` +
  `reaching the dependency ${b((s0.calls / T) * CYCLE)} -> ${b((s27.calls / T) * CYCLE)} req/s; ` +
  `no difference between allowed 1 and 27, because the failure ratio is 0.025`);
console.log(`failing day's gain (outage): dropped requests ${k0.dropped} -> ${k3.dropped} -> ${k9.dropped} -> ` +
  `${k27.dropped}; successful ${k0.successful} -> ${k27.successful}`);
console.log(`the storm's growth (outage): window calls/round ${b(k0.windowRate)} -> ${b(k3.windowRate)} -> ` +
  `${b(k9.windowRate)} -> ${b(k27.windowRate)} = x${b(k27.windowRate / (EVENT / CYCLE), 1)} the usual rate; ` +
  `arithmetic upper bound x${k27.allowed}`);
console.log(`under degradation the multiplier follows not allowance but failure ratio: allowed 9 and 27 give the same calls ` +
  `(${z9.calls} and ${z27.calls}), window rate ${b(z27.windowRate)}`);
console.log(`budget 0.30 spread across the chain: outage calls ${k27.calls} -> ${kb.calls}, window rate ` +
  `${b(k27.windowRate)} -> ${b(kb.windowRate)}, dropped ${k27.dropped} -> ${kb.dropped}; degradation calls ` +
  `${z27.calls} -> ${zb.calls}, dropped ${z0.dropped} -> ${z27.dropped} -> ${zb.dropped}`);
console.log(`upper bound independent of the run: at d layers retrying r times, multiplier r^d; for r=3, ` +
  `${[1, 2, 3, 4].map((d) => `d=${d} -> ${3 ** d}`).join(", ")}`);
const THRESHOLD = 20;                            // AY2: timeout 20 rounds = K01's 200 ms threshold
const worst = Array.from({ length: 27 }, (_, n) => 1 + Math.min(CAP, BASE * 2 ** n))
  .reduce((a, x) => a + x, 0);
console.log(`27 retries' worst-case total wait is ${worst} rounds = ${worst / CYCLE} s, ${b(worst / THRESHOLD, 1)} ` +
  `times K01's ${THRESHOLD}-round read threshold`);
const narrow = run({ tries: [3, 3, 3], budget: 0.1, scenario: OUTAGE });
console.log(`sensitivity: at budget 0.10 instead of 0.30, outage calls would go ${kb.calls} -> ${narrow.calls}, ` +
  `window rate ${b(kb.windowRate)} -> ${b(narrow.windowRate)}, dropped ${kb.dropped} -> ${narrow.dropped}`);

model: 4000 rounds (40 s), failure at round 500; usual capacity 6 calls/round (600 req/s), event arrivals 0.9722/round; seed 20260802

-- healthy (AY4: 0.025 rare failure) --
configuration             allowed   calls multiplier  peak calls/round successful  dropped
no retries                      1    3832      x1.00                 6       3736       96
gateway only (3)                3    3953      x1.03                 6       3851        0
two layers (3x3)                9    3953      x1.03                 6       3851        0
three layers (3x3x3)           27    3953      x1.03                 6       3851        0
three layers + budget 0.30      27    3953      x1.03                 6       3851        0

-- outage (1 s): for 100 rounds, capacity 0 --
configuration             allowed   calls multiplier  peak calls/round successful  dropped  window calls/round   times usual rate
no retries                      1    3855      x1.00                 5       3775       80                0.80               x0.8
gateway only (3)                3    4071      x1.05                 8       3782       94                2.89               x3.0
two layers (3x3)                9    4435      x1.15                13       3850       20                5.77               x5.9
three layers (3x3x3)           27    4523      x1.17                17       3870        0                6.24               x6.4
three layers + budget 0.30      27    4065      x1.05                10       3777      103                2.88               x3.0

-- degradation (10 s): for 1000 rounds, capacity 1 --
configuration             allowed   calls multiplier  peak calls/round successful  dropped  window calls/round   times usual rate
no retries                      1    3855      x1.00                 5       3534      321                0.95               x1.0
gateway only (3)                3    4913      x1.28                 8       3754       94                2.01               x2.1
two layers (3x3)                9    5296      x1.39                 9       3803        0                2.39               x2.5
three layers (3x3x3)           27    5296      x1.39                 9       3803        0                2.39               x2.5
three layers + budget 0.30      27    4288      x1.11                 7       3643      220                1.40               x1.4

failure-free day's cost: calls 3832 -> 3953 (multiplier x1.03), reaching the dependency 95.80 -> 98.83 req/s; no difference between allowed 1 and 27, because the failure ratio is 0.025
failing day's gain (outage): dropped requests 80 -> 94 -> 20 -> 0; successful 3775 -> 3870
the storm's growth (outage): window calls/round 0.80 -> 2.89 -> 5.77 -> 6.24 = x6.4 the usual rate; arithmetic upper bound x27
under degradation the multiplier follows not allowance but failure ratio: allowed 9 and 27 give the same calls (5296 and 5296), window rate 2.39
budget 0.30 spread across the chain: outage calls 4523 -> 4065, window rate 6.24 -> 2.88, dropped 0 -> 103; degradation calls 5296 -> 4288, dropped 321 -> 0 -> 220
upper bound independent of the run: at d layers retrying r times, multiplier r^d; for r=3, d=1 -> 3, d=2 -> 9, d=3 -> 27, d=4 -> 81
27 retries' worst-case total wait is 793 rounds = 7.93 s, 39.6 times K01's 20-round read threshold
sensitivity: at budget 0.10 instead of 0.30, outage calls would go 4065 -> 3926, window rate 2.88 -> 1.54, dropped 103 -> 90

These numbers belong to the measurement class and depend on the seed.

The Multiplier Sleeps on a Failure-Free Day

The first table gives the pattern’s failure-free-day cost, and it is small: calls climb from 3832 to 3953, the multiplier is 1.03, the rate reaching the dependency goes from 95.80 to 98.83 req/s. In exchange, dropped requests fall from 96 to 0 — on a failure-free day, retrying fully absorbs rare timeouts for a three-percent cost.

What matters is the column that never changes. As allowed retries climb from 1 to 27, the call count stays flat. The reason is arithmetic: at a 0.025 failure ratio, the odds a request needs a third attempt are below one in a thousand, so the twenty-seven-retry allowance never gets used. The call amplification sleeps on a failure-free day; the failure ratio is what wakes it. A chain carrying excess retry headroom shows up in no metric under normal conditions.

Under an Outage, the Storm Grows With Layers

The second table shows the wakeup. While the dependency answers no call for one second, the in-window call rate grows with each added layer: 0.80 with no retries, 2.89 at one layer, 5.77 at two, 6.24 calls/round at three. That last figure is 6.4 times the usual rate, and it is aimed at a dependency that is already down.

The arithmetic upper bound must be kept apart from the observed multiplier. Three layers at three retries each allow 27; the observed figure is 6.4. Backoff makes the difference — retries spread out over time, so not all fit in the window. The upper bound is the risk the design carries; the observed multiplier is the share of that risk realized at these parameters. The bound is independent of the run: at depth four, it is 81.

The table also gives the caller-side gain, and it is real: dropped requests are 80 in the no-retry run, 0 at three layers. Because backoff spreads retries past the window, every request with enough retry headroom eventually succeeds.

Something also goes the wrong way between two rows. In the single-layer run, dropped requests are 94 — more than the no-retry run’s 80. The reason sits in the peak column: after the window, the storm climbs to 8 calls per round, above the usual 6-call capacity. Three retries do not survive this second wave once the outage ends. Insufficient retry headroom can be worse than none at all: it grows the load without rescuing the request.

Under Degradation, the Failure Ratio Sets the Multiplier, Not the Allowance

The third table runs the same configurations under a profile where capacity drops to one instead of zero, and the ranking reverses. Two layers and three layers give the same number: 5296 calls, 2.39 calls/round in the window. Even as the allowance climbs from 9 to 27, nothing changes.

The reason is the reverse of the second table’s. Because capacity is one, a portion of calls still succeeds; a request failing nine times in a row is rare. What sets the multiplier is not the allowance but the probability that a retry fails. The rule: the allowance sets an upper bound, the failure mode chooses how much of it is realized. A full outage uses the allowance to the end; partial degradation does not.

The multiplier still climbing to 2.5 times under degradation is not small: a dependency whose capacity has fallen to a sixth receives two and a half times the requests from the very system straining it. This is what stretches out the degradation.

The Retry Budget Is Spread Across the Chain

The last row gives the fix, and the fix is not trimming the retry count layer by layer; it was already shown that each layer chooses a reasonable count and the totals multiply. A retry budget caps the ratio of extra retries to primary calls, and that cap holds across the whole chain.

With the budget at 0.30, outage window rate falls from 6.24 to 2.88 calls/round, 3.0 times the usual rate; total calls drop from 4523 to 4065. Under degradation, window rate falls from 2.39 to 1.40, total calls from 5296 to 4288. The cost is plain: dropped requests rise from 0 to 103 under outage, 0 to 220 under degradation. The budget gives back a portion of the requests the caller rescued, so the dependency can breathe.

The sensitivity row shows how far the budget can be tightened: at 0.10, outage window rate falls to 1.54, and dropped requests fall from 103 to 90 — here, a tighter budget improves both sides at once, because a smaller storm also shrinks the second wave after the window. The budget’s right value depends less on how many requests the failure rescues than on how many the storm crushes.

Summary

• Each layer of the chain retries on its own, and allowances multiply: three layers at three retries give an upper bound of 27; at d layers and r retries the bound is r^d, 81 at d=4.
• The multiplier sleeps on a failure-free day: as the allowance climbs from 1 to 27 the call count does not change, the multiplier is 1.03, and dropped requests fall from 96 to 0. Excess retry headroom shows up in no metric under normal conditions.
• Under a full outage, the storm grows with each layer: window call rate goes 0.80 → 2.89 → 5.77 → 6.24 calls/round, 6.4 times the usual rate. The observed multiplier stays below the arithmetic bound (27), since backoff spreads retries over time.
• Insufficient retry headroom can be worse than none: dropped requests are 94 in the single-layer run against 80 with no retries; after the window, the storm pushes the peak above usual capacity, to 8 calls/round.
• Under partial degradation, the failure ratio sets the multiplier, not the allowance: allowances of 9 and 27 give the same result (5296 calls, 2.39 calls/round), since no request exhausts nine retries.
• A retry budget is placed across the whole chain: at 0.30, the outage storm falls from 6.24 to 2.88 calls/round, degradation from 2.39 to 1.40; the cost is 103 dropped requests under outage, 220 under degradation.

Next Step

Every measurement here held one number fixed: the wait between retries. But wait time and retry allowance cannot be chosen independently, since their product decides how long the caller waits. The last row gives that: twenty-seven retries, with backoff capped at thirty-two rounds, keep a request alive for a worst case of 793 rounds, 7.93 seconds — 39.6 times K01’s twenty-round read threshold. So the retry policy’s real limit is not the retry count but the time the caller has, and that time was never budgeted here. The next lesson takes it up and spreads it across the chain: how much share the end-to-end threshold leaves each step, how the remaining share splits among the calls beneath it, and how canceled work is counted when the budget runs out on a slowing dependency.