--- title: 'Retry Storm' source: 'https://academia.sh/en/courses/performance-and-monitoring/retry-storm' course: 'Performance Anti-Patterns and Monitoring' language: en updated: '2026-08-23T07:01:28+00:00' license: 'CC BY-SA 4.0' --- # Retry Storm Recognizing from the outside a feedback loop that amplifies a failure: separating the same call-rate increase's two causes by edge rate, distinct request identity, and success ratio; the call rate climbing from 4.00 to 25.00 during the storm while distinct identity stays at 4.00; the product of call rate and success ratio staying constant at the capacity limit; and throttling the retry allowance costing 180 dropped requests. All nine patterns rest on one assumption: that a request is made once. The load coming from the edge is set from outside, and the system only carries it. This assumption falls the moment something breaks. The symptom is this: the call rate reaching the dependency rose to six times its usual value. The retry decision, exponential backoff, jitter, and retry budget were established and measured in the Application Architecture: Routing, State and Data, Caching, Queues and Asynchronous Processing, and Resilience and Reliability courses; none of it is recomputed here. The schedule is an **input**. The question here is diagnosis: how does such a rate increase look from the outside, and which metric separates it from a genuine demand increase. ## Same Symptom, Two Causes **First cause: external demand genuinely grew.** The request rate coming from the edge has risen, and the call rate reaching the dependency rises with it. The system is healthy; it is only doing more work. **Second cause: external demand is fixed, and the chain is retrying.** The dependency has broken, calls are failing, and every layer of the chain retries on its own. The allowances multiply, and the load reaching the dependency now comes not from outside but from the system itself. This is called a **retry storm**. Both grow the same number at the dependency. An alert that looks only at call rate cannot tell them apart. ## The Chain The run below is a **model**: a round is an abstract step and no real cluster is set up. **KK15 — chain and schedule.** Three layers, three retries per layer, four rounds between retries, the dependency's usual capacity is 12 calls per round, external demand is 4 requests per round. Rationale: the schedule was decided in earlier courses and is taken here as input; the capacity headroom is three times external demand. The fault is capacity dropping to 1 call between rounds 101–160. Its sensitivity is given through the retry allowance. These assumptions are not added to K01's table. ```js // storm/chain.mjs — MODEL of a three-layer chain. A round is an abstract step. The backoff // schedule and retry budget are not computed here; they are taken as an INPUT decided in // earlier courses (fixed interval, fixed allowance per layer). export const KK15 = { layers: 3, allowance: 3, interval: 4, capacity: 12, externalDemand: 4 }; export function run({ round = 400, externalDemand = KK15.externalDemand, allowance = KK15.allowance, faultStart = 101, faultEnd = 160, faultCapacity = 1 }) { const { layers, interval, capacity } = KK15; const edge = [], calls = [], successful = [], dropped = [], identities = []; const active = []; // { id, counters: [k0,k1,k2], time } let nextId = 1, backlog = 0; for (let t = 1; t <= round; t += 1) { backlog += typeof externalDemand === "function" ? externalDemand(t) : externalDemand; let arrived = 0; while (backlog >= 1) { backlog -= 1; arrived += 1; active.push({ id: nextId++, counters: new Array(layers).fill(0), time: t }); } edge.push(arrived); const thisRound = active.filter((r) => r.time === t); let tokens = t >= faultStart && t <= faultEnd ? faultCapacity : capacity; let ok = 0, drop = 0; const seen = new Set(); for (const r of thisRound) { seen.add(r.id); // the dependency sees the call (even when rejected) if (tokens > 0) { tokens -= 1; ok += 1; active.splice(active.indexOf(r), 1); continue; } let k = layers - 1; // counter odometer: carries upward from the deepest layer while (k >= 0) { r.counters[k] += 1; if (r.counters[k] < allowance) break; r.counters[k] = 0; k -= 1; } if (k < 0) { drop += 1; active.splice(active.indexOf(r), 1); } // every allowance is exhausted else r.time = t + interval; } calls.push(thisRound.length); successful.push(ok); dropped.push(drop); identities.push(seen); } return { edge, calls, successful, dropped, identities }; } // Window summary: distinct identity is counted over the whole window, not within a round. export function window(r, start, end) { const sum = (a) => a.slice(start - 1, end).reduce((x, y) => x + y, 0); const n = end - start + 1; const distinct = new Set(); for (const s of r.identities.slice(start - 1, end)) for (const id of s) distinct.add(id); const c = sum(r.calls); return { edge: sum(r.edge) / n, calls: c / n, distinct: distinct.size / n, amplification: distinct.size === 0 ? 0 : c / distinct.size, success: c === 0 ? 0 : sum(r.successful) / c, dropped: sum(r.dropped), }; } ``` ```js // storm/measure.mjs — two causes of the same symptom and the metrics that tell the storm apart import { KK15, run, window } from "./chain.mjs"; const WIDTHS = [24, 11, 14, 16, 14, 8]; const write = (h) => console.log(h.map((x, i) => (i ? String(x).padStart(WIDTHS[i]) : String(x).padEnd(WIDTHS[i]))).join(" ")); const line = () => console.log(WIDTHS.map((n) => "-".repeat(n)).join(" ")); const row = (name, p) => write([name, p.edge.toFixed(2), p.calls.toFixed(2), p.distinct.toFixed(2), p.amplification.toFixed(2), p.success.toFixed(4)]); console.log(`chain ${KK15.layers} layers, ${KK15.allowance} retries per layer, interval ${KK15.interval} rounds,` + ` dependency's usual capacity ${KK15.capacity} calls/round, external demand ${KK15.externalDemand} requests/round`); console.log(`arithmetic ceiling: ${KK15.allowance}^${KK15.layers} = ${KK15.allowance ** KK15.layers} calls/request;` + ` window rounds 101-160\n`); // Cause 1: external demand genuinely grows, no fault. Cause 2: demand fixed, dependency breaks. const organic = run({ externalDemand: (t) => (t >= 101 && t <= 160 ? 8 : 4), faultCapacity: KK15.capacity }); const storm = run({}); const noRetry = run({ allowance: 1 }); write(["case", "edge/round", "calls/round", "distinct/round", "amplification", "success"]); line(); for (const [name, r] of [["demand grew", organic], ["storm", storm], ["no retry", noRetry]]) { row(`${name}: before`, window(r, 1, 100)); row(`${name}: during`, window(r, 101, 160)); row(`${name}: after`, window(r, 161, 400)); } // Recovery: after the fault ends, how many rounds until the call rate drops back to its usual level. const baseline = window(storm, 1, 100).calls; const recovery = (r) => { for (let t = 161; t <= 400; t += 1) if (r.calls[t - 1] <= baseline * 1.1) return t - 160; return -1; }; // Calls satisfied per round = call rate x success ratio; this product is constant at the capacity limit. const product = (r) => { const p = window(r, 101, 160); return (p.calls * p.success).toFixed(2); }; console.log(`\ncall rate x success ratio (calls satisfied/round): demand grew ${product(organic)},` + ` storm ${product(storm)}, no retry ${product(noRetry)}`); console.log(`usual call rate ${baseline.toFixed(2)}/round; fault ends at round 160`); console.log(`recovery ${recovery(storm)} round(s) (no-retry regime: ${recovery(noRetry)} round(s));` + ` the storm's peak call rate ${Math.max(...storm.calls)}/round = ${(Math.max(...storm.calls) / baseline).toFixed(2)}x` + ` the usual rate`); // The cost of the fix: with the allowance throttled, what is gained and lost in the same fault. console.log(`\nthe cost of throttling the allowance (same fault, rounds 101-400):`); write(["allowance", "calls/round", "amplification", "dropped requests", "recovery", "success"]); line(); for (const allowance of [3, 2, 1]) { const r = run({ allowance }); const p = window(r, 101, 160), base = window(r, 1, 100).calls; let g = -1; for (let t = 161; t <= 400; t += 1) if (r.calls[t - 1] <= base * 1.1) { g = t - 160; break; } write([`${allowance} per layer`, p.calls.toFixed(2), p.amplification.toFixed(2), window(r, 101, 400).dropped, g, p.success.toFixed(4)]); } ``` ``` chain 3 layers, 3 retries per layer, interval 4 rounds, dependency's usual capacity 12 calls/round, external demand 4 requests/round arithmetic ceiling: 3^3 = 27 calls/request; window rounds 101-160 case edge/round calls/round distinct/round amplification success ------------------------ ----------- -------------- ---------------- -------------- -------- demand grew: before 4.00 4.00 4.00 1.00 1.0000 demand grew: during 8.00 8.00 8.00 1.00 1.0000 demand grew: after 4.00 4.00 4.00 1.00 1.0000 storm: before 4.00 4.00 4.00 1.00 1.0000 storm: during 4.00 25.00 4.00 6.25 0.0400 storm: after 4.00 6.50 4.75 1.37 0.7308 no retry: before 4.00 4.00 4.00 1.00 1.0000 no retry: during 4.00 4.00 4.00 1.00 0.2500 no retry: after 4.00 4.00 4.00 1.00 1.0000 call rate x success ratio (calls satisfied/round): demand grew 8.00, storm 1.00, no retry 1.00 usual call rate 4.00/round; fault ends at round 160 recovery 25 round(s) (no-retry regime: 1 round(s)); the storm's peak call rate 49/round = 12.25x the usual rate the cost of throttling the allowance (same fault, rounds 101-400): allowance calls/round amplification dropped requests recovery success ------------------------ ----------- -------------- ---------------- -------------- -------- 3 per layer 25.00 6.25 0 25 0.0400 2 per layer 22.33 5.58 68 17 0.0448 1 per layer 4.00 1.00 180 1 0.2500 ``` All the numbers are **computed**, coming out of a deterministic run; the chain, schedule, and fault window are **assumption**. ## How the Storm Looks From the Outside The two scenarios share something indistinguishable at first glance: the call rate reaching the dependency grows in both. Three metrics make the distinction, and all three can be collected at the dependency. **Edge rate.** When demand grows, the edge rate rises from 4.00 to 8.00; in the storm it stays at 4.00. If a call-rate increase happens without a matching change at the edge, the system itself is generating the extra load. **Distinct request identity.** This is the sharpest of the metrics. When demand grows, the distinct identity the dependency sees rises from 4.00 to 8.00 — there are new requests. In the storm, distinct identity **stays fixed at 4.00** while calls climb to 25.00: the same requests arrive again and again. The ratio of call count to distinct identity is the observed **call amplification**, and here it is **6.25** against 1.00. The arithmetic ceiling is 27; the observed value is lower because retries spread across time. **Success ratio.** When demand grows, success stays at 1.0000. In the storm, as the call rate climbs sixfold, success drops from 1.0000 to **0.0400**. When rate and success move in opposite directions, what is growing is not work but the repetition of the same work. A fourth signal is in time. The fault ends at round 160, but the call rate only drops back to its usual level **25 rounds** later; in the regime without retries this takes 1 round. The storm outlives the fault. ## The Signature of a Capacity Limit One number also reveals the storm's nature. The product of call rate and success ratio is the number of calls the dependency actually satisfies. In the storm this product is 1.00; in the no-retry regime, also 1.00. Twenty-five times more calls **does not change** the number satisfied at all — the dependency's capacity is whatever it is, and the retries only shrink each other's share. In the demand-growth scenario the same product is 8.00 — the edge rate itself. This distinction tells when retrying is harmful. If the dependency is dropping individual calls from a transient error, retries grow the product and the work gets done. If the dependency is at its capacity limit, the product is fixed: every new retry crowds out another request's retry. The call amplification measured in the Resilience and Reliability course turns directly into a failure extender in this second case. ## What Grows in Exchange for the Fix The last table counts the cost of the fix. When the per-layer retry allowance drops from 3 to 2, the call rate falls from 25.00 to 22.33 and recovery from 25 rounds to 17 — and **68 requests are dropped**. When the allowance drops to 1, the storm disappears entirely (calls 4.00, amplification 1.00, recovery 1 round), but dropped requests rise to **180**. The trade-off is clear: throttling the storm means requests that go entirely unanswered during the fault window. With an allowance of 3, not a single request is dropped; its cost is the dependency seeing six times more calls and the fault lasting twenty-five rounds longer. This trade-off cannot be made without measurement — both ends of it are numbers. ## The Condition Under Which Retrying Is Correct Retrying is correct as long as call amplification stays below the dependency's capacity headroom. With the numbers here: the usual call rate is 4.00 and capacity is 12 calls/round, so the headroom is 3.00 times. The observed amplification of 6.25 exceeds this headroom, and this is exactly why the storm forms. Had amplification stayed below the headroom, the retries would have cost nothing. The second condition is the nature of the fault. If the product of call rate and success ratio stays constant regardless of call rate, the dependency is at its capacity limit, and increasing the retry allowance saves no additional requests. If the product grows with the call rate, the error is transient and retrying is cheap. ## Summary - The symptom is the call rate reaching the dependency growing; it has two causes — external demand grew, or the chain is retrying. A measurement that looks only at call rate cannot separate them. - Edge rate rises from 4.00 to 8.00 in the first cause and stays at 4.00 in the storm. The distinct request identity the dependency sees rises to 8.00 in the first cause and stays fixed at 4.00 in the storm. - The observed call amplification is 1.00 against 6.25; success ratio 1.0000 against 0.0400. When rate and success move in opposite directions, what is growing is not work but the repetition of the same work. - The storm outlives the fault: even though capacity recovers at round 160, the call rate does not drop back to usual until 25 rounds later; in the regime without retries this takes 1 round. - The product of call rate and success ratio is 1.00 both in the storm and in the no-retry regime; twenty-five times the calls does not change the number satisfied. This is the signature of a dependency at its capacity limit. - The cost of throttling the allowance is dropped requests: 0 at an allowance of 3, 68 at 2, 180 at 1; in exchange, recovery drops from 25 rounds to 1. ## Next Step This topic tied ten symptoms to ten causes, and in each one what made the distinction was the same kind of thing: a number. Busy database was separated by scanned records, busy front end by separating work by its source on the main thread, chatty I/O by the trade-off between call count and bytes carried in per-request touches, extraneous fetching by end-to-end byte count of unneeded fields, improper instantiation by expensive client setup work, monolithic persistence by the index share serving its own class, no caching by repeated computation, noisy neighbor by work share, synchronous I/O by dependency utilization, and retry storm by distinct request identity. All ten of these diagnoses depend on one condition: **that measurement must exist.** Yet which metric to collect was never designed in this topic. Is distinct request identity counted at the dependency; is work share kept per tenant; is the dependency's own utilization observed; is a computation logged together with its key. None of these exist on their own; each is a design decision, and each has a cost. The next topic builds that layer: which metric categories make a complete monitoring surface, what is worth measuring, which threshold triggers an alert, how load is tested, and how many nodes the measured number converts to.