--- title: 'Retry and Backoff' source: 'https://academia.sh/en/courses/observability/retry-and-backoff' course: 'Observability and Reliability' language: en updated: '2026-08-23T07:00:28+00:00' license: 'CC BY-SA 4.0' --- # Retry and Backoff Measuring which layer a retry sits in: building the same rule separately into the client wrapper, the middle layer, and the gateway, whether the healthy call site gets retried too, the multiplier produced when three placements overlap, and the repeated side effect counted in deliveries. The previous lesson placed the duration limit and freed the caller, but left it holding an error. The closest answer is retrying: if the dependency dropped only briefly, a second call may succeed. This decision also has a place, more dangerous than the timeout — a timeout did not change the dependency's load, a retry multiplies it directly. Exponential backoff, jitter, the cap, and retry-budget arithmetic were built in **M14/K06 Application Architecture: Routing, State and Data**, **M16/K05**, and **M19/K05 Resilience and Reliability**; the pattern's effect on the system was measured in M19/K05. None of it repeats here. This lesson has one question: **which layer the retry sits in**, and what happens to calls reaching the dependency when that layer changes. The same rule can be placed in three spots. The **client wrapper** retries a single outbound call. The **middle layer** reruns the request's whole body. The **gateway** recalls the entire service and has not one line in the service's code. All three make the same promise; each repeats something different. ## Mechanism The notification dependency carries this lesson's measure. **RS5 — notification actually delivers; what's lost is the acknowledgment:** the delivery counter goes up on every call, but the first `failure` calls return 503. So what the retry repeats is not just the call but the **side effect**. Membership is a healthy dependency and always responds. ```js // services.mjs — 8202 notification (failing, counts deliveries), 8203 membership (healthy), // 8201 loan: carries both retry placements. Counters live at the 8202/counter endpoint. import { createServer, request } from "node:http"; const s = { loan: 0, membership: 0, notification: 0, delivered: 0, remaining: 0 }; // delivered: what notification actually sends createServer((req, res) => { // 8202 notification const q = new URL(req.url, "http://x").searchParams; if (req.url === "/counter") return res.end(JSON.stringify(s)); if (req.url.startsWith("/reset")) { for (const k of ["loan", "membership", "notification", "delivered"]) s[k] = 0; s.remaining = Number(q.get("failure")); return res.end("{}"); } s.notification += 1; s.delivered += 1; // delivery happens; the acknowledgment is lost for the first `remaining` calls if (s.remaining > 0) { s.remaining -= 1; res.statusCode = 503; } res.end("{}"); }).listen(8202); createServer((req, res) => { s.membership += 1; res.end("{}"); }).listen(8203); // healthy dependency const call = (port, path) => new Promise((resolve) => { const r = request({ port, path, agent: false }, (y) => { y.resume(); y.on("end", () => resolve(y.statusCode)); }); r.on("error", () => resolve(0)); r.end(); }); // >>> placement: client wrapper — retries only the call site it wraps const callWithRetry = async (port, path, tries) => { for (let n = 1; ; n += 1) { const k = await call(port, path); if (k < 500 || n >= tries) return k; } }; // <<< const handle = async (inner) => { // body of the loan request: two outbound call sites await callWithRetry(8203, "/member", 1); // healthy dependency, not retried return callWithRetry(8202, "/send", inner); }; createServer(async (req, res) => { // 8201 loan const q = new URL(req.url, "http://x").searchParams; s.loan += 1; // >>> placement: middle layer — reruns the request's whole body let k = 0; for (let n = 1; n <= Number(q.get("middle")); n += 1) { k = await handle(Number(q.get("inner"))); if (k < 500) break; } // <<< res.statusCode = k; res.end("{}"); }).listen(8201); ``` The third placement lives in the measurement script, because the gateway is outside the service. **RS6 — every layer's retry count is 3**; the comparison only means something while that number stays fixed. The `>>> placement:` markers exist to count code cost; the script reads them straight from the source. ```js // measure.mjs — which layer the retry sits in, the multiplier of overlapping retries, code cost. import { readFileSync } from "node:fs"; import { request } from "node:http"; const call = (port, path) => new Promise((resolve) => { const r = request({ port, path, agent: false }, (y) => { y.resume(); y.on("end", () => resolve(y.statusCode)); }); r.on("error", () => resolve(0)); r.end(); }); const read = (path) => new Promise((c) => request({ port: 8202, path }, (r) => { let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(g === "{}" ? null : JSON.parse(g))); }).end()); const print = (g, a, ...r) => console.log(String(a).padEnd(32) + r.map((x) => String(x).padStart(g)).join("")); // >>> placement: gateway — not one line lives in the service's code const gateway = async (outer, middle, inner) => { for (let n = 1; ; n += 1) { const k = await call(8201, `/loan?middle=${middle}&inner=${inner}`); if (k < 500 || n >= outer) return k; } }; // <<< const attempt = async (failure, outer, middle, inner) => { await read(`/reset?failure=${failure}`); const k = await gateway(outer, middle, inner); return { k, ...(await read("/counter")) }; }; console.log("transient failure: notification delivers on the first 2 calls but loses the acknowledgment; single user request"); print(20, "placement (outer/middle/inner)", "result", "loan calls", "membership calls", "notification calls", "delivered"); for (const [name, o, m, i] of [["none (1/1/1)", 1, 1, 1], ["wrapper (1/1/3)", 1, 1, 3], ["middle layer (1/3/1)", 1, 3, 1], ["gateway (3/1/1)", 3, 1, 1]]) { const r = await attempt(2, o, m, i); print(20, name, r.k, r.loan, r.membership, r.notification, r.delivered); } console.log("\npersistent failure: overlapping retries"); print(20, "outer / middle / inner", "multiplier", "notification calls", "delivered", "membership calls"); for (const [o, m, i] of [[1, 1, 1], [3, 1, 1], [3, 3, 1], [3, 3, 3], [5, 3, 3]]) { const r = await attempt(999, o, m, i); print(20, `${o} / ${m} / ${i}`, o * m * i, r.notification, r.delivered, r.membership); } const code = (file) => [...readFileSync(file, "utf8").matchAll(/\/\/ >>> placement: ([^\n—]+)—[^\n]*\n([\s\S]*?)\/\/ << [name.trim(), file, g.trim().split("\n").length]); const sites = [...readFileSync("services.mjs", "utf8").matchAll(/callWithRetry\(\d+, "[^"]+", ([^)]+)\)/g)].map((m) => m[1]); console.log("\ncode cost and coverage"); print(15, "placement", "file", "line"); for (const [name, d, n] of [...code("services.mjs"), ...code("measure.mjs")]) print(15, name, d, n); console.log(`the loan body has ${sites.length} outbound call sites; the wrapper retries ${sites.filter((x) => x !== "1").length} ` + `of them, the middle layer and the gateway retry all ${sites.length}.`); console.log("configuration surface: 3 retry counts, in 3 separate places; none sees the others' value."); process.exit(0); ``` ```bash node services.mjs > /dev/null 2>&1 & SP=$! sleep 1; node measure.mjs; kill $SP ``` ``` transient failure: notification delivers on the first 2 calls but loses the acknowledgment; single user request placement (outer/middle/inner) result loan calls membership calls notification calls delivered none (1/1/1) 503 1 1 1 1 wrapper (1/1/3) 200 1 1 3 3 middle layer (1/3/1) 200 1 3 3 3 gateway (3/1/1) 200 3 3 3 3 persistent failure: overlapping retries outer / middle / inner multiplier notification calls delivered membership calls 1 / 1 / 1 1 1 1 1 3 / 1 / 1 3 3 3 3 3 / 3 / 1 9 9 9 9 3 / 3 / 3 27 27 27 9 5 / 3 / 3 45 45 45 15 code cost and coverage placement file line client wrapper services.mjs 6 middle layer services.mjs 2 gateway measure.mjs 6 the loan body has 2 outbound call sites; the wrapper retries 1 of them, the middle layer and the gateway retry all 2. configuration surface: 3 retry counts, in 3 separate places; none sees the others' value. ``` ## How Three Placements Close the Same Failure The first table meets a transient failure with three placements, and the result column is the same in all three: a 503 turns into 200. The retry keeps its promise. The difference is in the other columns. Calls reaching the dependency stay at 3 in all three. This is not surprising; the failure lasts two calls and clears on the third. The retry's load on the dependency comes from the retry count, not from the placement. Calls reaching the healthy dependency, though, are 1, 3, and 3. This is where the split lies. The client wrapper retries only the call site it wraps; the membership call happens once, and that dependency — which has nothing to do with the failure — is never strained. The middle layer reruns the request body from the start, repeating the failure-free call site too. The gateway does the same and, on top of it, calls the loan service three times — shown in the loan-calls column as 1, 1, 3. The rule that follows is a selection criterion: **the outer placement cannot discriminate.** The gateway does not know which call failed, so it repeats the whole request; the further out you go, the more repeated work grows. The inner placement discriminates, but it only covers the call site that passes through it — the previous lesson's uncovered-call-site problem applies here exactly the same. The delivered column is 3 in all three placements. For a single user request, notification was sent three times. Changing the placement does not fix this; the side effect comes from the retry itself. Whether a call site is safe to repeat is a property of the call site, not the placement, and the fix belongs there too: that call site must keep its retry count at 1. ## Overlapping Retries The second table opens all three placements at once. Under a persistent failure, every layer spends its share to the end, and calls reaching the dependency track the multiplier column exactly: 1, 3, 9, 27, 45. A single user request turns into 27 calls on the dependency, and a single notification gets delivered 27 times. The real information in this table is not the multiplier itself, but that **no layer sees it.** Whoever wrote the gateway wrote 3 retries, whoever wrote the middle layer wrote 3, whoever wrote the wrapper wrote 3; none chose a wrong number. The configuration surface sits in three separate places and none can see the others' value. The last row shows how fast this grows: the outer layer going from three to five takes the total from 27 to 45 — **one number changing in one place adds eighteen calls to the dependency.** The membership-calls column is 9 on those same rows, not 27. The healthy dependency is only called as many times as the product of the two outer layers, because the innermost placement does not repeat it. Which factors of the multiplier land on which call site is a direct result of placement. ## Code Cost and Coverage The third table gives the size of each placement: client wrapper 6 lines, middle layer 2, gateway 6 — and the gateway's six lines are not in the service's code at all. This is what makes the outer placement attractive: every call site is covered without the service changing, even ones not yet written. It is also exactly the cost; coverage having no selectivity is not a flaw, it is that placement's definition. The inner placement is the opposite: of the two outbound call sites in the loan body, it retries one and deliberately leaves the other alone. Selectivity is only achieved by deciding call site by call site, and that decision is not made once — it is made again every time a new call site is added. ## Testability Placement also decides where the behavior can be pinned down in a test. ```js // test/retry.test.mjs — testability of placement: where the assertion is pinned. import { test } from "node:test"; import assert from "node:assert/strict"; import { request } from "node:http"; const call = (port, path) => new Promise((resolve) => { const r = request({ port, path, agent: false }, (y) => { y.resume(); y.on("end", () => resolve(y.statusCode)); }); r.on("error", () => resolve(0)); r.end(); }); const read = (path) => new Promise((c) => request({ port: 8202, path }, (r) => { let g = ""; r.on("data", (p) => (g += p)); r.on("end", () => c(g === "{}" ? null : JSON.parse(g))); }).end()); const run = async (failure, outer, middle, inner) => { await read(`/reset?failure=${failure}`); for (let n = 1; n <= outer; n += 1) if (await call(8201, `/loan?middle=${middle}&inner=${inner}`) < 500) break; return read("/counter"); }; test("wrapper placement: the assertion pins to a single call site", async () => { assert.equal((await run(2, 1, 1, 3)).notification, 3); assert.equal((await run(2, 1, 1, 3)).membership, 1); // healthy call site is not retried }); test("gateway placement: the same assertion changes with the inner setting", async () => { assert.equal((await run(999, 3, 1, 1)).notification, 3); assert.equal((await run(999, 3, 1, 3)).notification, 9); // gateway unchanged, result triples }); test("retry multiplies delivery", async () => { const r = await run(2, 1, 1, 3); assert.equal(r.delivered, 3); // single user request, three deliveries }); ``` ```bash node services.mjs > /dev/null 2>&1 & SP=$! sleep 1 node --test --test-force-exit --test-reporter=tap test/retry.test.mjs | grep -E "^(ok|not ok|# (tests|pass|fail) )" kill $SP ``` ``` ok 1 - wrapper placement: the assertion pins to a single call site ok 2 - gateway placement: the same assertion changes with the inner setting ok 3 - retry multiplies delivery # tests 3 # pass 3 # fail 0 ``` The first test shows the wrapper placement's testability: the assertion pins to a single call site, the number is local, and no other setting can change it. The second proves the gateway placement's problem with the test itself: the gateway's own setting is 3 in both runs, but the expected call count goes from 3 to 9, because a different number changed further in. An assertion about the outer placement's behavior depends on a setting it cannot see, and breaks when that setting changes. The third pins the side effect directly: one user request, three deliveries. ## Summary - Retry arithmetic and its system effect were measured in M14/K06, M16/K05, and M19/K05 Resilience and Reliability; measured here is which layer it sits in. - Under a transient failure all three placements save the request, and calls reaching the dependency stay at 3 in all three; the difference is in calls reaching the healthy dependency: 1 wrapper, 3 middle layer, 3 gateway. The gateway also calls the loan service 3 times instead of 1. - The outer placement cannot discriminate: not knowing which call failed, it repeats the whole request. The inner placement discriminates, but covers only the call site passing through it. - With all three placements open at once, calls reaching the dependency track the multiplier exactly: 1, 3, 9, 27, 45. No layer sees another's number; the outer one going from three to five adds eighteen calls at the dependency. The healthy dependency gets 9 calls, not 27. - The delivery counter is 3 in all three placements: the repeated side effect is a call-site problem, not a placement one, and the fix is keeping that call site's retry count at 1. - Code cost is 6 / 2 / 6 lines, and the gateway's six are not in the service's code. In tests the wrapper's assertion is local; the gateway's depends on a setting it cannot see and breaks when that setting changes. ## Next Step A retry closes a transient failure but worsens a persistent one: all 27 calls failed, none saved anything, and a dependency already down took twenty-seven times the load on top of it. What's missing is not a number but a **decision** — recognizing the dependency is down and not making the call at all. The next lesson places the circuit breaker in the code and measures: whether the breaker's state is kept per dependency or per call site, when the two placements open under the same failure, how many parameters make up the configuration surface, and what behavior a wrong threshold produces.