--- title: 'Timeout and Retry Budget' source: 'https://academia.sh/en/courses/application-layer/timeout-and-retry-budget' course: 'The Application Layer and Service Interaction' language: en updated: '2026-08-23T14:25:22+00:00' license: 'CC BY-SA 4.0' --- # Timeout and Retry Budget Splitting the end-to-end latency threshold across the steps in the chain: computing how far the product of timeout and retry count inflates the budget in the worst case, deadline propagation turning a threshold overrun into an early failure, a short timeout absorbing transient slowness, and using a model to measure how far the internal request rate multiplies under widespread slowdown when no retry budget is set. After four lessons, a call can be made: the services are stateless, their addresses are found at runtime, the conversation style is chosen, and their contracts can evolve without breaking. One question is still open, and it returns to this topic's first measure. When the gateway calls two services and one of them does not answer, how long does it wait, how many times does it retry, and what does that waiting add to the end-to-end response. The retry decision, exponential backoff, jitter, and the circuit breaker were established in the Application Architecture course; the algorithms are not retold here. What this lesson takes up is a **budget**. The budget's source is K01: the tracking read's hard threshold is **200 milliseconds**, and the source of that threshold is the response time the recipient expects from the tracking page. The number of steps in the chain was also already measured — the third lesson counted two internal calls for one external request and showed this comes from the domain split, not the protocol. ## Splitting the Budget The chain is this: the gateway calls two services in parallel, and each service calls its own store. Depth 2, arms 2, four step calls in total. The gateway's own work also asks for a share; this is **this course's assumption**. **U3 — the gateway's own share is 20 ms.** Rationale: validation, gate checks, and merging the two responses were moved to the edge in K02, and that work takes up space in the budget. The sensitivity is computed below; it is not added to K01's table. ```js // chain/budget.mjs — splitting the end-to-end budget across the chain; deterministic arithmetic const END_TO_END = 200; // K01 Non-Functional Requirements: read threshold (hard) 200 ms const GATEWAY_SHARE = 20; // assumption U3: the gateway's own share of the budget (ms) const STEP_MEDIAN = 8; // model parameter: a step call's usual duration (ms) const remaining = END_TO_END - GATEWAY_SHARE; console.log(`end-to-end ${END_TO_END} ms, gateway's share ${GATEWAY_SHARE} ms, remaining for steps ${remaining} ms\n`); console.log(`${"depth".padStart(9)}${"retries".padStart(8)}${"no-budget worst(ms)".padStart(22)}` + `${"budget ratio".padStart(14)}${"allocated timeout(ms)".padStart(31)}${"ratio to median".padStart(16)}`); for (const depth of [1, 2, 3, 4]) { for (const tries of [1, 3]) { const noBudget = depth * tries * END_TO_END + GATEWAY_SHARE; const allocated = remaining / (depth * tries); console.log(`${String(depth).padStart(9)}${String(tries).padStart(8)}${String(noBudget).padStart(22)}` + `${`x${(noBudget / END_TO_END).toFixed(1)}`.padStart(14)}${allocated.toFixed(1).padStart(31)}` + `${`x${(allocated / STEP_MEDIAN).toFixed(2)}`.padStart(16)}`); } } const maxDepth = Math.floor(remaining / (3 * STEP_MEDIAN)); console.log(`\nat 3 retries and a step median of ${STEP_MEDIAN} ms, the timeout can go no more than ` + `${maxDepth} steps deep before falling below the median`); console.log(`sensitivity: if the gateway's share went ${GATEWAY_SHARE} -> ${GATEWAY_SHARE * 2} ms, at depth 2, retries 3, ` + `the timeout would go ${(remaining / 6).toFixed(1)} -> ${((END_TO_END - 2 * GATEWAY_SHARE) / 6).toFixed(1)} ms`); ``` ``` end-to-end 200 ms, gateway's share 20 ms, remaining for steps 180 ms depth retries no-budget worst(ms) budget ratio allocated timeout(ms) ratio to median 1 1 220 x1.1 180.0 x22.50 1 3 620 x3.1 60.0 x7.50 2 1 420 x2.1 90.0 x11.25 2 3 1220 x6.1 30.0 x3.75 3 1 620 x3.1 60.0 x7.50 3 3 1820 x9.1 20.0 x2.50 4 1 820 x4.1 45.0 x5.63 4 3 2420 x12.1 15.0 x1.88 at 3 retries and a step median of 8 ms, the timeout can go no more than 7 steps deep before falling below the median sensitivity: if the gateway's share went 20 -> 40 ms, at depth 2, retries 3, the timeout would go 30.0 -> 26.7 ms ``` These numbers are in the **computed value** class. The third column shows the most common mistake: if every step is given the end-to-end threshold itself as its timeout, then at depth 2 and 3 retries, the worst case is 1220 milliseconds — 6.1x the budget. Setting the timeout equal to the threshold does not protect the budget; **every step becomes able to spend the entire budget on its own.** The fifth column gives the correct split, and it comes with a limit. The timeout allotted to a step is divided by the product of depth and retry count: 30 milliseconds at depth 2 and 3 retries, 15 milliseconds at depth 4. The last column says why this is a limit — as the timeout approaches the step's usual duration, ordinary requests start timing out too. At a step median of eight milliseconds and three retries, the budget can reach a depth of at most seven steps. This is the bond between service splitting and the latency threshold: **every step added to the chain divides the budget, and it cannot be divided forever.** ## Model The run below is a **model**: it uses a logical clock, step durations are read from a cycle, and the slowdown is triggered by a parameter. It runs in two regimes — in the normal regime most steps are fast and a step fails to answer once in a while; in the widespread slowdown regime most steps fail to answer. No real service fleet or network is set up. ```js // chain/run.mjs — the chain's MODEL: logical clock, step durations as a parameter, two regimes. // Chain: gateway -> (delivery-ops, billing) parallel -> each calls its own store. Depth 2, arms 2. const N = 40, THRESHOLD = 200, GATEWAY_SHARE = 20, DEPTH = 2, ARM = 2, STEP = DEPTH * ARM; const REGIME = { // model parameter: duration of a step call (ms) normal: [6, 7, 6, 8, 6, 7, 6, 45, 6, 7, 6, 8, 6, 7, 6, 1000], "widespread slowdown": [6, 1000, 1000, 1000, 6, 1000, 1000, 1000], }; function run({ timeout, tries, deadline, retryBudget }, regime) { let s = 0, primary = 0, extra = 0; const count = { withinThreshold: 0, outsideThreshold: 0, dropped: 0 }, endToEnd = []; const duration = () => REGIME[regime][s++ % REGIME[regime].length]; const allowed = () => retryBudget === null || extra < retryBudget * primary; const step = (remaining) => { // one step: timeout and retry let elapsed = 0; for (let i = 0; i < tries; i += 1) { const stepTimeout = deadline ? Math.min(timeout, remaining - elapsed) : timeout; if (stepTimeout <= 0) return { ok: false, elapsed }; if (i === 0) primary += 1; else extra += 1; const d = duration(); if (d <= stepTimeout) return { ok: true, elapsed: elapsed + d }; elapsed += stepTimeout; if (i + 1 < tries && allowed() === false) return { ok: false, elapsed }; } return { ok: false, elapsed }; }; const arm = () => { // one arm: DEPTH consecutive steps let elapsed = 0, ok = true; for (let i = 0; i < DEPTH && ok; i += 1) { const r = step(THRESHOLD - GATEWAY_SHARE - elapsed); elapsed += r.elapsed; ok = r.ok; } return { ok, elapsed }; }; for (let i = 0; i < N; i += 1) { const arms = Array.from({ length: ARM }, arm); // parallel: duration is the arms' maximum const elapsed = GATEWAY_SHARE + Math.max(...arms.map((a) => a.elapsed)); endToEnd.push(elapsed); if (arms.every((a) => a.ok) === false) count.dropped += 1; else if (elapsed > THRESHOLD) count.outsideThreshold += 1; else count.withinThreshold += 1; } const sorted = [...endToEnd].sort((a, b) => a - b); return { ...count, median: sorted[N >> 1], worst: sorted[N - 1], multiplier: (primary + extra) / (N * STEP) }; } const CONFIGURATION = [ ["no budget (200/3)", { timeout: 200, tries: 3, deadline: false, retryBudget: null }], ["+ deadline propagation", { timeout: 200, tries: 3, deadline: true, retryBudget: null }], ["+ allocated (30/3)", { timeout: 30, tries: 3, deadline: true, retryBudget: null }], ["+ retry budget 0.30", { timeout: 30, tries: 3, deadline: true, retryBudget: 0.3 }], ]; const PEAK_READ = 416.67, INTERNAL_BASE = PEAK_READ * STEP; // K01: peak read; lesson 03: step count console.log(`${N} external requests, depth ${DEPTH}, arms ${ARM}, steps ${STEP}; ` + `end-to-end threshold ${THRESHOLD} ms (K01), gateway share ${GATEWAY_SHARE} ms (U3)`); console.log(`internal req/s base = ${PEAK_READ} x ${STEP} = ${INTERNAL_BASE.toFixed(2)}`); for (const regime of Object.keys(REGIME)) { console.log(`\n-- ${regime}: ${REGIME[regime].filter((v, i, a) => a.indexOf(v) === i).join("/")} ms cycle --`); console.log(`${"configuration".padEnd(24)}${"within threshold".padStart(18)}${"outside threshold".padStart(19)}` + `${"dropped".padStart(9)}${"median".padStart(9)}${"worst".padStart(9)}${"multiplier".padStart(12)}` + `${"internal req/s".padStart(16)}`); for (const [name, y] of CONFIGURATION) { const r = run(y, regime); console.log(`${name.padEnd(24)}${String(r.withinThreshold).padStart(18)}${String(r.outsideThreshold).padStart(19)}` + `${String(r.dropped).padStart(9)}${String(r.median).padStart(9)}${String(r.worst).padStart(9)}` + `${`x${r.multiplier.toFixed(2)}`.padStart(12)}${(INTERNAL_BASE * r.multiplier).toFixed(2).padStart(16)}`); } } console.log(`\nunlimited-retry upper bound: 3 tries per step -> ${(INTERNAL_BASE * 3).toFixed(2)} req/s`); ``` ``` 40 external requests, depth 2, arms 2, steps 4; end-to-end threshold 200 ms (K01), gateway share 20 ms (U3) internal req/s base = 416.67 x 4 = 1666.68 -- normal: 6/7/8/45/1000 ms cycle -- configuration within threshold outside threshold dropped median worst multiplier internal req/s no budget (200/3) 30 10 0 71 233 x1.06 1770.85 + deadline propagation 30 0 10 71 200 x1.00 1666.68 + allocated (30/3) 40 0 0 62 63 x1.14 1895.85 + retry budget 0.30 40 0 0 62 63 x1.14 1895.85 -- widespread slowdown: 6/1000 ms cycle -- configuration within threshold outside threshold dropped median worst multiplier internal req/s no budget (200/3) 0 0 40 626 626 x2.00 3333.36 + deadline propagation 0 0 40 200 200 x0.67 1114.59 + allocated (30/3) 0 0 40 116 116 x2.00 3333.36 + retry budget 0.30 0 0 40 86 86 x0.97 1614.60 unlimited-retry upper bound: 3 tries per step -> 5000.04 req/s ``` ## Reading the Four Rows **Deadline propagation turns a threshold overrun into an early failure.** In the normal regime, the no-budget arrangement answers thirty of forty requests within the threshold, ten outside it — worst case 233 milliseconds. Once the caller passes the remaining budget along with the call, the worst case is cut off at exactly 200 milliseconds and those ten requests stop being outside the threshold and become dropped requests. The trade-off is explicit: a choice is made between a late answer and an on-time failure. The user who received an answer at 233 milliseconds was already not getting a threshold-meeting response; the work spent producing it was also unrewarded. **A short timeout absorbs transient slowness.** In the allocated arrangement, when the timeout falls to 30 milliseconds, all forty of forty requests answer within the threshold and the worst case is 63 milliseconds. The reason is the model's assumption: the slowness is transient, and the same step's second call returns fast. A long timeout carries this transient slowness through to the end-to-end response; a short timeout cuts it off and retries. The cost shows up as the call multiplier rising from 1.00 to 1.14: the internal request rate climbs from 1666.68 to 1895.85 req/s. **The retry budget does not engage in the normal regime.** The fourth row is identical to the third. The observed multiplier is 1.14 and the budget is 1.30, so the limit never comes into play. A budget being invisible under normal conditions is the sign that it is sized correctly. ## What Changes Under Widespread Slowdown The second table runs the same configurations in a regime where most steps do not answer, and the ranking reverses. In the no-budget arrangement, all forty of forty requests drop, and the caller waits 626 milliseconds to learn this. Deadline propagation cuts this off at 200 milliseconds; the multiplier falls to 0.67, because there are also primary calls that never get made once the budget runs out. The internal request rate falls from 3333.36 to 1114.59 req/s. The third row is the lesson's most important number. When the timeout is cut to 30 milliseconds, the end-to-end duration falls to 116 milliseconds — but the multiplier climbs back to 2.00 and the internal request rate becomes 3333.36 req/s. A short timeout fits more retries into the same budget; **a service that is already slow receives twice the requests from the system that is slowing it down.** A decision that was good in the normal regime is harmful in the failure regime. The fourth row cuts this off. Once the retry budget limits extra retries to thirty percent of primary calls, the multiplier falls to 0.97 and the internal request rate to 1614.60 req/s: a below-baseline load instead of a doubling. The end-to-end duration also falls, from 116 to 86 milliseconds, because wasted retries are not made. The unlimited-retry upper bound is the comparison point — if every step is tried three times, 5000.04 req/s, nearly ten times K01's peak rate at the edge as internal load. All of this holds only for calls that can be retried. A state-changing call being retryable depends on processing the same call twice giving the same result as processing it once; this property is called **idempotent**, the same concept as the idempotency property of HTTP methods. A call that is not idempotent can only be retried if the client generates an **idempotency key** that is written into the contract; both were established in the Web API Design course. ## Summary - The end-to-end threshold is a budget and it is split across the chain; if every step is given the full threshold as its timeout, the worst case at depth 2 and 3 retries climbs to 6.1x the budget. - The allocated timeout is divided by the product of depth and retry count (30 ms at depth 2, retries 3) and hits a limit as it approaches the step's usual duration: at an 8 ms median and 3 retries, the budget can reach a depth of at most 7 steps. - Deadline propagation cuts the worst case off at the threshold: in the normal regime, 233 ms and 10 outside-threshold responses turn into 200 ms and 10 dropped requests — an on-time failure instead of a late answer. - A short timeout absorbs transient slowness (40/40 within threshold, worst case 63 ms) at the cost of the call amplification rising from 1.00 to 1.14 and the internal request rate climbing from 1666.68 to 1895.85 req/s. - Under widespread slowdown the same decision reverses: a short timeout drives the multiplier to 2.00 and the internal request rate to 3333.36 req/s; a retry budget of 0.30 brings this to 0.97 and 1614.60 req/s and cuts the end-to-end duration from 116 to 86 ms. - Retrying is free only for idempotent calls; a call that is not idempotent can be retried only if the contract carries an idempotency key. ## Next Step This topic designed **synchronous** calling: the caller waits, the budget does its work, and in the end either an answer arrives or a failure does. The budget calculation made a limit visible — as the chain lengthens, either the budget splits into pieces and every step narrows, or the end-to-end duration grows. Both options sit inside the same assumption: the caller is waiting for the result of the work. For some jobs there is a third path, and that path removes the assumption — the caller never waits at all. A state event arriving from the carrier being written into the store does not have to finish in order to answer the request. The next topic measures what dropping a request into a queue changes, and it asks this as a capacity question: on the synchronous path, capacity has to be chosen for the instantaneous peak; once a queue is added, it can be brought closer to the average. What that gain costs in pending work and waiting time, and which of K01's numbers change, is computed there.