--- title: 'Timeout Budgets' source: 'https://academia.sh/en/courses/resilience-patterns/timeout-budgets' course: 'Resilience and Reliability' language: en updated: '2026-08-23T07:01:31+00:00' license: 'CC BY-SA 4.0' --- # Timeout Budgets How the budget runs out at a slowing dependency: splitting the arm's remaining share across consecutive sub-calls with a greedy, equal, or weighted policy, the greedy split making the healthy step appear at fault, counting the caller's held slot separately from the dependency's abandoned work, and the tight split writing a gain only together with a cancellation signal. The retry policy is settled: try count, wait between tries, jitter width. One number stays open and decides more than the policy itself — how long the caller waits before giving up. Without it, the backoff grows toward open space, not a limit. The Application Layer and Service Interaction course established this duration as a **budget**: the end-to-end threshold splits across the chain's steps, the remaining time passes with the call, and the retry count multiplies it. That computation is an input here, not retold. This lesson goes one step further: when a chain step **slows down**, where does the budget run out, how does the share split across sub-calls, and how much work does the dependency still do after the caller gives up. ## Failure Mode The chain comes from this course's shared structure. The tracking query enters at the gateway and calls two arms in parallel. The delivery-operations arm reads a single record from its own store; the billing arm reads **two records** — first the tariff, then the shipment's size — and computes the fee from both. The Application Layer course counted four step calls per external request; once the billing arm's store step splits into two consecutive reads, the count climbs to five. **AY10 — the tariff store slows down.** It does not go down, answer incorrectly, or suffer a network partition; only its response time climbs to tens of times its usual six-millisecond median. Duration **12 minutes**, frequency **once a month**. Rationale: the slowdown starts with a maintenance window or a query-plan change and lasts until noticed and rolled back; twelve minutes sits a little above K01's ten-minute recovery assumption. Its sensitivity is below, not added to K01's table. Slowdown is harder than being down. A down dependency rejects the connection immediately, and the caller learns this in under a millisecond. One that is slowing down rejects nothing, it just makes the caller wait — and with no limit set, that wait is paid for with the caller's resources. The budget is the only mechanism deciding how much of it is accepted. ## Splitting the Remaining Share The arm's share is fixed: K01's 200-millisecond read threshold minus the Application Layer course's U3 assumption of a 20-millisecond gateway share leaves 180 milliseconds, which the billing arm must distribute across two consecutive reads. Three policies exist. The **greedy** split gives the entire remainder to each call: the first read can wait up to 180 milliseconds, the second gets whatever is left. The **equal** split halves the share. The **weighted** split allocates it by the typical durations' ratio — tariff 6 milliseconds, size 12, so one to two. All three look nearly identical on a failure-free day, since both reads stay well under their share. The difference appears the moment one read exceeds it. ```js // budget/chain.mjs — MODEL of the tracking query chain. Logical clock: step durations are // read from a cycle, real time is not measured; no service, network, or store is set up. export const K01 = { threshold: 200, readPeak: 416.67, edgePeak: 513.89, responseBytes: 480 }; export const GATEWAY_SHARE = 20; // K03 service-design/05, assumption U3 export const TYPICAL = { delivery: 8, tariff: 6, size: 12 }; // model parameter (ms) const D = (body, tail) => [...body, ...body, tail]; // a long tail once in sixteen export const REGIME = { // sub-call duration (ms), sixteen-step cycle "failure-free": { delivery: D([8, 9, 8, 10, 8, 9, 8, 10], 40), tariff: D([6, 7, 6, 8, 6, 7, 6, 8], 80), size: D([12, 13, 12, 14, 12, 13, 12, 14], 100), }, "tariff slowdown": { // AY10: only the tariff store slows down delivery: D([8, 9, 8, 10, 8, 9, 8, 10], 40), tariff: D([150, 400, 220, 400, 180, 300, 260, 400], 400), size: D([12, 13, 12, 14, 12, 13, 12, 14], 100), }, }; // Splitting the remaining share for the arm across two consecutive sub-calls. export function shares(policy, remaining) { if (policy === "greedy") return [remaining, remaining]; if (policy === "equal") return [remaining / 2, remaining / 2]; const t = TYPICAL.tariff + TYPICAL.size; // weighted: ratio of the typical durations return [(remaining * TYPICAL.tariff) / t, (remaining * TYPICAL.size) / t]; } export function run({ regime, policy, cancel, n = 48 }) { const R = K01.threshold - GATEWAY_SHARE; // remaining share for the arm const s = { answered: 0, dropped: 0, wastedWork: 0, totalDuration: 0, failed: { delivery: 0, tariff: 0, size: 0 }, endToEnd: [] }; const cursor = { delivery: 0, tariff: 0, size: 0 }; const duration = (name) => REGIME[regime][name][cursor[name]++ % REGIME[regime][name].length]; const call = (name, limit) => { // one sub-call, timeout `limit` if (limit <= 0) return { ok: false, elapsed: 0 }; const d = duration(name); if (d <= limit) return { ok: true, elapsed: d }; if (cancel === false) s.wastedWork += d - limit; // caller gave up, dependency keeps working return { ok: false, elapsed: limit }; }; for (let i = 0; i < n; i += 1) { const a = call("delivery", R); // delivery-ops arm: one store call const p = shares(policy, R); // billing arm: two consecutive records const b1 = call("tariff", Math.min(p[0], R)); const b2 = call("size", Math.min(p[1], R - b1.elapsed)); for (const [name, r] of [["delivery", a], ["tariff", b1], ["size", b2]]) { if (r.ok === false) s.failed[name] += 1; } const elapsed = GATEWAY_SHARE + Math.max(a.elapsed, b1.elapsed + b2.elapsed); s.endToEnd.push(elapsed); s.totalDuration += elapsed; if (a.ok && b1.ok && b2.ok && elapsed <= K01.threshold) s.answered += 1; else s.dropped += 1; } const sorted = [...s.endToEnd].sort((x, y) => x - y); return { ...s, median: sorted[n >> 1], worst: sorted[n - 1], average: s.totalDuration / n, n }; } ``` Both reads are attempted on every request; the billing arm decides only after adding both outcomes. This carries the measurement: if the first read eats the budget, the second read experiences this as **its own failure**. ```js // budget/run.mjs — comparing the three split policies over two days import { K01, GATEWAY_SHARE, TYPICAL, shares, run } from "./chain.mjs"; const R = K01.threshold - GATEWAY_SHARE; const POLICY = ["greedy", "equal", "weighted"]; const CALLS = 5; // sub-calls per external request: 2 arms + 1 + 2 records const DEADLINE_BYTES = 8; // field carrying the remaining time const slot = (ms) => (K01.readPeak * ms) / 1000; // L = lambda x W console.log(`end-to-end threshold ${K01.threshold} ms (K01), gateway share ${GATEWAY_SHARE} ms (K03/U3), remaining for the arm ${R} ms`); console.log(`typical durations: delivery ${TYPICAL.delivery}, tariff ${TYPICAL.tariff}, size ${TYPICAL.size} ms`); console.log(`${"policy".padEnd(11)}${"tariff share(ms)".padStart(17)}${"size share(ms)".padStart(15)}`); for (const p of POLICY) { const [a, b] = shares(p, R); console.log(`${p.padEnd(11)}${a.toFixed(1).padStart(17)}${b.toFixed(1).padStart(15)}`); } for (const regime of ["failure-free", "tariff slowdown"]) { console.log(`\n-- ${regime} (48 external requests, no cancellation signal) --`); console.log(`${"policy".padEnd(11)}${"answered".padStart(9)}${"dropped".padStart(8)}${"blame:tariff".padStart(13)}` + `${"blame:size".padStart(11)}${"median".padStart(8)}${"worst".padStart(8)}${"avg.".padStart(7)}` + `${"held slots".padStart(13)}${"wasted work(ms)".padStart(16)}`); for (const p of POLICY) { const r = run({ regime, policy: p, cancel: false }); console.log(`${p.padEnd(11)}${String(r.answered).padStart(9)}${String(r.dropped).padStart(8)}` + `${String(r.failed.tariff).padStart(13)}${String(r.failed.size).padStart(11)}` + `${String(r.median).padStart(8)}${String(r.worst).padStart(8)}${r.average.toFixed(1).padStart(7)}` + `${slot(r.average).toFixed(1).padStart(13)}${String(r.wastedWork).padStart(16)}`); } } console.log(`\n-- tariff slowdown: slots the system keeps busy (peak read ${K01.readPeak}/s) --`); console.log(`${"policy".padEnd(11)}${"caller".padStart(9)}${"abandoned".padStart(12)}` + `${"total(no cancel)".padStart(17)}${"total(cancel)".padStart(16)}`); for (const p of POLICY) { const r = run({ regime: "tariff slowdown", policy: p, cancel: false }); const c = slot(r.average), t = slot(r.wastedWork / r.n); console.log(`${p.padEnd(11)}${c.toFixed(1).padStart(9)}${t.toFixed(1).padStart(12)}` + `${(c + t).toFixed(1).padStart(17)}${c.toFixed(1).padStart(16)}`); } const bytes = K01.edgePeak * CALLS * DEADLINE_BYTES; const a = run({ regime: "failure-free", policy: "greedy", cancel: false }); console.log(`\n-- failure-free day's cost (computed value) --`); console.log(`deadline field ${DEADLINE_BYTES} B x ${CALLS} sub-calls = ${CALLS * DEADLINE_BYTES} B/request,` + ` ${((CALLS * DEADLINE_BYTES) / K01.responseBytes * 100).toFixed(1)}% of the tracking response (${K01.responseBytes} B)`); console.log(`at ${K01.edgePeak} req/s peak edge -> ${bytes.toFixed(2)} B/s = ${(bytes / 1024).toFixed(2)} KiB/s`); for (const p of ["equal", "weighted"]) { const r = run({ regime: "failure-free", policy: p, cancel: false }); const diff = r.dropped - a.dropped; console.log(`${p.padEnd(11)}: ${diff}/${r.n} more dropped requests than greedy` + ` = ${((diff / r.n) * 100).toFixed(2)}% -> ${((K01.readPeak * diff) / r.n).toFixed(2)} req/s at peak read`); } console.log(`\n-- AY10's sensitivity: how long the tariff store stays slow --`); console.log(`${"duration(min)".padStart(14)}${"a month".padStart(8)}${"minutes/mo".padStart(11)}${"of failure share(28.2 min)".padStart(27)}`); for (const [min, times] of [[12, 1], [12, 2], [24, 1]]) { console.log(`${String(min).padStart(14)}${String(times).padStart(8)}${String(min * times).padStart(11)}` + `${`${(((min * times) / 28.2) * 100).toFixed(1)}%`.padStart(27)}`); } ``` ``` end-to-end threshold 200 ms (K01), gateway share 20 ms (K03/U3), remaining for the arm 180 ms typical durations: delivery 8, tariff 6, size 12 ms policy tariff share(ms) size share(ms) greedy 180.0 180.0 equal 90.0 90.0 weighted 60.0 120.0 -- failure-free (48 external requests, no cancellation signal) -- policy answered dropped blame:tariff blame:size median worst avg. held slots wasted work(ms) greedy 48 0 0 0 40 200 46.2 19.2 0 equal 46 2 0 2 40 190 45.8 19.1 20 weighted 46 2 2 0 40 180 45.3 18.9 40 -- tariff slowdown (48 external requests, no cancellation signal) -- policy answered dropped blame:tariff blame:size median worst avg. held slots wasted work(ms) greedy 6 42 36 42 200 200 197.8 82.4 5540 equal 0 48 48 2 123 200 126.0 52.5 9700 weighted 0 48 48 0 93 180 96.4 40.2 11120 -- tariff slowdown: slots the system keeps busy (peak read 416.67/s) -- policy caller abandoned total(no cancel) total(cancel) greedy 82.4 48.1 130.5 82.4 equal 52.5 84.2 136.7 52.5 weighted 40.2 96.5 136.7 40.2 -- failure-free day's cost (computed value) -- deadline field 8 B x 5 sub-calls = 40 B/request, 8.3% of the tracking response (480 B) at 513.89 req/s peak edge -> 20555.60 B/s = 20.07 KiB/s equal : 2/48 more dropped requests than greedy = 4.17% -> 17.36 req/s at peak read weighted : 2/48 more dropped requests than greedy = 4.17% -> 17.36 req/s at peak read -- AY10's sensitivity: how long the tariff store stays slow -- duration(min) a month minutes/mo of failure share(28.2 min) 12 1 12 42.6% 12 2 24 85.1% 24 1 24 85.1% ``` All these numbers belong to the **computed value** class: they are counted over a deterministic sequence of durations. The time unit is a logical clock, not a measured latency; the run gives the same output independent of the machine. ## Failure-Free Day's Cost The budget carries two costs even on a failure-free day. The first is the field itself: passing the remaining time with the call adds an eight-byte field to every sub-call, 40 bytes per external request across five — 8.3 percent of K01's 480-byte tracking response, and 20.07 KiB/s of internal traffic at the peak edge rate. Small but not zero, and it grows linearly as the chain deepens. The second is costlier: **a tight split cuts off healthy requests.** In the failure-free regime greedy answers forty-eight of forty-eight requests; equal and weighted answer forty-six, dropping two each. These are false positives on a day with no failure at all — the model's long tail, arriving once in sixteen, exceeds the step's share: 4.17 percent, 17.36 requests/s at peak read. Which step gets cut off depends on the policy: under equal it is size (share 90, tail 100); under weighted, tariff (share 60, tail 80). Choosing the share by the step's tail lowers this cost but does not zero it. Splitting the budget is a decision to cut a tail, and the cut tail is always made of real requests. ## What Changes on a Failing Day The second table runs under AY10 and the three policies produce three distinct behaviors. **Greedy rescues six requests.** The tariff returns in 150 or 180 milliseconds on some rounds; with a 180-millisecond share these succeed, leaving enough for the size read. Under equal and weighted, the same reads are cut off at 90 and 60 milliseconds, and forty-eight of forty-eight requests drop — the cost of a tight split continues on a failing day too. **The trade-off shows up in end-to-end duration.** Under greedy the median is 200 milliseconds: every dropped request eats the whole budget. Under equal it is 123, under weighted 93 — the caller learns the same failure in under half the time. Its resource counterpart is `held slots`: by queueing theory's $L = \lambda W$ relation, at K01's peak read rate of 416.67 requests/s, greedy keeps 82.4 concurrent caller slots busy, weighted 40.2. The relation is not this course's finding; it comes from the background jobs topic and is used here as a converter. **The blame columns hide the failure mode.** Under greedy, the tariff fails thirty-six times, the size **forty-two**. The size's regime never changed; all forty-two failures come from the tariff eating the budget. Under weighted, the size fails zero times. The difference is not a counting difference but a diagnostic one: greedy hides the slowing step behind the healthy one, and the name showing up most in the timeout log is not where the failure happened. This is **spread** through the budget. The steps look isolated — separate stores, separate records — but they share a single resource, time. Fixing the share is the name for dividing up that shared resource. ## Abandoned Work The last column flips the table read so far. When the caller gives up, the tariff store does not stop; it finishes the read it started. This is work nobody waits for, and the model counts it as `wasted work`: 5540 milliseconds under greedy, 11120 under weighted. A tight split **doubles** abandoned work — giving up earlier leaves a longer remainder ahead of the dependency. The third table settles it by adding both sides. With no cancellation signal, total busy slots are 130.5 under greedy, 136.7 under equal and weighted — nearly the same: splitting only moves the load from caller to dependency, nothing else. A tight split's gain shows up only with a **cancellation signal** — the caller tells the dependency it has given up, and the read stops. Then abandoned work zeroes out and total slots become 82.4 against 40.2. This is the deciding sentence: **deadline propagation is not complete without a cancellation signal.** Passing the remaining time with the call says when the caller will give up; the signal says that it has. Without the first, the budget cannot be split; without the second, splitting only relocates cost. The last table is AY10's sensitivity. A single twelve-minute slowdown spends 42.6 percent of K01's monthly 28.2-minute failure share. Twice a month, or once for twenty-four minutes, the share climbs to 85.1 percent, leaving 4.2 minutes for the rest of the month. Splitting the budget does not bring these minutes back — requests still drop — but it fixes how many resources a dropped request occupies and confirms which step the failure shows up in. ## Summary - The arm's remaining share (200 ms threshold minus 20 ms gateway share = 180 ms) splits three ways: greedy 180/180, equal 90/90, weighted 60/120. - The failure-free day's cost has two items: a deadline field adding 40 bytes per external request (8.3 percent of the response, 20.07 KiB/s at peak edge), and a tight split cutting off valid requests — 2 in 48, 4.17 percent, 17.36 requests/s at peak read. - Under AY10, greedy rescues six requests but keeps the median at 200 ms; weighted rescues none but brings the median to 93 ms, dropping held slots from 82.4 to 40.2. - Under greedy, the healthy step (size) appears to fail forty-two times, under weighted zero times; the split sets not just the resource but the **diagnosis** of the failure. - A tight split raises abandoned work from 5540 to 11120 ms; with no cancellation signal, total busy slots run 130.5 against 136.7 — no gain. With one, the total becomes 82.4 against 40.2 — a gain only with cancellation. - A single twelve-minute slowdown spends 42.6 percent of K01's monthly 28.2-minute failure share; twice a month, 85.1 percent. ## Next Step The budget decided how long a single request would wait and distributed that decision across the chain. One assumption stayed silent: the caller is **able** to make the call for every request. Facing a slowing dependency, the budget cuts the request off, reclaims the resource, and shows the failure at the right step — but does not stop the next request from arriving. Requests keep coming at the same rate, each spending the same budget on the same slow step. The problem's other face is on the write path: status events from the carrier flow in at a fixed rate, and if the consumer feeding the projection falls behind, work piles up somewhere the budget never touches. The next lesson takes up that pile and rounds the question around: how does the system **tell** the party feeding it work that it can no longer keep up, how far does that notice spread, and where does work collect if it is never made.