Timeout and Retry Budget

After four lessons, a call can be made: the services are stateless, their addresses are found at runtime, the conversation style is chosen, and their contracts can evolve without breaking. One question is still open, and it returns to this topic’s first measure. When the gateway calls two services and one of them does not answer, how long does it wait, how many times does it retry, and what does that waiting add to the end-to-end response.

The retry decision, exponential backoff, jitter, and the circuit breaker were established in the Application Architecture course; the algorithms are not retold here. What this lesson takes up is a budget. The budget’s source is K01: the tracking read’s hard threshold is 200 milliseconds, and the source of that threshold is the response time the recipient expects from the tracking page. The number of steps in the chain was also already measured — the third lesson counted two internal calls for one external request and showed this comes from the domain split, not the protocol.

Splitting the Budget

The chain is this: the gateway calls two services in parallel, and each service calls its own store. Depth 2, arms 2, four step calls in total. The gateway’s own work also asks for a share; this is this course’s assumption.

U3 — the gateway’s own share is 20 ms. Rationale: validation, gate checks, and merging the two responses were moved to the edge in K02, and that work takes up space in the budget. The sensitivity is computed below; it is not added to K01’s table.

// chain/budget.mjs — splitting the end-to-end budget across the chain; deterministic arithmetic
const END_TO_END = 200;     // K01 Non-Functional Requirements: read threshold (hard) 200 ms
const GATEWAY_SHARE = 20;   // assumption U3: the gateway's own share of the budget (ms)
const STEP_MEDIAN = 8;      // model parameter: a step call's usual duration (ms)
const remaining = END_TO_END - GATEWAY_SHARE;

console.log(`end-to-end ${END_TO_END} ms, gateway's share ${GATEWAY_SHARE} ms, remaining for steps ${remaining} ms\n`);
console.log(`${"depth".padStart(9)}${"retries".padStart(8)}${"no-budget worst(ms)".padStart(22)}` +
  `${"budget ratio".padStart(14)}${"allocated timeout(ms)".padStart(31)}${"ratio to median".padStart(16)}`);
for (const depth of [1, 2, 3, 4]) {
  for (const tries of [1, 3]) {
    const noBudget = depth * tries * END_TO_END + GATEWAY_SHARE;
    const allocated = remaining / (depth * tries);
    console.log(`${String(depth).padStart(9)}${String(tries).padStart(8)}${String(noBudget).padStart(22)}` +
      `${`x${(noBudget / END_TO_END).toFixed(1)}`.padStart(14)}${allocated.toFixed(1).padStart(31)}` +
      `${`x${(allocated / STEP_MEDIAN).toFixed(2)}`.padStart(16)}`);
  }
}
const maxDepth = Math.floor(remaining / (3 * STEP_MEDIAN));
console.log(`\nat 3 retries and a step median of ${STEP_MEDIAN} ms, the timeout can go no more than ` +
  `${maxDepth} steps deep before falling below the median`);
console.log(`sensitivity: if the gateway's share went ${GATEWAY_SHARE} -> ${GATEWAY_SHARE * 2} ms, at depth 2, retries 3, ` +
  `the timeout would go ${(remaining / 6).toFixed(1)} -> ${((END_TO_END - 2 * GATEWAY_SHARE) / 6).toFixed(1)} ms`);

end-to-end 200 ms, gateway's share 20 ms, remaining for steps 180 ms

    depth retries   no-budget worst(ms)  budget ratio          allocated timeout(ms) ratio to median
        1       1                   220          x1.1                          180.0          x22.50
        1       3                   620          x3.1                           60.0           x7.50
        2       1                   420          x2.1                           90.0          x11.25
        2       3                  1220          x6.1                           30.0           x3.75
        3       1                   620          x3.1                           60.0           x7.50
        3       3                  1820          x9.1                           20.0           x2.50
        4       1                   820          x4.1                           45.0           x5.63
        4       3                  2420         x12.1                           15.0           x1.88

at 3 retries and a step median of 8 ms, the timeout can go no more than 7 steps deep before falling below the median
sensitivity: if the gateway's share went 20 -> 40 ms, at depth 2, retries 3, the timeout would go 30.0 -> 26.7 ms

These numbers are in the computed value class. The third column shows the most common mistake: if every step is given the end-to-end threshold itself as its timeout, then at depth 2 and 3 retries, the worst case is 1220 milliseconds — 6.1x the budget. Setting the timeout equal to the threshold does not protect the budget; every step becomes able to spend the entire budget on its own.

The fifth column gives the correct split, and it comes with a limit. The timeout allotted to a step is divided by the product of depth and retry count: 30 milliseconds at depth 2 and 3 retries, 15 milliseconds at depth 4. The last column says why this is a limit — as the timeout approaches the step’s usual duration, ordinary requests start timing out too. At a step median of eight milliseconds and three retries, the budget can reach a depth of at most seven steps.

This is the bond between service splitting and the latency threshold: every step added to the chain divides the budget, and it cannot be divided forever.

Model

The run below is a model: it uses a logical clock, step durations are read from a cycle, and the slowdown is triggered by a parameter. It runs in two regimes — in the normal regime most steps are fast and a step fails to answer once in a while; in the widespread slowdown regime most steps fail to answer. No real service fleet or network is set up.

// chain/run.mjs — the chain's MODEL: logical clock, step durations as a parameter, two regimes.
// Chain: gateway -> (delivery-ops, billing) parallel -> each calls its own store. Depth 2, arms 2.
const N = 40, THRESHOLD = 200, GATEWAY_SHARE = 20, DEPTH = 2, ARM = 2, STEP = DEPTH * ARM;
const REGIME = {                                  // model parameter: duration of a step call (ms)
  normal: [6, 7, 6, 8, 6, 7, 6, 45, 6, 7, 6, 8, 6, 7, 6, 1000],
  "widespread slowdown": [6, 1000, 1000, 1000, 6, 1000, 1000, 1000],
};

function run({ timeout, tries, deadline, retryBudget }, regime) {
  let s = 0, primary = 0, extra = 0;
  const count = { withinThreshold: 0, outsideThreshold: 0, dropped: 0 }, endToEnd = [];
  const duration = () => REGIME[regime][s++ % REGIME[regime].length];
  const allowed = () => retryBudget === null || extra < retryBudget * primary;

  const step = (remaining) => {                    // one step: timeout and retry
    let elapsed = 0;
    for (let i = 0; i < tries; i += 1) {
      const stepTimeout = deadline ? Math.min(timeout, remaining - elapsed) : timeout;
      if (stepTimeout <= 0) return { ok: false, elapsed };
      if (i === 0) primary += 1; else extra += 1;
      const d = duration();
      if (d <= stepTimeout) return { ok: true, elapsed: elapsed + d };
      elapsed += stepTimeout;
      if (i + 1 < tries && allowed() === false) return { ok: false, elapsed };
    }
    return { ok: false, elapsed };
  };
  const arm = () => {                               // one arm: DEPTH consecutive steps
    let elapsed = 0, ok = true;
    for (let i = 0; i < DEPTH && ok; i += 1) {
      const r = step(THRESHOLD - GATEWAY_SHARE - elapsed);
      elapsed += r.elapsed; ok = r.ok;
    }
    return { ok, elapsed };
  };

  for (let i = 0; i < N; i += 1) {
    const arms = Array.from({ length: ARM }, arm);   // parallel: duration is the arms' maximum
    const elapsed = GATEWAY_SHARE + Math.max(...arms.map((a) => a.elapsed));
    endToEnd.push(elapsed);
    if (arms.every((a) => a.ok) === false) count.dropped += 1;
    else if (elapsed > THRESHOLD) count.outsideThreshold += 1;
    else count.withinThreshold += 1;
  }
  const sorted = [...endToEnd].sort((a, b) => a - b);
  return { ...count, median: sorted[N >> 1], worst: sorted[N - 1],
    multiplier: (primary + extra) / (N * STEP) };
}

const CONFIGURATION = [
  ["no budget (200/3)", { timeout: 200, tries: 3, deadline: false, retryBudget: null }],
  ["+ deadline propagation", { timeout: 200, tries: 3, deadline: true, retryBudget: null }],
  ["+ allocated (30/3)", { timeout: 30, tries: 3, deadline: true, retryBudget: null }],
  ["+ retry budget 0.30", { timeout: 30, tries: 3, deadline: true, retryBudget: 0.3 }],
];
const PEAK_READ = 416.67, INTERNAL_BASE = PEAK_READ * STEP;   // K01: peak read; lesson 03: step count

console.log(`${N} external requests, depth ${DEPTH}, arms ${ARM}, steps ${STEP}; ` +
  `end-to-end threshold ${THRESHOLD} ms (K01), gateway share ${GATEWAY_SHARE} ms (U3)`);
console.log(`internal req/s base = ${PEAK_READ} x ${STEP} = ${INTERNAL_BASE.toFixed(2)}`);
for (const regime of Object.keys(REGIME)) {
  console.log(`\n-- ${regime}: ${REGIME[regime].filter((v, i, a) => a.indexOf(v) === i).join("/")} ms cycle --`);
  console.log(`${"configuration".padEnd(24)}${"within threshold".padStart(18)}${"outside threshold".padStart(19)}` +
    `${"dropped".padStart(9)}${"median".padStart(9)}${"worst".padStart(9)}${"multiplier".padStart(12)}` +
    `${"internal req/s".padStart(16)}`);
  for (const [name, y] of CONFIGURATION) {
    const r = run(y, regime);
    console.log(`${name.padEnd(24)}${String(r.withinThreshold).padStart(18)}${String(r.outsideThreshold).padStart(19)}` +
      `${String(r.dropped).padStart(9)}${String(r.median).padStart(9)}${String(r.worst).padStart(9)}` +
      `${`x${r.multiplier.toFixed(2)}`.padStart(12)}${(INTERNAL_BASE * r.multiplier).toFixed(2).padStart(16)}`);
  }
}
console.log(`\nunlimited-retry upper bound: 3 tries per step -> ${(INTERNAL_BASE * 3).toFixed(2)} req/s`);

40 external requests, depth 2, arms 2, steps 4; end-to-end threshold 200 ms (K01), gateway share 20 ms (U3)
internal req/s base = 416.67 x 4 = 1666.68

-- normal: 6/7/8/45/1000 ms cycle --
configuration             within threshold  outside threshold  dropped   median    worst  multiplier  internal req/s
no budget (200/3)                       30                 10        0       71      233       x1.06         1770.85
+ deadline propagation                  30                  0       10       71      200       x1.00         1666.68
+ allocated (30/3)                      40                  0        0       62       63       x1.14         1895.85
+ retry budget 0.30                     40                  0        0       62       63       x1.14         1895.85

-- widespread slowdown: 6/1000 ms cycle --
configuration             within threshold  outside threshold  dropped   median    worst  multiplier  internal req/s
no budget (200/3)                        0                  0       40      626      626       x2.00         3333.36
+ deadline propagation                   0                  0       40      200      200       x0.67         1114.59
+ allocated (30/3)                       0                  0       40      116      116       x2.00         3333.36
+ retry budget 0.30                      0                  0       40       86       86       x0.97         1614.60

unlimited-retry upper bound: 3 tries per step -> 5000.04 req/s

Reading the Four Rows

Deadline propagation turns a threshold overrun into an early failure. In the normal regime, the no-budget arrangement answers thirty of forty requests within the threshold, ten outside it — worst case 233 milliseconds. Once the caller passes the remaining budget along with the call, the worst case is cut off at exactly 200 milliseconds and those ten requests stop being outside the threshold and become dropped requests. The trade-off is explicit: a choice is made between a late answer and an on-time failure. The user who received an answer at 233 milliseconds was already not getting a threshold-meeting response; the work spent producing it was also unrewarded.

A short timeout absorbs transient slowness. In the allocated arrangement, when the timeout falls to 30 milliseconds, all forty of forty requests answer within the threshold and the worst case is 63 milliseconds. The reason is the model’s assumption: the slowness is transient, and the same step’s second call returns fast. A long timeout carries this transient slowness through to the end-to-end response; a short timeout cuts it off and retries. The cost shows up as the call multiplier rising from 1.00 to 1.14: the internal request rate climbs from 1666.68 to 1895.85 req/s.

The retry budget does not engage in the normal regime. The fourth row is identical to the third. The observed multiplier is 1.14 and the budget is 1.30, so the limit never comes into play. A budget being invisible under normal conditions is the sign that it is sized correctly.

What Changes Under Widespread Slowdown

The second table runs the same configurations in a regime where most steps do not answer, and the ranking reverses.

In the no-budget arrangement, all forty of forty requests drop, and the caller waits 626 milliseconds to learn this. Deadline propagation cuts this off at 200 milliseconds; the multiplier falls to 0.67, because there are also primary calls that never get made once the budget runs out. The internal request rate falls from 3333.36 to 1114.59 req/s.

The third row is the lesson’s most important number. When the timeout is cut to 30 milliseconds, the end-to-end duration falls to 116 milliseconds — but the multiplier climbs back to 2.00 and the internal request rate becomes 3333.36 req/s. A short timeout fits more retries into the same budget; a service that is already slow receives twice the requests from the system that is slowing it down. A decision that was good in the normal regime is harmful in the failure regime.

The fourth row cuts this off. Once the retry budget limits extra retries to thirty percent of primary calls, the multiplier falls to 0.97 and the internal request rate to 1614.60 req/s: a below-baseline load instead of a doubling. The end-to-end duration also falls, from 116 to 86 milliseconds, because wasted retries are not made. The unlimited-retry upper bound is the comparison point — if every step is tried three times, 5000.04 req/s, nearly ten times K01’s peak rate at the edge as internal load.

All of this holds only for calls that can be retried. A state-changing call being retryable depends on processing the same call twice giving the same result as processing it once; this property is called idempotent, the same concept as the idempotency property of HTTP methods. A call that is not idempotent can only be retried if the client generates an idempotency key that is written into the contract; both were established in the Web API Design course.

Summary

• The end-to-end threshold is a budget and it is split across the chain; if every step is given the full threshold as its timeout, the worst case at depth 2 and 3 retries climbs to 6.1x the budget.
• The allocated timeout is divided by the product of depth and retry count (30 ms at depth 2, retries 3) and hits a limit as it approaches the step’s usual duration: at an 8 ms median and 3 retries, the budget can reach a depth of at most 7 steps.
• Deadline propagation cuts the worst case off at the threshold: in the normal regime, 233 ms and 10 outside-threshold responses turn into 200 ms and 10 dropped requests — an on-time failure instead of a late answer.
• A short timeout absorbs transient slowness (40/40 within threshold, worst case 63 ms) at the cost of the call amplification rising from 1.00 to 1.14 and the internal request rate climbing from 1666.68 to 1895.85 req/s.
• Under widespread slowdown the same decision reverses: a short timeout drives the multiplier to 2.00 and the internal request rate to 3333.36 req/s; a retry budget of 0.30 brings this to 0.97 and 1614.60 req/s and cuts the end-to-end duration from 116 to 86 ms.
• Retrying is free only for idempotent calls; a call that is not idempotent can be retried only if the contract carries an idempotency key.

Next Step

This topic designed synchronous calling: the caller waits, the budget does its work, and in the end either an answer arrives or a failure does. The budget calculation made a limit visible — as the chain lengthens, either the budget splits into pieces and every step narrows, or the end-to-end duration grows. Both options sit inside the same assumption: the caller is waiting for the result of the work. For some jobs there is a third path, and that path removes the assumption — the caller never waits at all. A state event arriving from the carrier being written into the store does not have to finish in order to answer the request. The next topic measures what dropping a request into a queue changes, and it asks this as a capacity question: on the synchronous path, capacity has to be chosen for the instantaneous peak; once a queue is added, it can be brought closer to the average. What that gain costs in pending work and waiting time, and which of K01’s numbers change, is computed there.