Asynchronous Request–Reply

Every lesson in this course had the work run detached from the party that made the request: the state event was written to a queue, the workflow ran in the background, the end-of-day batch was started by a leader. One question always stayed outside — how the requesting party learns the result. When a seller requests a thirty-day billing report, the job is queued and finishes seconds later; but the seller’s client is waiting for a reply.

Asynchronous request–reply is the arrangement that answers the request immediately with a job id and serves the result from a separate address. This lesson measures the cost of that arrangement: how many times the client asks before it sees the result, how late it sees it, and what those questions add to the peak request rate at the edge.

Why the Synchronous Path Does Not Work

The last lesson of the Service Design topic treated the end-to-end threshold as a budget: the tight threshold on a tracking read is 200 milliseconds, and every step in the chain takes a share of that budget. The period report’s duration was likewise derived from K01 in this course’s second lesson: 3,000 records per seller, a scan rate of 833.33 records/s, that is, 3.60 seconds.

Once the two numbers sit side by side, the decision is arithmetic: the report job is 18 times the threshold. Splitting the budget does not help either, because splitting narrows every step; if the step itself is larger than the entire budget, there is nothing left to split. The product of timeout and retry count does not produce a third option here. The third option is that the caller does not wait.

Three Addresses

The arrangement splits the work a single call would do into three addresses, and each address answers exactly one question. The request address asks for the report and returns immediately; the reply carries not the result but confirmation that the job was accepted, plus a job id. The status address says whether the job for that id has finished. The result address returns the output of a finished job.

The split has three consequences. First, the request address is now a short write and fits the 200-millisecond threshold; the status address is a single record read and fits the same threshold. Second, once the job id is written, the client can close the connection and fetch the result later; the client crashing does not drop the job. Third, sending the same request twice must not start a second job — the request address must be idempotent, and the same idempotency key must return the same job id. These three consequences are the arrangement itself; what remains to measure is how many times the client asks the status address.

// async/polling.mjs — poll count and time-to-see in a long-running report request.
// Durations are computed values derived from K01; the run is arithmetic, not an environment-dependent measurement.
export const JOB_DURATION = 3.6;      // K01: 3000 records / 833.33 records/s (Task Queues and Background Jobs lesson)
export const THRESHOLD = 0.2;         // K01: the tight threshold on a tracking read is 200 ms

// The client leaves the request, then polls the status address. With exponential backoff the
// interval doubles up to the cap; the algorithm is established in M14/K06, applied here only to poll count.
export function poll({ interval, exponential = false, cap = 2, duration = JOB_DURATION }) {
  let t = 0, p = interval, polls = 0, empty = 0;
  for (;;) {
    t += p; polls += 1;
    if (t >= duration) break;
    empty += 1;
    if (exponential) p = Math.min(p * 2, cap);
  }
  return { polls, empty, seen: t, extra: t - duration,
    requests: 1 + polls, emptyRatio: empty / polls };
}

// async/measure.mjs — the poll interval is scanned, then tied to K01's volume and threshold
import { poll, JOB_DURATION, THRESHOLD } from "./polling.mjs";

const S = [...[0.25, 0.5, 1, 2].map((a) => [`fixed ${a} s`, poll({ interval: a })]),
  ["exp 0.25 s (cap 2 s)", poll({ interval: 0.25, exponential: true })]];

console.log(`job duration ${JOB_DURATION.toFixed(2)} s (K01, computed value), end-to-end threshold ${THRESHOLD * 1000} ms (K01)`);
console.log(`if kept synchronous: 1 request, connection open ${JOB_DURATION.toFixed(2)} s, ` +
  `${(JOB_DURATION / THRESHOLD).toFixed(0)}x the threshold`);
console.log();
console.log(`${"strategy".padEnd(26)}${"polls".padStart(9)}${"empty".padStart(7)}` +
  `${"empty ratio".padStart(12)}${"seen s".padStart(10)}${"extra s".padStart(10)}${"requests".padStart(10)}`);
for (const [label, r] of S)
  console.log(`${label.padEnd(26)}${String(r.polls).padStart(9)}${String(r.empty).padStart(7)}` +
    `${r.emptyRatio.toFixed(3).padStart(12)}${r.seen.toFixed(2).padStart(10)}` +
    `${r.extra.toFixed(2).padStart(10)}${String(r.requests).padStart(10)}`);

// K01: daily invoice lines 4000 = seller count (computed value); peak multiplier V8 = 3 (assumption);
// edge peak 513.89 requests/s (computed value).
const SELLERS = 4000, V8 = 3, EDGE_PEAK = 513.89, DAY = 86_400;
console.log(`\n${"KK5".padStart(5)}${"reports/day".padStart(13)}${"strategy".padStart(27)}` +
  `${"requests/day".padStart(13)}${"peak requests/s".padStart(16)}${"share of edge".padStart(15)}`);
for (const KK5 of [0.25, 0.5]) {          // KK5: period report requests per seller per day
  const reports = SELLERS * KK5;
  for (const [label, r] of [S[0], S[4]]) {
    const peak = (reports * r.requests / DAY) * V8;
    console.log(`${KK5.toFixed(2).padStart(5)}${reports.toFixed(0).padStart(13)}${label.padStart(27)}` +
      `${(reports * r.requests).toFixed(0).padStart(13)}${peak.toFixed(4).padStart(16)}` +
      `${`${(100 * peak / EDGE_PEAK).toFixed(3)}%`.padStart(15)}`);
  }
}
const sync = (reports) => ((reports / DAY) * V8 * JOB_DURATION);
console.log(`\nif kept synchronous, open connections at peak: KK5 = 0.25 -> ` +
  `${sync(SELLERS * 0.25).toFixed(3)}, KK5 = 0.50 -> ${sync(SELLERS * 0.5).toFixed(3)}`);

job duration 3.60 s (K01, computed value), end-to-end threshold 200 ms (K01)
if kept synchronous: 1 request, connection open 3.60 s, 18x the threshold

strategy                      polls  empty empty ratio    seen s   extra s  requests
fixed 0.25 s                     15     14       0.933      3.75      0.15        16
fixed 0.5 s                       8      7       0.875      4.00      0.40         9
fixed 1 s                         4      3       0.750      4.00      0.40         5
fixed 2 s                         2      1       0.500      4.00      0.40         3
exp 0.25 s (cap 2 s)              4      3       0.750      3.75      0.15         5

  KK5  reports/day                   strategy requests/day peak requests/s  share of edge
 0.25         1000               fixed 0.25 s        16000          0.5556         0.108%
 0.25         1000       exp 0.25 s (cap 2 s)         5000          0.1736         0.034%
 0.50         2000               fixed 0.25 s        32000          1.1111         0.216%
 0.50         2000       exp 0.25 s (cap 2 s)        10000          0.3472         0.068%

if kept synchronous, open connections at peak: KK5 = 0.25 -> 0.125, KK5 = 0.50 -> 0.250

Reading the Numbers

The four fixed-interval rows show a trade-off. When the interval grows fourfold (0.25 → 1 second), poll count drops from 15 to 4; in exchange, time-to-see rises from 3.75 to 4.00 seconds. Extra wait rises from 0.15 to 0.40 seconds, that is, 2.67 times. The trade-off is asymmetric: four times fewer questions, only 0.25 seconds later a result.

The empty-poll ratio names the actual waste. At the quarter-second interval, 14 of 15 polls receive a “still running” reply: the ratio is 0.933. At the two-second interval, the ratio drops to 0.500. A fixed interval does not know how long the job will take, so it asks at the same frequency at the start of the job and at the end; yet the knowledge that a job will finish grows over time.

The last row puts this to use. Exponential-backoff polling starts at a quarter second, doubles the interval up to a two-second cap, and sees the result in 4 polls at 3.75 seconds — close to the fixed two-second interval’s poll count, and the same as the fixed quarter-second interval’s time-to-see. Extra wait stays at 0.15 seconds. None of the fixed-interval rows deliver both numbers together. The backoff algorithm was established in the Resilience and Reliability course; here it is applied not to retrying but to polling, and it behaves the same way.

Back to the Estimate

Seeing what polling requests add to the peak rate at the edge requires the report volume. K01 computed 4,000 invoice lines a day, and since each line is one seller’s one day, the seller count is 4,000. Report request frequency is not in K01.

KK5 — 0.25 period report requests per seller per day. The rationale is that sellers request the period report not every day but once or twice a week. It is not added to K01’s table; its sensitivity is calculated at 0.50. The peak multiplier is K01’s V8 assumption (3).

Read the result two ways. At the fixed quarter-second poll, 16,000 requests result a day and amount to 0.5556 requests/s at the peak — 0.108 percent of K01’s 513.89 requests/s at the edge. At exponential backoff, 5,000 requests, 0.1736 requests/s, and 0.034 percent. If KK5 doubles, the shares become 0.216 percent and 0.068 percent. That is, polling’s cost at the edge is a few thousandths in every case.

The synchronous alternative costs little either: connections open at once at the peak amount to 0.125 (0.250 at KK5 = 0.50). Read together, the two numbers make the pattern’s rationale clear: asynchronous request–reply is not a resource decision, it is a budget decision. Both paths are cheap in resources; the distinction is that a 3.60-second job can never drop below a 200-millisecond threshold. In a system with a threshold, the only way out for work that does not fit it is to not complete the request with a reply.

Summary

• The 3.60-second period report is 18 times K01’s 200-millisecond end-to-end threshold; splitting the budget narrows a step, it does not enlarge it.
• The arrangement sets up three addresses — request, status, result; the request address is a short write and fits the threshold, and a second request with the same idempotency key does not start a new job.
• At a fixed interval, poll count and time-to-see trade off asymmetrically: as the interval rises from 0.25 to 1 second, polls drop from 15 to 4 while extra wait rises from 0.15 to 0.40 seconds.
• The empty-poll ratio is 0.933 at a fixed quarter second and 0.500 at two seconds; a fixed interval does not use the fact that the probability of a job finishing grows over time.
• Exponential-backoff polling delivers both numbers together: 4 polls and 3.75 seconds to see the result, extra wait 0.15 seconds.
• Back to K01: at KK5 = 0.25, polling amounts to 0.5556 requests/s at the peak (0.108 percent of the edge rate), 0.1736 requests/s with exponential backoff (0.034 percent); kept synchronous, that would be 0.125 open connections at the peak. The pattern is a budget decision, not a resource decision.

Course Wrap-Up

The course designed what happens after the request reaches the application, across two topics, and asked the same three questions in every lesson: what is this decision, which number in the introductory courses’ accounting does it move, and which number grows in exchange. The measures the lessons leave behind are collected in the table below.

Lesson	Decision	Number moved	Grows in exchange
Stateless Services	moving state out of the service	replicas needed 4 → 3; per-replica 200.93 → 171.30 requests/s	1,027.78 operations/s to the state store, 7.40 times K01’s store load
Service Discovery	finding the address at runtime	requests per failure 102,778 → 510.46	list window 2,980 ms; the registry’s own load 3.00 heartbeats/s
Communication Styles	the shape of the internal call	internal bytes 373 / 195 / 207; internal requests 833.34 requests/s	names the caller knows 6 → 8; internal bound 0.41–0.78 times 1.60 Mbit/s
Inter-Service Contracts	the schema’s evolution order	broken consumers 4/5 → 0	record 243 → 423 bytes; internal bound 0.810 → 1.410 Mbit/s
Timeout and Retry Budget	distributing the budget across the chain	worst-case duration from 6.1 times the budget to 200 ms	internal requests 1,666.68 → 1,895.85 requests/s; 3,333.36 under a widespread slowdown
Message Queues	spreading the load over time	processing capacity 97.22 → 48.61 events/s	backlog peak 525,000, longest wait 3.00 hours
Task Queues and Background Jobs	taking the long job out of the request path	tracking requests waiting over 100 ms 29.02 percent → 0	report duration 3.60 → 4.14 s; 4 slots and 4.00 workers
Competing Consumers	how many consumers the work is split across	at least 9.45 events/s per consumer for 48.61 events/s; contribution 0.128 at N = 8	order breakage 102 → 2,279 shipments
Queue-Based Load Leveling	buffering an unforeseen spike	arrivals climbing to 972 events/s while backlog holds at 525,000, wait at 3.00 hours	queue bound 525,391; 350,049 dropped events under the volume increase
Priority Queue	splitting jobs by service level	the top class’s wait drops from 6.5 minutes to 0.0	300,000 jobs stay queued in the bottom class; under weighted share, B’s backlog 440,000 → 740,000
Sequential Convoy	putting the order constraint into a lane	wrong final state 1 → 0; constrained peak write 55.55 events/s	consumer requirement ×1.357, ×1.786 with a coarse key
Choreography and Orchestration	how the workflow is assembled	compensation responsibility 3 → 1 unit; uncompensated steps 80 → 0/day	messages 1.3889 → 2.2222 messages/s, ratio 1.600
Scheduler–Agent–Supervisor	finding the half-finished job	half-finished jobs 160 → 0/day, 4 percent of invoice lines	320 re-drives and 80 wasted runs/day
Leader Election	giving singular responsibility to a single node	duplicate and unproduced invoice lines 2,000 + 2,000 → 0/yr	a two-round delay; two-leader rounds stay at 2
Asynchronous Request–Reply	the caller not waiting	a 3.60 s job moves outside the 200 ms threshold	polling at 0.1736 requests/s at the peak, 0.034 percent of the edge rate

The table’s rule deserves its own name. This course has two measures, and the choice is made by whether the caller waits. If the caller waits, the measure is chain latency: step count, the product of timeout and retry count, worst-case end-to-end duration, and requests dropped once the budget is exceeded. If the caller does not wait, the measure is backlog and drain: the peak of waiting work, drain duration, each added consumer’s contribution, and work dropped at the buffer’s bound. The two measures do not substitute for each other; a queue decision cannot be defended with a latency budget, nor a chain decision with a backlog peak. The course’s own assumptions — the state store’s load, the registry’s time to live, the gateway’s share, the peak window, long-job demand, carrier outages, the extra stream’s duration, the consumer group’s capacity, the order-constraint ratio, the workflow’s failure ratio, the stall ratio, handover frequency, and report request frequency — never mixed into K01’s table; each was written down with its own name, rationale, and sensitivity.

The question the course leaves behind sits one layer deeper. Services became stateless, found their addresses, evolved without breaking their contracts, and split their budgets; work was queued, given priority, kept in order, coordinated as a flow, and had its half-finished jobs found. But this entire design rests on the assumption of a single data store. When state was moved out of the service, it was put somewhere, and how that place would scale was never asked: replicating the store, partitioning the data, federating contexts into their own stores, and which layer the cache sits in were not designed in this course. The requests/s reaching the store and GB of stored data rows in K01’s accounting never moved across this entire course. The next course, Scaling the Data Layer, takes up those rows.