Message Queues

The previous topic designed the synchronous call: the caller waits, a timeout budget runs, and at its end either a response or an error arrives. Its last lesson worked out that budget’s arithmetic and exposed a limit: as the chain grows, either the budget splits into pieces and each step narrows, or the end-to-end duration grows. Both options share one assumption — the caller waits for the result. For some jobs a third path exists, and it drops that assumption: the caller never waits at all. Writing the carrier’s state event into the store does not have to finish before the request can be answered.

This topic takes up that path. Message queue mechanics were built and measured in the Caching, Queues and Asynchronous Processing course: producer writes, consumer receives, acknowledgement, visibility timeout, delivery guarantees, consumer groups, ordering, the dead letter queue. None of it is retold here. This course asks a different question, in one sentence: a queue is a capacity decision. What is gained, what is paid, and which number from the introductory course changes when load spreads over time.

Which Number Sets Capacity

On the synchronous write path one rule is not up for debate: capacity is chosen by the instantaneous peak rate. If the carrier sends a state event and waits for a response, every request arriving at that instant needs somewhere to be processed; one that does not get it is rejected. The Introduction to System Design course’s Back-of-the-Envelope Estimation lesson gives this number: peak writes of 97.22 requests/s — a computed value that follows arithmetically from 2,800,000 daily state events, a daily average of 32.41 requests/s, and V8’s peak factor of 3.

A queue splits that rule in two. The endpoint the carrier talks to still has to accept the peak, but its work there shrinks to a single append: write the event, return the acknowledgement. Actually processing it — updating the shipment record, writing the route step, preparing the pricing line — moves behind the queue, where capacity becomes something you choose. That freedom is the queue’s gain; its limit is the queue itself, since every unconsumed event piles up.

Calculating the backlog needs a number missing from K01’s table. V8 says how high the peak is, not how long it lasts.

K1 — peak window: 3 hours a day. This course’s own assumption; not added to K01’s table. Rationale: carriers’ delivery rounds produce the state events, and a round is packed into one slice of the day. Since the daily total comes from K01, the off-peak rate is no longer free: three hours at 97.22 requests/s produces 1,050,000 events, the remaining 1,750,000 spread over the other twenty-one hours, and the off-peak rate works out to 23.15 requests/s. Its sensitivity is measured in the second table below.

Backlog

The setup below builds the queue, producer, and consumer as in-process modules. A tick is one second; it runs a day second by second and tracks the backlog against the chosen processing capacity. This is a model, not a measurement: every number in it follows arithmetically from K01 and K1, so results are deterministic and machine-independent — the computed-value class. It runs four days so the starting backlog becomes periodic, and the numbers are taken from the third day.

// queue/spread.mjs — feeding a fixed-capacity consumer from the daily event profile.
// MODEL: queue and consumer are in-process modules, one tick is one second. Results are
// deterministic (computed-value class) and machine-independent; there is no measured duration.
const DAY = 86_400;
const DAILY_EVENTS = 2_800_000;    // K01 Back-of-the-Envelope Estimation: daily state events
const PEAK_FACTOR = 3;             // K01 assumption V8
const PEAK_START = 9 * 3600;       // where the peak window sits inside the day
const THRESHOLD = 300;             // "delayed event" threshold (s)

function profile(peakHours) {
  const average = DAILY_EVENTS / DAY;
  const peak = average * PEAK_FACTOR;
  const peakSec = peakHours * 3600;
  const offPeak = (DAILY_EVENTS - peak * peakSec) / (DAY - peakSec);
  return { average, peak, offPeak,
    arrival: (t) => (t >= PEAK_START && t < PEAK_START + peakSec ? peak : offPeak) };
}

function run(capacity, p) {         // runs four days; the first two days make the starting
  let backlog = 0;                  // backlog periodic, numbers are taken from the third day
  let highest = 0, total = 0, delayed = 0, drain = -1;
  const MEASURE = 2 * DAY;          // start of the measurement day
  for (let t = 0; t < 4 * DAY; t += 1) {
    const arriving = p.arrival(t % DAY);
    if (t >= MEASURE && t < MEASURE + DAY) {
      if (backlog / capacity > THRESHOLD) delayed += arriving;
      total += arriving;
    }
    backlog += arriving;
    backlog -= Math.min(backlog, capacity);
    if (t >= MEASURE && t < MEASURE + DAY && backlog > highest) highest = backlog;
    if (drain < 0 && t > MEASURE + PEAK_START && backlog < 1) drain = t - MEASURE - PEAK_START;
  }
  return { highest, drain, delayed, rate: delayed / total,
    wait: highest / capacity };
}

function table(peakHours) {
  const p = profile(peakHours);
  console.log(`K1 = ${peakHours} hour peak window -> peak ${p.peak.toFixed(2)}/s, ` +
    `off-peak ${p.offPeak.toFixed(2)}/s, average ${p.average.toFixed(2)}/s`);
  console.log("capacity  ratio to peak  highest backlog  longest wait  drain  delayed events  delayed rate");
  for (const ratio of [1, 2 / 3, 0.5, 0.4, 0.35, 1 / 3]) {
    const k = p.peak * ratio;
    const r = run(k, p);
    const w = r.wait < 3600 ? `${(r.wait / 60).toFixed(1)} min` : `${(r.wait / 3600).toFixed(2)} h`;
    const d = r.drain < 0 ? "none" : `${(r.drain / 3600).toFixed(2)} h`;
    console.log(`${k.toFixed(2).padStart(8)}  ${ratio.toFixed(2).padStart(13)}  ` +
      `${r.highest.toFixed(0).padStart(16)}  ${w.padStart(12)}  ${d.padStart(5)}  ` +
      `${r.delayed.toFixed(0).padStart(14)}  ${r.rate.toFixed(3).padStart(12)}`);
  }
}

console.log(`${DAILY_EVENTS} state events a day (K01), peak factor ${PEAK_FACTOR} (V8)`);
console.log(`"delayed event" = an event waiting longer than ${THRESHOLD} seconds\n`);
table(3);
console.log();
table(6);

2800000 state events a day (K01), peak factor 3 (V8)
"delayed event" = an event waiting longer than 300 seconds

K1 = 3 hour peak window -> peak 97.22/s, off-peak 23.15/s, average 32.41/s
capacity  ratio to peak  highest backlog  longest wait  drain  delayed events  delayed rate
   97.22           1.00                 0       0.0 min  0.00 h               0         0.000
   64.81           0.67            350000        1.50 h  5.33 h         1175324         0.420
   48.61           0.50            525000        3.00 h  8.73 h         1484861         0.530
   38.89           0.40            630000        4.50 h  14.12 h         1939884         0.693
   34.03           0.35            682500        5.57 h  20.43 h         2464667         0.880
   32.41           0.33            700000        6.00 h  24.00 h         2761134         0.986

K1 = 6 hour peak window -> peak 97.22/s, off-peak 10.80/s, average 32.41/s
capacity  ratio to peak  highest backlog  longest wait  drain  delayed events  delayed rate
   97.22           1.00                 0       0.0 min  0.00 h               0         0.000
   64.81           0.67            700000        3.00 h  9.60 h         2177789         0.778
   48.61           0.50           1050000        6.00 h  13.71 h         2366670         0.845
   38.89           0.40           1260000        9.00 h  18.46 h         2560693         0.915
   34.03           0.35           1365000       11.14 h  22.33 h         2714390         0.969
   32.41           0.33           1400000       12.00 h  24.00 h         2780556         0.993

What Is Gained, What Is Paid

The first row of the first table is the synchronous design itself: capacity equal to the peak means zero backlog and zero wait. Every row below gives up capacity and buys wait time instead.

The gain is linear. At half the peak, the processing side shrinks by half; at a third, it drops to a third. The lower bound is no coincidence: a third of the peak is the daily average, and V8 defines the peak as three times the average. The upper bound on a queue’s capacity gain is the peak factor itself — sizing to the average instead of the peak gains exactly V8, and no more.

The cost is not linear. As capacity drops from 1.00 to 0.67, the highest backlog rises from 0 to 350,000; at 0.50 it reaches 525,000, at 0.40 it reaches 630,000. Giving up the last seventeen percent of capacity adds 105,000 to the backlog, while the first thirty-three percent added 350,000 — but wait time reverses that order: the longest wait climbs from 1.50 hours to 3.00, then 4.50. Since wait is backlog divided by capacity, the numerator grows as the denominator shrinks, and the two effects collide. In the row closest to the average, the wait is six hours.

The drain column draws a line. At 0.35 of peak capacity, the backlog reaches zero 20.43 hours after the peak; at 0.33 it takes exactly 24.00 hours — zero at the exact moment the next day’s peak begins. This is a knife’s edge: sizing capacity to the average leaves a single busy day’s backlog carrying into the next, growing day after day. Sizing to the average is a theoretical limit, not a design you can actually choose.

The delayed-events column says who pays. Events waiting longer than five minutes number 1,175,324 at 0.67 of peak, 42 percent of the day’s events. That climbs to 53 percent at half capacity, 69 percent at 0.40. This number is directly the freshness the tracking query sees: when the carrier sends “out for delivery,” the recipient’s tracking page shows it only once the backlog drains.

The second table is K1’s sensitivity. Growing the peak window from three hours to six does not change the peak rate — that comes from V8 — but the off-peak rate drops from 23.15 to 10.80 requests/s, and backlogs exactly double: 1,050,000 instead of 525,000 at 0.50 of peak, six hours of wait instead of three. The assumption’s effect on the design is direct and one-directional: the peak’s duration matters as much as its height, and K01’s table did not carry this information.

Two Designs Facing an Outage

Looking only at the table, the decision could read against the queue. One measurement is missing: a queue does not just spread load over time, it decouples the producer from the consumer’s failures. The second setup adds a case where the store refuses writes for fifteen minutes mid-peak. The outage is a parameter.

// queue/outage.mjs — comparing two designs when the store refuses writes for 15 minutes.
// MODEL: one tick is one second, the outage is triggered by a parameter. Results are
// deterministic (computed-value class).
const DAY = 86_400;
const DAILY_EVENTS = 2_800_000;    // K01: daily state events
const PEAK = (DAILY_EVENTS / DAY) * 3;              // K01 calculation + V8: peak write events/s
const PEAK_START = 9 * 3600, PEAK_SEC = 3 * 3600;   // K1: 3-hour peak window
const OFF_PEAK = (DAILY_EVENTS - PEAK * PEAK_SEC) / (DAY - PEAK_SEC);
const OUTAGE_START = 10 * 3600, OUTAGE_SEC = 900;   // outage sits in the middle of the peak window

const arrival = (t) => (t >= PEAK_START && t < PEAK_START + PEAK_SEC ? PEAK : OFF_PEAK);
const isOutage = (t) => t >= OUTAGE_START && t < OUTAGE_START + OUTAGE_SEC;

function run(capacity, hasQueue, hasOutage) {
  let backlog = 0, lost = 0, highest = 0, drain = -1;
  for (let t = 0; t < 2 * DAY; t += 1) {
    const day = t % DAY;
    const arriving = arrival(day);
    const cap = hasOutage && isOutage(day) ? 0 : capacity;
    if (hasQueue) backlog += arriving;
    else if (arriving > cap) { lost += (arriving - cap) * (t >= DAY ? 1 : 0); }
    backlog -= Math.min(backlog, cap);
    if (t >= DAY) {
      if (backlog > highest) highest = backlog;
      if (drain < 0 && day > PEAK_START && backlog < 1) drain = day - PEAK_START;
    }
  }
  return { lost, highest, drain, wait: highest / capacity };
}

const format = (r, capacity) => `${capacity.toFixed(2).padStart(8)}  ` +
  `${r.lost.toFixed(0).padStart(13)}  ${r.highest.toFixed(0).padStart(16)}  ` +
  `${(r.wait / 3600).toFixed(2).padStart(11)}  ` +
  `${(r.drain < 0 ? NaN : r.drain / 3600).toFixed(2).padStart(11)}`;

console.log(`outage: ${OUTAGE_SEC} s, inside the peak window; events arriving during it: ` +
  `${(PEAK * OUTAGE_SEC).toFixed(0)}\n`);
console.log("design                   capacity  lost events  highest backlog  wait (h)  drain (h)");
console.log(`${"synchronous write".padEnd(22)} ${format(run(PEAK, false, false), PEAK)}`);
console.log(`${"synchronous + outage".padEnd(22)} ${format(run(PEAK, false, true), PEAK)}`);
for (const ratio of [0.5, 0.4]) {
  const k = PEAK * ratio;
  console.log(`${`queue ${ratio.toFixed(2)}x peak`.padEnd(22)} ${format(run(k, true, false), k)}`);
  console.log(`${`queue ${ratio.toFixed(2)}x + outage`.padEnd(22)} ${format(run(k, true, true), k)}`);
}

const BEHIND_CACHE_READ = 41.67;   // K01: reads behind cache/s
const STORE_BASELINE = 138.89;     // K01: requests reaching the store/s
console.log(`\nK01 revisited: requests reaching the store/s, baseline ${STORE_BASELINE} (reads ${BEHIND_CACHE_READ} + writes ${PEAK.toFixed(2)})`);
for (const ratio of [1, 0.5, 0.4]) {
  const k = PEAK * ratio, store = BEHIND_CACHE_READ + k;
  console.log(`  processing capacity ${k.toFixed(2).padStart(6)}/s -> reaching the store ${store.toFixed(2).padStart(6)}/s, ` +
    `${(store / STORE_BASELINE).toFixed(2)}x the baseline, write/read at the store ${(k / BEHIND_CACHE_READ).toFixed(2)}`);
}

outage: 900 s, inside the peak window; events arriving during it: 87500

design                   capacity  lost events  highest backlog  wait (h)  drain (h)
synchronous write         97.22              0                 0         0.00         0.00
synchronous + outage      97.22          87500                 0         0.00         0.00
queue 0.50x peak          48.61              0            525000         3.00         8.73
queue 0.50x + outage      48.61              0            568750         3.25         9.20
queue 0.40x peak          38.89              0            630000         4.50        14.12
queue 0.40x + outage      38.89              0            665000         4.75        14.73

K01 revisited: requests reaching the store/s, baseline 138.89 (reads 41.67 + writes 97.22)
  processing capacity  97.22/s -> reaching the store 138.89/s, 1.00x the baseline, write/read at the store 2.33
  processing capacity  48.61/s -> reaching the store  90.28/s, 0.65x the baseline, write/read at the store 1.17
  processing capacity  38.89/s -> reaching the store  80.56/s, 0.58x the baseline, write/read at the store 0.93

The fifteen-minute outage drops 87,500 events in the synchronous design — straight arithmetic: 900 seconds × 97.22 requests/s. Even if the carrier retries, those retries are a separate design decision, addressed in the Resilience and Reliability course; what is measured here is what the design itself loses on its own.

The same outage drops zero events in the queued design. Instead it pays 43,750 events added to the backlog and fifteen minutes added to the wait: at half capacity, the highest backlog rises from 525,000 to 568,750, the wait from 3.00 to 3.25 hours, the drain from 8.73 to 9.20 hours. This is not a comparison but a conversion: the queue turns loss into delay. Where loss is unacceptable and delay tolerable, that is a gain; otherwise, a loss.

Back to the Numbers

The last three lines rewrite one of K01’s calculations. The Back-of-the-Envelope Estimation lesson found the peak request rate reaching the store at 138.89 requests/s: 41.67 reads behind the cache plus 97.22 writes. That write share was the synchronous design’s assumption — every event written to the store the moment it arrives.

A queue makes the write share something you choose. At half the peak, the request rate reaching the store drops to 90.28 requests/s, 0.65 of the baseline; at 0.40 of peak, 80.56 requests/s, 0.58 of the baseline. The write/read ratio at the store shifts too: K01’s 2.33 becomes 1.17 and 0.93. K01 found the system write-heavy where the store sees it; a queue can reverse that balance, and sizing the store starts from this new number.

The decision comes down to one line. A queue gains capacity only when three conditions hold together: a real gap between peak and average (V8 greater than one), the caller not waiting for the result, and a tolerable delay in the work’s effect. The third is skipped most often; the “delayed events” column is its cost.

Summary

• On the synchronous write path, capacity is chosen by the instantaneous peak; K01’s peak write rate of 97.22 requests/s is this design’s requirement.
• A queue splits the capacity decision in two: the acceptance path still takes the peak, processing capacity becomes a choice; the gain’s upper bound is the peak factor V8, three times.
• The gain is linear, the cost is not: as capacity drops to 0.67 / 0.50 / 0.40 of the peak, the highest backlog becomes 350,000 / 525,000 / 630,000, and the longest wait becomes 1.50 / 3.00 / 4.50 hours.
• Capacity equal to the average is a knife’s edge: the backlog drains in exactly 24.00 hours, and 98.6 percent of events wait longer than five minutes.
• The K1 assumption is sensitive: growing the peak window from three to six hours doubles every backlog and wait, while the peak rate stays the same.
• A queue converts loss into delay: a fifteen-minute store outage drops 87,500 events in the synchronous design but zero in the queued one, at the cost of the wait rising from 3.00 to 3.25 hours.

Next Step

Every event in this lesson was the equal of every other: each state event was a record of the same size, doing the same work, lasting milliseconds. The backlog calculation rested on that too — capacity is a single number because the jobs are all one type. In the same system one job breaks this assumption: the period report a seller requests scans thousands of shipment records and takes seconds. Run inside a request, such a job does not just spend its own time — it holds up the short requests behind it. The next lesson separates these two kinds of work: it measures what a long job on the request path does to short requests, and counts which measurement improves once it moves into a task queue.