Queue-Based Load Leveling

The workload in the previous two lessons quietly carried one assumption: the peak is known, when it arrives is known, its duration is known. The first lesson built this from K01’s V8 factor and its own K1 assumption. In a real stream, none of the three is guaranteed. When the carrier’s connection drops for twenty minutes, no events arrive; once it returns, everything piled up on the handheld devices arrives at once, at a rate with nothing to do with the day’s peak.

This lesson takes up that spike. Queue-based load leveling is the decision to place a buffer between producer and consumer, decoupling the rate the consumer sees from the rate the producer produces. The name recalls the load balancer from the Traffic Layer course, and the two are different things: a load balancer spreads one request across several replicas, splitting load in space; queue-based load leveling spreads load headed to the same consumer over time. The difference from the first lesson fits one sentence: there, what got leveled was a predicted daily profile; here, what gets leveled is an unpredicted spike.

Two Different Kinds of Spike

A spike is not one phenomenon, and this lesson’s claim is that the two kinds must be separated. Two assumptions follow; both are this course’s own, and neither is added to K01’s table.

K4 — carrier outage: 20 minutes. Rationale: when a carrier’s handheld devices lose coverage, scans pile up on the device and get sent in bulk once the connection returns. The outage falls inside the peak window; the number of events resent is 1200 × 97.22 = 116,667. This is a rate event: volume does not change, the same events arrive in less time. How much less time is measured by the resend multiplier R, run with the values two, four, and ten.

K5 — extra write stream’s duration: 1 hour. Rationale: when a new carrier connects, or a campaign period opens, an extra one-hour stream enters the peak window. The extra stream’s rate is set equal to the peak; the number of events added is 3600 × 97.22 = 350,000. This is a volume event: the arrival rate only doubles, but the total work the system processes that day grows.

Their sensitivities are linear: doubling the outage doubles the events resent, doubling the extra stream doubles the events added. Why the distinction matters will come out of the measurement.

The Model

The setup uses the first lesson’s daily profile exactly: peak 97.22 events/s, three hours per K1, off-peak 23.15 events/s. One of the two events is added on top. Consumer capacity is the first lesson’s chosen value, half the peak. The queue limit is also a parameter; a limit of zero means the unbuffered design, where every event above capacity is turned away — this behavior was built under the name load shedding in the Caching, Queues and Asynchronous Processing course and is used here only as a comparison baseline. Being a model, the results are deterministic and machine-independent.

// leveling/spike.mjs — two shapes of spike: a bulk resend after an outage (rate rises, volume
// fixed) and an extra write stream (volume rises). MODEL: one tick is one second, the daily
// profile is the same as lesson 01's. Results are deterministic (computed-value class) and
// machine-independent.
const DAY = 86_400;
const DAILY_EVENTS = 2_800_000;          // K01: daily state events
const PEAK = (DAILY_EVENTS / DAY) * 3;   // K01 calculation + V8: peak write events/s
const PEAK_START = 9 * 3600, PEAK_SEC = 3 * 3600;   // K1: 3-hour peak window
const OFF_PEAK = (DAILY_EVENTS - PEAK * PEAK_SEC) / (DAY - PEAK_SEC);
const K4 = 20 * 60;                      // this course's assumption K4: carrier outage (s)
const K5 = 3600;                         // this course's assumption K5: extra stream's duration (s)
const START = PEAK_START + 2400;         // both events sit inside the peak window

const normal = (t) => (t >= PEAK_START && t < PEAK_START + PEAK_SEC ? PEAK : OFF_PEAK);

function run({ capacity, limit, R = 1, mode = "none" }) {
  let pending = 0, backlog = 0, dropped = 0, highest = 0, drain = -1, totalArrived = 0;
  const MEASURE = 2 * DAY;
  for (let t = 0; t < 4 * DAY; t += 1) {
    const day = t % DAY;
    let arriving = normal(day);
    if (mode === "retry") {
      if (day >= START && day < START + K4) { pending += arriving; arriving = 0; }
      else { const extra = Math.min(pending, (R - 1) * PEAK); pending -= extra; arriving += extra; }
    } else if (mode === "volume" && day >= START && day < START + K5) arriving += PEAK;
    const room = limit === 0 ? capacity : limit - backlog;
    const accepted = Math.min(arriving, Math.max(0, room));
    if (t >= MEASURE && t < MEASURE + DAY) { dropped += arriving - accepted; totalArrived += arriving; }
    backlog += limit === 0 ? Math.min(accepted, capacity) : accepted;
    backlog -= Math.min(backlog, capacity);
    if (t >= MEASURE && t < MEASURE + DAY && backlog > highest) highest = backlog;
    if (drain < 0 && t > MEASURE + PEAK_START && backlog < 1 && pending < 1) drain = t - MEASURE - PEAK_START;
  }
  return { dropped, totalArrived, highest, drain, wait: highest / capacity };
}

const arrivalRate = (o) => (o.mode === "retry" ? o.R * PEAK : o.mode === "volume" ? 2 * PEAK : PEAK);
const row = (name, o) => {
  const r = run(o), v = arrivalRate(o);
  console.log(`${name.padEnd(24)} ${v.toFixed(0).padStart(7)} ${o.capacity.toFixed(2).padStart(8)} ` +
    `${r.dropped.toFixed(0).padStart(7)} ${r.highest.toFixed(0).padStart(9)} ` +
    `${(r.wait / 3600).toFixed(2).padStart(11)} ` +
    `${(r.drain < 0 ? NaN : r.drain / 3600).toFixed(2).padStart(11)} ` +
    `${(v / o.capacity).toFixed(2).padStart(7)}`);
};

console.log(`K01 peak write ${PEAK.toFixed(2)} events/s; K4 outage ${K4} s -> ` +
  `${(PEAK * K4).toFixed(0)} events resent`);
console.log(`K5 extra stream ${K5} s x ${PEAK.toFixed(2)} events/s -> ${(PEAK * K5).toFixed(0)} events added\n`);
console.log("design                   arrival/s  capacity  dropped  backlog  wait(h)  drain(h)  leveling");
row("unbuffered, no event", { capacity: PEAK, limit: 0 });
row("unbuffered retry R=4", { capacity: PEAK, limit: 0, R: 4, mode: "retry" });
row("unbuffered volume", { capacity: PEAK, limit: 0, mode: "volume" });
row("queue 0.5x, no event", { capacity: PEAK / 2, limit: Infinity });
for (const R of [2, 4, 10]) row(`queue 0.5x retry R=${R}`, { capacity: PEAK / 2, limit: Infinity, R, mode: "retry" });
row("queue 0.5x volume", { capacity: PEAK / 2, limit: Infinity, mode: "volume" });

const smallestLimit = (o) => {
  let low = 0, high = 2_000_000;
  while (high - low > 500) { const mid = (low + high) / 2; if (run({ ...o, limit: mid }).dropped < 1) high = mid; else low = mid; }
  return high;
};
console.log("\nsmallest queue limit needed (capacity 0.5x peak):");
for (const [name, o] of [["no event", {}], ["retry R=4", { R: 4, mode: "retry" }],
  ["retry R=10", { R: 10, mode: "retry" }], ["volume", { mode: "volume" }]]) {
  console.log(`  ${name.padEnd(12)} ${smallestLimit({ capacity: PEAK / 2, ...o }).toFixed(0).padStart(9)}`);
}
console.log("\ndropped events with limit 525000 (chosen for the event-free day):");
for (const [name, o] of [["retry R=4", { R: 4, mode: "retry" }], ["retry R=10", { R: 10, mode: "retry" }],
  ["volume", { mode: "volume" }]]) {
  console.log(`  ${name.padEnd(12)} ${run({ capacity: PEAK / 2, limit: 525_000, ...o }).dropped.toFixed(0).padStart(9)}`);
}

K01 peak write 97.22 events/s; K4 outage 1200 s -> 116667 events resent
K5 extra stream 3600 s x 97.22 events/s -> 350000 events added

design                   arrival/s  capacity  dropped  backlog  wait(h)  drain(h)  leveling
unbuffered, no event          97    97.22       0         0        0.00        0.00    1.00
unbuffered retry R=4         389    97.22  116667         0        0.00        0.00    4.00
unbuffered volume            194    97.22  350000         0        0.00        0.00    2.00
queue 0.5x, no event          97    48.61       0    525000        3.00        8.73    2.00
queue 0.5x retry R=2         194    48.61       0    525000        3.00        8.73    4.00
queue 0.5x retry R=4         389    48.61       0    525000        3.00        8.73    8.00
queue 0.5x retry R=10        972    48.61       0    525000        3.00        8.73   20.00
queue 0.5x volume            194    48.61       0    875000        5.00       12.55    4.00

smallest queue limit needed (capacity 0.5x peak):
  no event        525391
  retry R=4       525391
  retry R=10      525391
  volume          875488

dropped events with limit 525000 (chosen for the event-free day):
  retry R=4           49
  retry R=10          49
  volume          350049

A Rate Spike Does Not Write to the Buffer

The four middle rows of the table are this lesson’s most counterintuitive result. As the resend multiplier climbs from two to four to ten, the arrival rate climbs from 194 to 389 to 972 events/s — ten times K01’s peak write rate. In those same rows, the highest backlog stays fixed at 525,000, the longest wait at 3.00 hours, the drain at 8.73 hours. All three numbers are identical to the event-free day’s.

The reason is volume conservation. During the outage no events arrive while the consumer keeps working, so the queue drains for those twenty minutes; the resend puts the same events back. The pit the outage digs and the peak the spike builds cancel out, because their sum has not changed. All the consumer ever sees is the same amount of work within the same day.

The leveling column on the right gives the queue’s work as a single number: arrival rate over consumer capacity. It is 2.00 on the event-free day, then 4.00, 8.00, and 20.00 on the resend rows. A twentyfold arrival spike leaves no trace on the consumer’s side. The queue’s real job fits one line: it levels rate, not volume.

In the unbuffered design, the same event drops 116,667 events, and the number dropped is independent of R — the rate it arrives at makes no difference, everything above capacity goes. In this design the only fix is sizing capacity to the arrival peak; for R = 10 that requires 972 events/s, ten times K01’s peak write rate, and a capacity that sits entirely idle for the rest of the day.

Slowing the producer down looks like an option too — the path built under the name backpressure in the Caching, Queues and Asynchronous Processing course. It does not apply here, because the producer is the carrier’s handheld device: slowed down, it just buffers events in its own memory, and the spike is only postponed. Backpressure works in chains where the producer can genuinely be paused.

A Volume Spike Does Write to the Buffer

The last row behaves very differently with the same queue at the same capacity. The extra write stream only doubles the arrival rate — 194 events/s, the same as the gentlest resend case — yet the highest backlog rises from 525,000 to 875,000, 1.67 times. The longest wait stretches from 3.00 to 5.00 hours, the drain from 8.73 to 12.55 hours.

Comparing the two events’ arrival rates, the result is unambiguous: a 972-event/s spike adds nothing to the buffer while a 194-event/s stream adds 350,000 events. What fills the buffer is not arrival rate but unprocessed volume. In a design discussion, “how many requests a second arrived” does not determine buffer size; the question that does is how much of the incoming work stayed unprocessed that day.

Where the Limit Is Set

An unbounded queue is not a design; every buffer is finite, and once full it either stops the producer or drops events. Where the limit goes is a measurable question, and two groups of output answer it.

The smallest limit needed is 525,391 on the event-free day, and — independent of R — still 525,391 for the resend event, but 875,488 for the volume event. A buffer sized to the predicted daily profile absorbs a tenfold arrival spike without dropping one event; the same buffer falls short against a volume increase.

The last group counts this directly. With the limit set to 525,000, chosen for the event-free day, the resend event drops 49 events — the result of rounding the limit 391 short of the true requirement, in practice zero. With the same limit, the volume event drops 350,049 events, a number that is the added volume itself.

The rule follows: the buffer limit is calculated from a volume number, not a rate number — from the worst day’s gap between work produced and work processed. What drop behavior to use once the limit is chosen is a separate decision, and its options (blocking the producer, dropping the newest, dropping the oldest) were measured in the Caching, Queues and Asynchronous Processing course.

Back to the Numbers

K01’s table does not have the number this lesson needs, and that absence is itself a finding. V8’s peak factor is 3, making the peak write rate 97.22 events/s; that number is the predicted peak. The arrival rate the acceptance path actually has to meet came out to 194, 389, and 972 events/s in the measurement — two, four, and ten times what V8 produces. If a design sizes its acceptance path to V8, a single twenty-minute outage saturates it.

Processing capacity never changed, though: the 48.61 events/s the first lesson chose is enough even at a tenfold arrival spike. K01’s “peak write 97.22 requests/s” line thus splits into two numbers — the instantaneous arrival peak the acceptance path has to see, and the volume rate the processing path has to meet — and the ratio between them is the queue’s leveling factor.

Summary

• Queue-based load leveling spreads load over time; a load balancer spreads load across replicas, that is, in space. The profile the first lesson leveled was predicted; the spike here is not.
• The 116,667 events arriving after a twenty-minute outage push the arrival rate to 972 events/s, yet the highest backlog stays at 525,000, the wait at 3.00 hours, the drain at 8.73 hours.
• A one-hour extra stream raised the arrival rate to only 194 events/s, yet moved the backlog to 875,000 (1.67 times), the wait to 5.00 hours, the drain to 12.55 hours.
• What fills the buffer is volume, not arrival rate: the smallest queue limit needed came out to 525,391 for the resend event, independent of R, and 875,488 for the volume increase.
• A limit of 525,000, chosen for the predicted profile, drops 49 events at a tenfold arrival spike and 350,049 events at the volume increase.
• K01’s peak write rate splits into two numbers: the instantaneous arrival peak the acceptance path sees can climb to ten times V8’s 97.22, while the volume rate the processing path meets stays at 48.61 events/s.

Next Step

Every queue so far carried one type of work, and consumers took whatever was next without distinguishing it. In the same system, the jobs entering a queue are not each other’s equal: the carrier’s state event shows up on the recipient’s tracking page and its delay is felt directly, the end-of-day billing record can wait until morning, and the seller’s report job can wait longer than either. This lesson’s 525,000-event backlog, treated as a single line, makes all three wait the same three hours. The next lesson turns the line itself into a decision: it measures what separating jobs by service level gains each class, what it costs each class, and what unlimited priority stops completely.