Task Queues and Background Jobs

Every event in the previous lesson was the equal of every other: each state event was a record of the same size, doing the same work, lasting milliseconds. The backlog calculation rested on that too — capacity can be expressed as a single number because the jobs are all one type. In the same system there is a job that breaks this assumption: the period report a seller requests scans thousands of shipment records and takes seconds. Run inside a request, such a job does not just spend its own time — it also holds up the short requests behind it.

This lesson takes that job off the request path. Task queues use the same mechanics as message queues; what differs is content — what gets written to the queue is not an event notification but a job someone wants done. Worker processes, pull-based dispatch, supervision, graceful shutdown, and recording a long job’s progress in pieces were built in the Caching, Queues and Asynchronous Processing course; none of it is retold here. What is measured here is only the separation’s capacity payoff: how much the short request’s wait improves, and what the design’s scaling unit becomes.

Where the Job’s Length Comes From

The long job’s length is not invented; it follows from K01’s numbers. The Back-of-the-Envelope Estimation lesson found 4,000 invoice lines a day, each line one seller’s one day, so the seller count is 4,000. With 400,000 daily shipments, that is 100 shipments a day per seller. Per V11 the pricing period is 30 days, so a period report scans 3,000 shipment records. The same lesson gives the scan rate too: 833.33 records/s, or 1.20 ms per record — the report job takes 3.60 seconds. All these numbers belong to the computed-value class.

The short job being compared is the tracking query, reading a single record — one scan unit. The ratio between them is directly 3,000. The model’s tick unit follows from this: a tick is 1.20 ms, and no duration inside the model is measured — each is converted from ticks.

Two assumptions are needed, and both are this course’s own; neither is added to K01’s table.

K2 — morning window: 30 minutes. Rationale: K01’s V10 row says “the seller wants the report in the morning”; report requests bunch up at the start of the business day.

K3 — share of sellers requesting a report in the window: 0.50. Rationale: period reports are requested in bulk on certain days of the month, when half the sellers are assumed to check in. Sensitivity is measured below in the 0.25 rows.

The read load comes from K01 and K02: peak reads of 416.67 requests/s, replica count 4 — the Traffic Layer course’s Session Stickiness lesson raised that number from 3 to 4. Reads per replica come to 104.17 requests/s.

Two Jobs in the Same Pool

The setup builds a single replica as an in-process model. The replica has a fixed number of slots; each slot holds one request at a time, and once free takes the next from the front of the queue. Arrivals come from two independent fixed-seed generators, so every design faces the same request sequence. Being a model, the results are deterministic and machine-independent.

// task/pool.mjs — comparing a long job staying on the request path against moving it into a
// task queue. MODEL: tick = 1.20 ms (K01: bulk job scan 833.33 records/s), arrivals come from a
// fixed-seed generator. Results are deterministic; duration is not measured, it is converted
// from ticks.
const TICK_MS = 1.2;               // K01: 1.20 ms per record
const READ_PEAK = 416.67;          // K01: peak read requests/s
const REPLICAS = 4;                // K02 session stickiness lesson: 4 replicas
const SELLERS = 4000;              // K01: daily invoice lines = number of sellers
const REPORT_RECORDS = 3000;       // K01: 100 shipments/day per seller x V11 = 30 days
const K2_WINDOW_MIN = 30;          // this course's assumption K2: morning window
const K3_RATIO = 0.5;              // this course's assumption K3: share of sellers requesting a report in the window
const SEED = 20260730;

const TICKS = Math.round((K2_WINDOW_MIN * 60 * 1000) / TICK_MS);
const trackingP = (READ_PEAK / REPLICAS) * (TICK_MS / 1000);
const reportCount = (ratio) => (SELLERS * ratio) / REPLICAS;

function generator(seed) {         // splitmix32: independent arrival sequences for two streams
  let a = seed >>> 0;
  return () => {
    a = (a + 0x9e3779b9) >>> 0;
    let z = Math.imul(a ^ (a >>> 16), 0x21f0aaad) >>> 0;
    z = Math.imul(z ^ (z >>> 15), 0x735a2d97) >>> 0;
    return ((z ^ (z >>> 15)) >>> 0) / 4294967296;
  };
}

function run({ slots, separate, workers, ratio = K3_RATIO }) {
  const reportP = reportCount(ratio) / TICKS;
  const randTracking = generator(SEED), randReport = generator(SEED + 1);
  const busy = new Array(slots).fill(0), reportBusy = new Array(workers).fill(0);
  const queue = [], tasks = [], wait = [], reportDuration = [];
  let reportsArrived = 0;
  for (let t = 0; t < TICKS; t += 1) {
    if (randTracking() < trackingP) queue.push({ t, duration: 1 });
    if (randReport() < reportP) { reportsArrived += 1; (separate ? tasks : queue).push({ t, duration: REPORT_RECORDS, report: true }); }
    for (let i = 0; i < slots; i += 1) {
      if (busy[i] > 0) busy[i] -= 1;
      if (busy[i] === 0 && queue.length > 0) {
        const job = queue.shift();
        if (job.report) reportDuration.push(t - job.t + job.duration); else wait.push(t - job.t);
        busy[i] = job.duration;
      }
    }
    for (let i = 0; i < workers; i += 1) {
      if (reportBusy[i] > 0) reportBusy[i] -= 1;
      if (reportBusy[i] === 0 && tasks.length > 0) {
        const job = tasks.shift();
        reportDuration.push(t - job.t + job.duration);
        reportBusy[i] = job.duration;
      }
    }
  }
  const sorted = [...wait].sort((a, b) => a - b);
  const percentile = (p) => sorted[Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length))];
  return { tracking: wait.length, p50: percentile(50), p99: percentile(99), longest: sorted[sorted.length - 1],
    late: wait.filter((w) => w * TICK_MS > 100).length,
    reportsArrived, completed: reportDuration.length,
    reportAvg: reportDuration.reduce((a, b) => a + b, 0) / reportDuration.length };
}

console.log(`tick = ${TICK_MS} ms, window ${K2_WINDOW_MIN} min = ${TICKS} ticks, ${REPLICAS} replicas`);
console.log(`per replica tracking ${(READ_PEAK / REPLICAS).toFixed(2)} requests/s, ` +
  `K3 = ${K3_RATIO} -> expected reports per replica ${reportCount(K3_RATIO)}`);
console.log(`report job = ${REPORT_RECORDS} records = ${(REPORT_RECORDS * TICK_MS / 1000).toFixed(2)} s\n`);

const row = (name, r) => console.log(`${name.padEnd(24)} ${String(r.tracking).padStart(7)} ` +
  `${(r.p50 * TICK_MS).toFixed(2).padStart(9)} ${(r.p99 * TICK_MS).toFixed(2).padStart(9)} ` +
  `${(r.longest * TICK_MS).toFixed(0).padStart(11)} ${String(r.late).padStart(9)} ` +
  `${(r.late / r.tracking).toFixed(4).padStart(6)} ${`${r.completed}/${r.reportsArrived}`.padStart(9)} ` +
  `${(r.reportAvg * TICK_MS / 1000).toFixed(2).padStart(9)}`);

console.log("design                    tracking  p50 (ms)  p99 (ms)  longest(ms)  >100 ms     rate  done/arrived  report (s)");
for (const slots of [2, 4, 8]) row(`shared s=${slots}`, run({ slots, separate: false, workers: 0 }));
for (const slots of [2, 4]) row(`shared s=${slots}, K3=0.25`, run({ slots, separate: false, workers: 0, ratio: 0.25 }));
for (const slots of [2, 4]) row(`separate s=${slots}, w=2`, run({ slots, separate: true, workers: 2 }));

const fewest = (make) => { for (let s = 1; s <= 12; s += 1) if (make(s).late === 0) return s; return 0; };
const sharedSlots = fewest((slots) => run({ slots, separate: false, workers: 0 }));
const separateSlots = fewest((slots) => run({ slots, separate: true, workers: 2 }));
const reportSeconds = reportCount(K3_RATIO) * REPLICAS * (REPORT_RECORDS * TICK_MS / 1000);
const workers = reportSeconds / (K2_WINDOW_MIN * 60);
console.log(`\nfewest slots that zero out tracking wait above 100 ms: shared ${sharedSlots}, separate ${separateSlots}`);
console.log(`scaling unit -> shared ${sharedSlots * REPLICAS} slots (all on the read path), ` +
  `separate ${separateSlots * REPLICAS} slots + task workers`);
console.log(`system-wide report work = ${reportSeconds.toFixed(0)} worker-seconds / ` +
  `${K2_WINDOW_MIN * 60} s = ${workers.toFixed(2)} workers (utilization 1.00)`);

tick = 1.2 ms, window 30 min = 1500000 ticks, 4 replicas
per replica tracking 104.17 requests/s, K3 = 0.5 -> expected reports per replica 500
report job = 3000 records = 3.60 s

design                    tracking  p50 (ms)  p99 (ms)  longest(ms)  >100 ms     rate  done/arrived  report (s)
shared s=2                187459      0.00   5887.20        9398     54392 0.2902   457/457      4.24
shared s=4                187459      0.00    634.80        2808      3088 0.0165   457/457      3.61
shared s=8                187459      0.00      0.00           0         0 0.0000   457/457      3.60
shared s=2, K3=0.25       187459      0.00   2766.00        4984     15204 0.0811   223/223      3.77
shared s=4, K3=0.25       187459      0.00      0.00        1240       293 0.0016   223/223      3.60
separate s=2, w=2         187459      0.00      0.00           0         0 0.0000   457/457      4.14
separate s=4, w=2         187459      0.00      0.00           0         0 0.0000   457/457      4.14

fewest slots that zero out tracking wait above 100 ms: shared 7, separate 1
scaling unit -> shared 28 slots (all on the read path), separate 4 slots + task workers
system-wide report work = 7200 worker-seconds / 1800 s = 4.00 workers (utilization 1.00)

In thirty minutes, 187,459 tracking requests arrived, and against an expected 500 reports the seed produced 457 — one realization of the Bernoulli arrival, the same across every row.

Low Utilization, Long Queue

The shared pool’s numbers look inconsistent at first glance. In a two-slot replica, report work uses less than half the slots: 457 reports × 3.60 seconds = 1,645 slot-seconds, 46 percent of the 3,600 slot-seconds two slots provide over a thirty-minute window. The tracking stream wants only 12.5 percent. Total utilization is under 60 percent, so the pool has enough capacity.

Yet 29.02 percent of tracking requests wait longer than 100 ms, p99 wait is 5887 ms, the longest wait 9398 ms. Why sufficient capacity can still produce this table is clear: the queue drains in order, and if the job ahead takes 3,000 ticks, a one-tick job behind it waits 3,000 ticks too. When two reports run at once, both slots close, and every tracking request arriving during that span joins the line. A queue can grow long while average utilization stays low because of how job durations are distributed; a single utilization number hides that distribution.

Adding slots fixes it, but expensively. At four slots the rate drops to 1.65 percent, p99 to 634.80 ms; at eight it hits zero. The threshold is measured: the fewest slots that zero out waits above 100 ms is seven. The short request alone needs one slot — that is the number the separate design measures. So six slots exist purely to keep it from queuing behind the long job.

K3’s sensitivity points the same way. Halving the share of sellers requesting a report drops the affected-request rate from 29.02 to 8.11 percent at two slots, and from 1.65 to 0.16 percent at four — but it does not reach zero. Lower demand reduces how often the problem occurs, not its structure; even one long job holds up the request behind it for its own duration.

The Scaling Unit

In the separate design, the report job goes into the queue and the caller returns with a job ID. Tracking measurements come out the same at every slot count: p99 zero, longest wait zero, nothing waiting above 100 ms. Pull the long job off the path, and the short request’s queue disappears with it.

The cost falls on the report side, and it is measured: with two workers, the average report duration is 4.14 seconds instead of 3.60 — it now waits in its own queue. The gap is 15 percent and closes by adding workers if wanted, because worker count is now independent of the tracking stream.

The real result is in the last three lines. In the shared design, zeroing out the tracking queue needs seven slots per replica — 28 slots across four replicas. All of them sit on the read path, each configured to accept tracking requests in the replicas where read traffic is distributed. In the separate design, one slot per replica suffices for tracking — 4 slots across four replicas — and the report job moves to its own workforce. That workforce’s size is arithmetic too: system-wide, 2,000 reports × 3.60 seconds = 7,200 worker-seconds, which over the thirty-minute window comes to 4.00 workers — utilization of 1.00, so a real design would leave some margin.

Separation’s gain is not a number, it is independence. In the shared design, growing report demand means growing the replica, and a bigger replica also grows the read path, though the read path never needed that capacity. In the separate design, report demand moves worker count, read demand moves replica count, and the two never look at each other. The peak read rate from the introductory course, 416.67 requests/s, has not changed; what changed is how large the replicas serving that rate have to be — from seven slots per replica down to one.

Summary

• The long job’s length is derived from K01: 100 shipments/day per seller × 30 days = 3,000 records, which at a scan rate of 833.33 records/s takes 3.60 seconds; the tracking request is, on that same scale, a single record.
• In the shared pool, even with utilization under 60 percent, 29.02 percent of tracking requests wait longer than 100 ms, with a p99 of 5887 ms: a single utilization number hides the distribution of job durations.
• The cost of zeroing out the queue is slots: the fewest slots that eliminate waits above 100 ms is seven in the shared design, one in the separate design.
• Halving the K3 assumption drops the affected-request rate from 29.02 to 8.11 percent, but it does not reach zero; demand changes how often the problem occurs, not its structure.
• In the separate design, tracking wait is zero at every slot count; the cost paid is the report duration rising from 3.60 to 4.14 seconds with two workers.
• The scaling unit splits apart: the shared design needs 28 slots across four replicas, the separate design needs 4 slots plus 4.00 workers system-wide, and the two numbers move independently of each other.

Next Step

The “4.00 workers” in the last line is not a capacity but a lower bound; at utilization of exactly one, the queue never drains. The number itself leaves a question open: does adding a worker really translate into addition? The previous lesson’s backlog table treated consumer capacity as a single number and never asked where it came from — yet capacity is the product of consumer count and per-consumer rate, and one of those factors does not grow forever. The next lesson takes up multiple consumers pulling from the same queue: it measures each added consumer’s contribution to drain time, where that contribution stops, and what breaks in exchange for the throughput gained.