Worker Processes

The previous lesson finished making the message itself durable: attempt counter, exponential backoff, jitter, and the dead-letter queue. A message no longer gets lost; it is retried when it fails, and isolated when it does not recover.

All of those guarantees concerned the message itself. The party that processes the message is still standing in as a nameless “consumer.” Yet that party is a process: it holds memory, it crashes, it restarts, it gets shut down. How many of it there are is a decision, and that decision determines how many jobs the service finishes per hour. This lesson treats the consumer as a unit of scaling and teaches how to measure how many of that unit there should be.

What Is a Worker

A worker is an independent unit of execution that pulls a job from the queue and processes it. It can be a separate process, a separate thread, or a separate container; what they share is that each can crash on its own and be replicated on its own.

Three properties separate a worker from a server handling requests. A worker has no client; nobody is waiting for a response, so the job can take as long as it needs. A worker’s source of work is not a network listener but a queue; it cannot refuse to accept load, only consume it slowly. And a worker scales horizontally: since a second worker feeds from the same queue, adding one requires no coordination.

The jobs moved to the background in the library loan service are well known: sending overdue notifications, generating the monthly loan report, processing an uploaded cover image, recomputing the branch inventory summary. This lesson takes the first as its example.

The Job Itself

A worker process carries a single responsibility: run an incoming job definition and report the result. It does not know where the queue stands, how many times the job has been attempted, or how many workers exist.

// worker.mjs — takes a job, does it, reports the result; pool.mjs forks it
if (!process.send) {
  console.log("this file does not run standalone; pool.mjs forks it");
  process.exit(0);
}

const INDEX = Number(process.argv[2]);            // sequence number the pool assigns to the worker
const wait = (ms) => new Promise((c) => setTimeout(c, ms));

// Sending the overdue notification: an outbound call that takes 100 ms.
async function sendNotification(job) {
  if (job.broken) process.exit(1);                // the job crashes the process itself
  await wait(100);
  return `member ${job.memberId} notified`;
}

process.on("message", async (job) => {
  const result = await sendNotification(job);
  process.send({ type: "done", jobId: job.jobId, index: INDEX, result });
});

process.on("SIGTERM", () => process.exit(0));    // the job is not left half-done when the pool shuts down

this file does not run standalone; pool.mjs forks it

When the file runs directly, process.send is undefined, and the worker says so and exits. The only case where it does anything meaningful is when a parent process forks it.

The 100-millisecond wait inside sendNotification stands in for an outbound call. It marks the job as I/O-bound: the duration passes not by using the processor but by waiting for a response. That distinction will shortly decide the worker-count question.

The Pool and Pull-Based Dispatch

There are two ways to split jobs among workers. In push-based dispatch, the parent process divides the jobs up front: twelve jobs, four workers, three jobs each. This works efficiently only when the jobs take equal time; if one runs long, that worker stays busy while the others sit idle. In pull-based dispatch, a worker asks for the next job as soon as it finishes one. The load balances itself.

The pool below is pull-based: every time dispatch is called it sends the job at the head of the queue, and closes the worker once the queue is empty.

// pool.mjs — forks a fixed number of workers, distributes the job queue to them, measures duration
import { fork } from "node:child_process";

const JOBS = Array.from({ length: 12 }, (_, i) => ({
  jobId: i + 1, loanId: 500 + i, memberId: 100 + i,
}));

function runPool(workerCount, jobs) {
  return new Promise((resolve) => {
    const queue = [...jobs];
    const completed = [];
    const workers = [];
    const start = performance.now();

    const dispatch = (worker) => {
      const job = queue.shift();
      if (job) worker.send(job);
      else worker.kill("SIGTERM");
    };

    for (let s = 1; s <= workerCount; s++) {
      const worker = fork(new URL("worker.mjs", import.meta.url), [String(s)]);
      worker.on("message", (message) => {
        completed.push(message);
        if (completed.length === jobs.length) {
          const duration = performance.now() - start;
          for (const w of workers) w.kill("SIGTERM");
          resolve({ duration, completed });
        } else dispatch(worker);
      });
      workers.push(worker);
      dispatch(worker);
    }
  });
}

for (const n of [1, 2, 4]) {
  const { duration, completed } = await runPool(n, JOBS);
  const slice = Math.floor(duration / 100);   // each job takes 100 ms; slice = 100 ms
  console.log(`worker=${n}  job=${completed.length}  duration=${slice} slices`);
}

worker=1  job=12  duration=12 slices
worker=2  job=12  duration=6 slices
worker=4  job=12  duration=3 slices

The duration is printed in units of slices of 100 milliseconds, the length of a single job; this makes the result readable independently of the machine’s speed. The slice count is rounded down, so process-startup overhead does not show up in the number. On a slower machine the slice counts may shift up by one.

What matters is the ratio: twelve jobs take twelve slices with one worker, six with two workers, three with four workers. Doubling the worker count halves the duration. This is the direct consequence of the job being divisible: sending a notification to one member does not wait on sending a notification to another.

The Limit of Scaling

There are two points where the ratio stops holding forever.

The first is the type of job. The job above was I/O-bound: the worker spent most of its time waiting for a response, so the payoff continued even past the number of cores. If the job were CPU-bound — resizing the cover image, for instance — the payoff would stop at the core count, and adding workers past that point would only produce context- switching overhead.

The second is a shared resource. If twelve workers draw from the same database connection pool, the bottleneck is the connection count, not the worker count. If sending notifications goes to a single external service, the bottleneck is the concurrency that service accepts. In both cases, adding workers does not shorten the total duration; it only spreads the waiting across more workers.

This is why the worker count is not a guess but a measurement result: the count is raised, the total duration is measured, and the process stops where the duration stops falling.

A Crashed Worker and Supervision

A worker can crash because it is an independent process: memory runs out, the operating system kills it, the job it is processing brings the process itself down. The pool has two responsibilities in response. Replacing the crashed worker, and not losing the job it was holding.

The second is the critical one. A worker reports nothing while it crashes; the pool only sees the exit event. If which job the worker was carrying is not known at that point, the job silently disappears. That is why the pool keeps every worker’s current job in a separate map.

// resilient.mjs — when a worker crashes it is replaced, and the job it held is put back in the queue
import { fork } from "node:child_process";

const JOBS = [
  { jobId: 1, loanId: 500, memberId: 100 },
  { jobId: 2, loanId: 501, memberId: 101 },
  { jobId: 3, loanId: 502, memberId: 102, broken: true },
  { jobId: 4, loanId: 503, memberId: 103 },
  { jobId: 5, loanId: 504, memberId: 104 },
];
const MAX_ATTEMPTS = 2;

const queue = [...JOBS];
const attempt = new Map();                       // jobId -> how many times attempted
const completed = [], deadLetter = [];
const inFlight = new Map();                      // worker -> the job it is currently processing
let open = 0, nextIndex = 1;

function spawnWorker() {
  const worker = fork(new URL("worker.mjs", import.meta.url), [String(nextIndex++)]);
  open++;
  worker.on("message", (m) => { inFlight.delete(worker); completed.push(m.jobId); dispatch(worker); });
  worker.on("exit", () => {
    open--;
    const job = inFlight.get(worker);             // set: the worker crashed while working the job
    if (job) {
      inFlight.delete(worker);
      if (attempt.get(job.jobId) < MAX_ATTEMPTS) queue.push(job);
      else deadLetter.push(job.jobId);
      spawnWorker();                              // replaces the crashed worker
    }
    if (open === 0) report();
  });
  dispatch(worker);
}

function dispatch(worker) {
  const job = queue.shift();
  if (!job) return worker.kill("SIGTERM");
  attempt.set(job.jobId, (attempt.get(job.jobId) ?? 0) + 1);
  inFlight.set(worker, job);
  worker.send(job);
}

function report() {
  console.log("completed jobs    :", completed.sort((a, b) => a - b).join(", "));
  console.log("dead-letter queue :", deadLetter.join(", ") || "-");
  console.log("workers spawned   :", nextIndex - 1);
  console.log("job 3 attempts    :", attempt.get(3));
}

spawnWorker();
spawnWorker();

completed jobs    : 1, 2, 4, 5
dead-letter queue : 3
workers spawned   : 4
job 3 attempts    : 2

Job three crashes its worker. On the exit event the pool puts the job back in the queue and spawns a replacement for the crashed worker. The second attempt gives the same result, so the job crosses the attempt limit and goes to the dead-letter queue introduced in the previous lesson. The other four jobs are unaffected; four workers are spawned in total, two at the start and two replacing the crashed ones.

The name for this pattern is supervision: the unit doing the work is separated from the unit managing its lifetime. A worker does not try to repair its own failure, it crashes; the pool repairs it. This separation noticeably simplifies the worker’s code.

Supervision has a hidden cost. A job put back in the queue may have been partially done at the moment it crashed: the notification may have been sent but not recorded as sent. This is why every retried job must be protected by the idempotency key introduced in the Web API Design curriculum; otherwise supervision buys durability at the cost of sending the member two notifications.

Graceful Shutdown

The pool closes a worker with SIGTERM. The worker catches this signal and exits. If the signal goes unhandled, the process terminates immediately under the default behavior, and the job it was holding is left half-done.

Graceful shutdown is three steps: stop taking new jobs, finish the one in hand, then exit. In practice the parent process first sends SIGTERM, grants the worker a bounded window to finish, and forces the issue with SIGKILL if the window runs out. This second signal cannot be caught by the worker; that is why the window granted must be larger than the longest job.

This is also the order deployment tools follow when stopping worker containers. A worker that does not catch the signal produces a half-done job on every redeployment.

Summary

• A worker is an independent unit of execution that pulls a job from the queue and processes it; being able to crash on its own and be replicated on its own makes it a unit of scaling.
• In pull-based dispatch, a worker asks for the next job as soon as it finishes one; the load balances itself even when job durations are unequal.
• The payoff from worker count is found by measurement: it can go past the core count for an I/O-bound job, it stops there for a CPU-bound job, and it stops much earlier when a shared resource is involved.
• Supervision replaces a crashed worker and puts the job it was holding back in the queue; the cost is that the job may have been partially done, which is why a retried job is protected with an idempotency key.
• Graceful shutdown means stop taking new jobs, finish the one in hand, then exit; if the window granted is shorter than the longest job, every shutdown produces a half-done job.

Next Step

In this lesson, jobs came from outside: the pool distributed a list it already held. Most background jobs, however, do not come from outside — they come from the calendar. The overdue notification runs every morning at six, the monthly loan report on the first of the month, the branch inventory summary every fifteen minutes. Binding a job to a time rule looks like a one-line definition, but underneath it are two questions: how is recurrence defined, and what happens when two runs of the same task overlap. The next lesson builds the recurring task definition and prevents overlap by measuring it.