Process Management and Resilience

So far, the process has always started by hand and stopped by closing the terminal. In production this job belongs to a supervisor: it starts the process, reopens it when it crashes, stops it during an update.

There’s a contract between supervisor and process, and this lesson covers the process’s side of it: catching the shutdown request, finishing running work without taking new work, and reporting its own state.

A Shutdown Request Is a Signal

Signals and traps were covered in the Shell Programming course; the same mechanism appears here as the process object’s events.

The supervisor’s shutdown request is the SIGTERM signal. Pressing the interrupt key in a terminal produces SIGINT. Both default to ending the process immediately; once a listener attaches, that default is disabled and the decision passes to the program.

There’s also a signal that cannot be caught. The supervisor uses it if the process does not shut down within the allowed time after the polite request, and the process ends without any cleanup. This is why the shutdown needs its own time limit: one that does not finish before the supervisor’s limit counts as never having happened.

The Phases of Graceful Shutdown

Graceful shutdown is the process stopping taking new work and ending by finishing the work it already has. It has four phases, and their order matters.

// resilient-server.mjs
import { createServer } from 'node:http';

const EXIT_DELAY_MS = 500;      // time to tell the supervisor "no longer ready"
const SHUTDOWN_LIMIT_MS = 3000; // longest time given to pending requests

let ready = false;
let shuttingDown = false;

const server = createServer((request, response) => {
  if (request.url === '/live') {                 // is the process up
    response.writeHead(200).end('alive\n');
    return;
  }
  if (request.url === '/ready') {                 // ready to accept requests
    const code = ready && !shuttingDown ? 200 : 503;
    response.writeHead(code).end(code === 200 ? 'ready\n' : 'not ready\n');
    return;
  }
  if (request.url === '/slow') {                 // simulates a pending request
    setTimeout(() => response.writeHead(200).end('done\n'), 800);
    return;
  }
  response.writeHead(404).end('not found\n');
});

server.listen(8791, '127.0.0.1', () => {
  ready = true;
  console.log(`listening, process id ${process.pid}`);
});

function shutdown(signal) {
  if (shuttingDown) return;
  shuttingDown = true;
  console.log(`${signal} received; readiness probe now returns 503`);

  setTimeout(() => {
    console.log('no longer accepting new connections');

    const forced = setTimeout(() => {
      console.log('time is up; cutting remaining connections');
      server.closeAllConnections();
    }, SHUTDOWN_LIMIT_MS);
    forced.unref();                           // must not keep the process alive on its own

    server.close(() => {
      clearTimeout(forced);
      console.log('all requests completed; exiting');
    });
    server.closeIdleConnections();
  }, EXIT_DELAY_MS).unref();
}

process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));

The trial is run with a script:

#!/bin/sh
# shutdown-test.sh
node resilient-server.mjs &
serverPid=$!
sleep 1

printf 'before shutdown /ready  -> '; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8791/ready

# Start a pending request, send the signal while waiting for its response
curl -s http://127.0.0.1:8791/slow > slow-response.txt &
requestPid=$!
sleep 0.1
kill -TERM "$serverPid"

sleep 0.2
printf 'during shutdown /ready -> '; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8791/ready

wait "$requestPid"
echo "response of the pending request: $(cat slow-response.txt)"
wait "$serverPid"
echo "process exit code: $?"

sh shutdown-test.sh

listening, process id 18312
before shutdown /ready  -> 200
SIGTERM received; readiness probe now returns 503
during shutdown /ready -> 503
no longer accepting new connections
all requests completed; exiting
response of the pending request: done
process exit code: 0

The process id changes on every run. Here’s the order the output tells.

Phase one: report not ready. The moment the signal arrives, the readiness probe starts returning 503. A load balancer in front of the process needs a window to take it out of the list; without it, the balancer keeps sending requests to a process that is shutting down, and those requests get dropped.

Phase two: stop taking new connections. When the delay expires, close gets called. This closes the listening socket but does not drop existing connections. Idle persistent connections are closed separately; otherwise a client holding a connection open without sending a request would stretch out the shutdown.

Phase three: finish running work. The close callback runs once the last request completes too. In the output above, the pending request gets its done response — the signal had been sent while that request was being processed.

Phase four: the time limit. If a client never releases its connection, the shutdown runs forever. The forcing timer cuts that off. The unref call matters: this timer must not be the sole reason the process stays alive, or it waits until the limit runs out even after everything else is done.

The exit code is zero. To the supervisor, this means “shut down on request,” enabling different behavior depending on the restart policy.

There’s other work to do at shutdown too: flushing pending log lines, closing open files, writing the remaining summary to disk — all inside the close callback, before the process ends. As seen in the Process Object lesson, asynchronous work cannot be done in the exit hook.

Liveness and Readiness

The two probes answer two separate questions, and mixing them up destabilizes the service.

The liveness question: is the process alive, or should it be killed and restarted? Its answer looks only at the process’s own state; it does not check its dependencies.

The readiness question: can this process accept requests right now? Its answer looks at transient conditions — has startup finished, has shutdown begun.

The reasoning: when a dependency is temporarily unreachable, the process is not ready but it is alive. If the liveness probe checked the dependency, a brief outage there would restart every process — behavior that both stretches out the outage and produces a failure of its own.

Probes have to be lightweight: if every check runs a database query, the check itself turns into load.

The Restart Contract

Restarting when the process dies is not the process’s job; it cannot restart itself. That job belongs to the operating system’s service supervisor, or to the primary process from the Cluster and Worker Threads lesson.

Three things fall to the process.

Giving a meaningful exit code. Zero means a requested shutdown, nonzero means a failure. This is why a separate code was chosen for a configuration error in the Configuration Management lesson: the supervisor can stop instead of endlessly retrying a failure that clearly will not fix itself.

Starting up fast and idempotently. If restarting is a recovery method, startup itself has to be reliable. Work done at startup must not fail when it finds traces of a previous run left half-finished — the write-to-temporary-name-and-replace pattern from the File System lesson provides that guarantee.

Accounting for the restart storm. Reopening a process that crashes immediately produces hundreds of startups a second and keeps the machine busy. Supervisors prevent this with increasing delay; the process must not shorten that delay, for instance by reporting a fake success while unhealthy.

A process trying to restart itself is a separate mistake: the way out of a corrupted state is abandoning the memory carrying that state — also why restarting works at all, the reasoning behind limiting the last-resort hook in the Error Handling Strategy lesson to “record and exit.”

Summary

• A shutdown request is a signal; once a listener is attached, the default termination behavior gives way to the program’s decision.
• Graceful shutdown has four phases: report not ready, stop taking new connections, finish running work, cut what’s left at the time limit.
• The forcing timer is marked with unref; otherwise the process stays alive until the limit runs out even after everything else is done.
• The liveness probe looks only at the process’s own state, the readiness probe at transient state; a liveness probe checking a dependency spreads outages.
• Restarting is the supervisor’s job; what falls to the process is giving a meaningful exit code, starting up fast and idempotently, and not feeding the storm.

Next Step

The measurement collector is up and running: configured, classifying its errors, writing its log in structured form, tested, observable, and shutting down cleanly. One question remains — how does its reusable part get carried into other projects? The final lesson takes up the packaging contract: declaring entry points, hiding internal files, and the rules of publishing a version.