The Application Runtime

The previous lesson ran the application server behind the proxy as a single process and left its question open: what happens to the other requests when one takes a long time, how do two copies of the same application share the same port, and is the unit of a scaling decision the process or the thread? This lesson answers those three questions by measuring them.

The measurement uses two endpoints of the library loan service: an expensive endpoint that verifies a member’s password, and a cheap endpoint that returns a single line from the catalog. When the two live in the same process, what the expensive work does to the cheap requests becomes visible through the delay between them.

What the Runtime Provides

The runtime is the layer application code sits on top of: it houses the engine that executes the code, the doors that open onto the operating system’s socket and file interfaces, and the event loop that queues the work between the two. The model established in The Node.js Runtime course holds here just as it did there; what is new in this lesson is how that same model determines a server application’s scaling decisions.

The model comes down to one sentence: application code runs on a single thread. While one request handler is running, no other handler runs. The event loop picks up the next queued work once a handler finishes. Input/output waits — disk reads, network calls, database queries — do not occupy this queue; the runtime hands the wait off to the operating system and queues the callback when the result arrives.

This model has two consequences. Waiting work is cheap: thousands of open connections can be held in a single process, because each one is just a socket and a callback. By contrast, computation-heavy work is expensive: a computation that occupies the processor locks the queue for its entire duration.

Blocking Work: How Long One Request Makes Others Wait

The server below performs member password verification in two separate ways. verify-blocking calls key derivation directly and holds the event loop while waiting for the result. verify-pooled hands the same computation to the runtime’s thread pool and releases the loop. The computation itself is the same in both.

// single-process.mjs — loan service in a single process; the same work done both blocking and non-blocking
import { createServer } from "node:http";
import { scrypt, scryptSync } from "node:crypto";

const SHELVES = [{ isbn: "978-0262033848", title: "Introduction to Algorithms", shelf: "R-12" }];

// Key derivation cost: high enough to produce over half a second of processor work.
const COST = { N: 1 << 16, r: 8, p: 8, maxmem: 512 * 1024 * 1024 };

createServer((req, res) => {
  res.sendDate = false;
  res.setHeader("content-type", "application/json; charset=utf-8");
  const path = new URL(req.url, "http://local").pathname;

  if (path === "/member/verify-blocking") {
    const digest = scryptSync("password", "salt", 32, COST).toString("hex").slice(0, 16);
    return res.writeHead(200).end(JSON.stringify({ digest, path }));
  }
  if (path === "/member/verify-pooled") {
    return scrypt("password", "salt", 32, COST, (err, key) => {
      if (err) return res.writeHead(500).end(JSON.stringify({ error: "derivation" }));
      res.writeHead(200).end(JSON.stringify({ digest: key.toString("hex").slice(0, 16), path }));
    });
  }
  if (path === "/books") {
    return res.writeHead(200).end(JSON.stringify({ books: SHELVES }));
  }
  res.writeHead(404).end(JSON.stringify({ error: "route_not_found" }));
}).listen(8430, "127.0.0.1", () => console.log("single process 127.0.0.1:8430 pid=" + process.pid));

The measurement follows a set pattern: the heavy request is sent first, and fifty milliseconds later four light requests follow it. The light requests have almost no work of their own; their durations come only from waiting their turn.

#!/usr/bin/env bash
# Starts single-process.mjs; measures the duration of four light requests arriving alongside a heavy request.
node single-process.mjs & server=$!
sleep 0.6
curl -sS -o /dev/null http://127.0.0.1:8430/books          # warm-up

measure() {  # $1 = path of the heavy endpoint
  pids=()
  curl -sS -o /dev/null -w "heavy    %{time_total} s\n" "http://127.0.0.1:8430$1" > heavy.txt &
  pids+=($!)
  sleep 0.05
  for i in 1 2 3 4; do
    curl -sS -o /dev/null -w "light-$i  %{time_total} s\n" http://127.0.0.1:8430/books > "light-$i.txt" &
    pids+=($!)
  done
  wait "${pids[@]}"
  cat heavy.txt light-1.txt light-2.txt light-3.txt light-4.txt
}

echo "--- heavy work on the event loop: /member/verify-blocking ---"
measure /member/verify-blocking
echo "--- heavy work on the thread pool: /member/verify-pooled ---"
measure /member/verify-pooled

kill "$server"
rm -f heavy.txt light-*.txt

single process 127.0.0.1:8430 pid=14674
--- heavy work on the event loop: /member/verify-blocking ---
heavy    0.635000 s
light-1  0.581384 s
light-2  0.581601 s
light-3  0.580992 s
light-4  0.581166 s
--- heavy work on the thread pool: /member/verify-pooled ---
heavy    0.618827 s
light-1  0.001071 s
light-2  0.000954 s
light-3  0.000756 s
light-4  0.001232 s

The process id and the durations change on every run; the heavy work’s absolute duration also depends on the machine’s processor. What stays fixed is the order-of-magnitude difference between the light requests’ durations in the two blocks.

In the first measurement, the light requests waited about 580 milliseconds. Their own work takes less than a millisecond; what they waited for was the heavy request’s remaining duration. All four finish at nearly the same moment, because all of them waited for the same block to lift, and once it lifted, they were processed one after another.

In the second measurement, the heavy request’s duration did not change — it is the same computation — but the light requests were answered in under one millisecond. The only difference is where the computation happens. In the pooled version, the event loop handed the computation to a thread and freed itself, served the requests arriving in between at their normal speed, and ran the callback once the result was ready.

The Pool Is Limited Too

The thread pool is not the solution to blocking work; it is where that work moves to. The pool’s thread count is fixed, and its default value is small. If more work is handed to the pool at once than its capacity, the excess queues up; this time it is not the event loop that saturates, but the pool. Because the single heavy request in the measurement did not fill the pool, no delay appeared.

This gives the first rule of scaling: a process’s processor capacity is fixed. Moving the computation from the loop to the pool reduces how much requests wait on each other, but it does not increase the process’s total processor work. The number of key derivations that can be done per second is bounded by the machine’s core count, independent of which thread does the work.

The Unit of Scaling: The Process

A single process can saturate a single core. On a multi-core machine, the way to use the remaining cores is to run several copies of the application. In a cluster setup, a primary process opens the listening socket and forks several worker processes; incoming connections are distributed among the workers. The workers share the same port, because the party that opened the socket is the primary process.

// cluster.mjs — three worker processes sharing the same port; each process has its own counter
import cluster from "node:cluster";
import { createServer } from "node:http";

const WORKER_COUNT = 3;

if (cluster.isPrimary) {
  let shuttingDown = false;
  for (let i = 0; i < WORKER_COUNT; i++) cluster.fork();

  cluster.on("exit", (worker) => {                        // a new worker replaces the one that died
    if (shuttingDown) return;
    console.log(`worker died pid=${worker.process.pid}, forking a replacement`);
    cluster.fork();
  });

  process.on("SIGTERM", () => {                         // workers go down together with the primary
    shuttingDown = true;
    for (const worker of Object.values(cluster.workers)) worker.kill();
    process.exit(0);
  });

  console.log(`primary pid=${process.pid}, worker count=${WORKER_COUNT}`);
} else {
  let loanCounter = 0;                                  // PROCESS-SPECIFIC memory: separate in each worker
  createServer((req, res) => {
    res.sendDate = false;
    res.setHeader("content-type", "application/json; charset=utf-8");
    if (new URL(req.url, "http://local").pathname === "/loan") loanCounter++;
    res.writeHead(200).end(JSON.stringify({ pid: process.pid, loanCounter }));
  }).listen(8431, "127.0.0.1");
}

The server answers every path and reports its own process id in the response; only the /loan path increments the counter. This distinction makes it possible to poll the distribution without changing the counter value.

#!/usr/bin/env bash
# Starts cluster.mjs; shows the distribution of 12 requests among the workers, the counters
# diverging, and a replacement arriving for a worker that died.
node cluster.mjs & cluster=$!
sleep 0.8

echo "--- 12 loan requests: which worker answered how many times ---"
for i in $(seq 12); do curl -sS http://127.0.0.1:8431/loan; echo; done |
  grep -o '"pid":[0-9]*' | sort | uniq -c

echo "--- four requests to the same endpoint: each worker's counter lives in its own memory ---"
for i in 1 2 3 4; do curl -sS http://127.0.0.1:8431/loan; echo; done

echo "--- a worker is killed, a replacement arrives ---"
victim=$(curl -sS http://127.0.0.1:8431/status | sed -n 's/.*"pid":\([0-9]*\).*/\1/p')
echo "killed worker pid=$victim"
kill -9 "$victim"
sleep 0.8
for i in $(seq 6); do curl -sS http://127.0.0.1:8431/status; echo; done |
  grep -o '"pid":[0-9]*' | sort -u

kill "$cluster"

primary pid=3068, worker count=3
--- 12 loan requests: which worker answered how many times ---
   4 "pid":3070
   4 "pid":3071
   4 "pid":3072
--- four requests to the same endpoint: each worker's counter lives in its own memory ---
{"pid":3072,"loanCounter":5}
{"pid":3070,"loanCounter":5}
{"pid":3071,"loanCounter":5}
{"pid":3072,"loanCounter":6}
--- a worker is killed, a replacement arrives ---
killed worker pid=3070
worker died pid=3070, forking a replacement
"pid":3071
"pid":3072
"pid":3098

The process ids are different on every run. What stays fixed are three observations.

The distribution is even. Twelve requests landed four apiece on three workers; the primary process distributed the connections in turn. Nothing changed on the client side: the request went to the same address, the same port. How many processes are running is invisible to the client.

A replacement arrives for the worker that died. When a worker is forcibly terminated, the primary process sees it and forks a new worker; the final list contains none of the old ids, but a new id in their place. This also shows that a process is a unit of isolation as well as a unit of scaling: one worker crashing does not affect the others, and the service does not go down.

Process Memory Is the Scaling Boundary

The most instructive lines in the output are the counter lines. After twelve requests, every worker’s counter reads four; once the next four requests are distributed, the values read 5, 5, 5, and 6. No worker knows the total count, because the counter lives in process memory, and processes do not share memory.

In the library application, the counterpart of this is direct. Loan count, session information, a temporary reservation list, a rate-limit counter — all of them break the moment they are kept in process memory and scaled. The request in which a member logs in lands on one worker, their next request lands on another, and the second worker does not recognize the session.

The consequence of this is the binding rule the runtime model places on application design: the process must be stateless. Every value that needs to survive between requests moves out of the process, into a shared store. The process keeps in its own memory only the values that live for the duration of that one request. Once this rule is met, increasing the process count comes down to a configuration decision; when it is not met, starting a second process leads to errors.

The same rule also follows from the process being mortal. A dead worker’s memory goes with it; the worker that replaces it starts with empty memory. A value kept in process memory is lost not only during scaling, but on every restart.

Summary

• Application code runs on a single thread; waiting input/output is cheap, while a computation that occupies the processor locks the queue for its entire duration.
• In the measurement, a half-second computation that blocked the event loop made four light requests arriving alongside it wait about 580 milliseconds; once the same computation was handed to the thread pool, the light requests were answered in under one millisecond.
• The thread pool moves the work; it does not increase capacity. The total processor work a process can handle is fixed.
• The unit of scaling is the process: the primary process opens the listening socket, worker processes share the same port, twelve requests were distributed four apiece to three workers, and a replacement was forked for the worker that died.
• Because processes do not share memory, a counter in process memory diverges across workers; state that needs to survive between requests must move out of the process, into a shared store.

Next Step

This lesson measured the outside of the process: how many copies run, which copy a request lands on, what the copies do not share. Inside the process, however, there is a path a single request takes, and up to now that path has been treated as a single box. How many separate stages are there from the moment a connection is accepted to the moment the response’s last byte is written, how long does each stage take, and how does a decision made at one stage constrain the next? The next lesson breaks the request’s life cycle into stages; it timestamps each stage and prints the order and durations.