---
title: 'The Application Runtime'
source: 'https://academia.sh/en/courses/server-fundamentals/application-runtime'
course: 'Server-Side Fundamentals'
language: en
updated: '2026-08-19T05:19:37+00:00'
license: 'CC BY-SA 4.0'
---

# The Application Runtime

The process model beneath the application: how long work that blocks the event loop makes other requests wait is measured, the limit of the thread pool is shown, and why the unit of scaling is the process is counted with workers that share the same port.

The previous lesson ran the application server behind the proxy as a single process and left
its question open: what happens to the other requests when one takes a long time, how do two
copies of the same application share the same port, and is the unit of a scaling decision the
process or the thread? This lesson answers those three questions by measuring them.

The measurement uses two endpoints of the library loan service: an expensive endpoint that
verifies a member's password, and a cheap endpoint that returns a single line from the
catalog. When the two live in the same process, what the expensive work does to the cheap
requests becomes visible through the delay between them.

## What the Runtime Provides

**The runtime** is the layer application code sits on top of: it houses the engine that
executes the code, the doors that open onto the operating system's socket and file interfaces,
and the **event loop** that queues the work between the two. The model established in The
Node.js Runtime course holds here just as it did there; what is new in this lesson is how that
same model determines a server application's scaling decisions.

The model comes down to one sentence: application code runs on a **single thread**. While one
request handler is running, no other handler runs. The event loop picks up the next queued
work once a handler finishes. Input/output waits — disk reads, network calls, database
queries — do not occupy this queue; the runtime hands the wait off to the operating system and
queues the callback when the result arrives.

This model has two consequences. Waiting work is cheap: thousands of open connections can be
held in a single process, because each one is just a socket and a callback. By contrast,
**computation-heavy work is expensive**: a computation that occupies the processor locks the
queue for its entire duration.

## Blocking Work: How Long One Request Makes Others Wait

The server below performs member password verification in two separate ways. `verify-blocking`
calls key derivation directly and holds the event loop while waiting for the result.
`verify-pooled` hands the same computation to the runtime's **thread pool** and releases the
loop. The computation itself is the same in both.

```js
// single-process.mjs — loan service in a single process; the same work done both blocking and non-blocking
import { createServer } from "node:http";
import { scrypt, scryptSync } from "node:crypto";

const SHELVES = [{ isbn: "978-0262033848", title: "Introduction to Algorithms", shelf: "R-12" }];

// Key derivation cost: high enough to produce over half a second of processor work.
const COST = { N: 1 << 16, r: 8, p: 8, maxmem: 512 * 1024 * 1024 };

createServer((req, res) => {
  res.sendDate = false;
  res.setHeader("content-type", "application/json; charset=utf-8");
  const path = new URL(req.url, "http://local").pathname;

  if (path === "/member/verify-blocking") {
    const digest = scryptSync("password", "salt", 32, COST).toString("hex").slice(0, 16);
    return res.writeHead(200).end(JSON.stringify({ digest, path }));
  }
  if (path === "/member/verify-pooled") {
    return scrypt("password", "salt", 32, COST, (err, key) => {
      if (err) return res.writeHead(500).end(JSON.stringify({ error: "derivation" }));
      res.writeHead(200).end(JSON.stringify({ digest: key.toString("hex").slice(0, 16), path }));
    });
  }
  if (path === "/books") {
    return res.writeHead(200).end(JSON.stringify({ books: SHELVES }));
  }
  res.writeHead(404).end(JSON.stringify({ error: "route_not_found" }));
}).listen(8430, "127.0.0.1", () => console.log("single process 127.0.0.1:8430 pid=" + process.pid));
```

The measurement follows a set pattern: the heavy request is sent first, and fifty milliseconds
later four light requests follow it. The light requests have almost no work of their own; their
durations come only from waiting their turn.

```bash
#!/usr/bin/env bash
# Starts single-process.mjs; measures the duration of four light requests arriving alongside a heavy request.
node single-process.mjs & server=$!
sleep 0.6
curl -sS -o /dev/null http://127.0.0.1:8430/books          # warm-up

measure() {  # $1 = path of the heavy endpoint
  pids=()
  curl -sS -o /dev/null -w "heavy    %{time_total} s\n" "http://127.0.0.1:8430$1" > heavy.txt &
  pids+=($!)
  sleep 0.05
  for i in 1 2 3 4; do
    curl -sS -o /dev/null -w "light-$i  %{time_total} s\n" http://127.0.0.1:8430/books > "light-$i.txt" &
    pids+=($!)
  done
  wait "${pids[@]}"
  cat heavy.txt light-1.txt light-2.txt light-3.txt light-4.txt
}

echo "--- heavy work on the event loop: /member/verify-blocking ---"
measure /member/verify-blocking
echo "--- heavy work on the thread pool: /member/verify-pooled ---"
measure /member/verify-pooled

kill "$server"
rm -f heavy.txt light-*.txt
```

```
single process 127.0.0.1:8430 pid=14674
--- heavy work on the event loop: /member/verify-blocking ---
heavy    0.635000 s
light-1  0.581384 s
light-2  0.581601 s
light-3  0.580992 s
light-4  0.581166 s
--- heavy work on the thread pool: /member/verify-pooled ---
heavy    0.618827 s
light-1  0.001071 s
light-2  0.000954 s
light-3  0.000756 s
light-4  0.001232 s
```

The process id and the durations change on every run; the heavy work's absolute duration also
depends on the machine's processor. What stays fixed is the order-of-magnitude difference
between the light requests' durations in the two blocks.

In the first measurement, the light requests waited about **580 milliseconds**. Their own work
takes less than a millisecond; what they waited for was the heavy request's remaining duration.
All four finish at nearly the same moment, because all of them waited for the same block to
lift, and once it lifted, they were processed one after another.

In the second measurement, the heavy request's duration did not change — it is the same
computation — but the light requests were answered in **under one millisecond**. The only
difference is where the computation happens. In the pooled version, the event loop handed the
computation to a thread and freed itself, served the requests arriving in between at their
normal speed, and ran the callback once the result was ready.

## The Pool Is Limited Too

The thread pool is not the solution to blocking work; it is where that work moves to. The
pool's thread count is fixed, and its default value is small. If more work is handed to the
pool at once than its capacity, the excess queues up; this time it is not the event loop that
saturates, but the pool. Because the single heavy request in the measurement did not fill the
pool, no delay appeared.

This gives the first rule of scaling: **a process's processor capacity is fixed.** Moving the
computation from the loop to the pool reduces how much requests wait on each other, but it does
not increase the process's total processor work. The number of key derivations that can be done
per second is bounded by the machine's core count, independent of which thread does the work.

## The Unit of Scaling: The Process

A single process can saturate a single core. On a multi-core machine, the way to use the
remaining cores is to run several copies of the application. In a **cluster** setup, a
**primary process** opens the listening socket and forks several **worker processes**;
incoming connections are distributed among the workers. The workers share the same port,
because the party that opened the socket is the primary process.

```js
// cluster.mjs — three worker processes sharing the same port; each process has its own counter
import cluster from "node:cluster";
import { createServer } from "node:http";

const WORKER_COUNT = 3;

if (cluster.isPrimary) {
  let shuttingDown = false;
  for (let i = 0; i < WORKER_COUNT; i++) cluster.fork();

  cluster.on("exit", (worker) => {                        // a new worker replaces the one that died
    if (shuttingDown) return;
    console.log(`worker died pid=${worker.process.pid}, forking a replacement`);
    cluster.fork();
  });

  process.on("SIGTERM", () => {                         // workers go down together with the primary
    shuttingDown = true;
    for (const worker of Object.values(cluster.workers)) worker.kill();
    process.exit(0);
  });

  console.log(`primary pid=${process.pid}, worker count=${WORKER_COUNT}`);
} else {
  let loanCounter = 0;                                  // PROCESS-SPECIFIC memory: separate in each worker
  createServer((req, res) => {
    res.sendDate = false;
    res.setHeader("content-type", "application/json; charset=utf-8");
    if (new URL(req.url, "http://local").pathname === "/loan") loanCounter++;
    res.writeHead(200).end(JSON.stringify({ pid: process.pid, loanCounter }));
  }).listen(8431, "127.0.0.1");
}
```

The server answers every path and reports its own process id in the response; only the `/loan`
path increments the counter. This distinction makes it possible to poll the distribution without
changing the counter value.

```bash
#!/usr/bin/env bash
# Starts cluster.mjs; shows the distribution of 12 requests among the workers, the counters
# diverging, and a replacement arriving for a worker that died.
node cluster.mjs & cluster=$!
sleep 0.8

echo "--- 12 loan requests: which worker answered how many times ---"
for i in $(seq 12); do curl -sS http://127.0.0.1:8431/loan; echo; done |
  grep -o '"pid":[0-9]*' | sort | uniq -c

echo "--- four requests to the same endpoint: each worker's counter lives in its own memory ---"
for i in 1 2 3 4; do curl -sS http://127.0.0.1:8431/loan; echo; done

echo "--- a worker is killed, a replacement arrives ---"
victim=$(curl -sS http://127.0.0.1:8431/status | sed -n 's/.*"pid":\([0-9]*\).*/\1/p')
echo "killed worker pid=$victim"
kill -9 "$victim"
sleep 0.8
for i in $(seq 6); do curl -sS http://127.0.0.1:8431/status; echo; done |
  grep -o '"pid":[0-9]*' | sort -u

kill "$cluster"
```

```
primary pid=3068, worker count=3
--- 12 loan requests: which worker answered how many times ---
   4 "pid":3070
   4 "pid":3071
   4 "pid":3072
--- four requests to the same endpoint: each worker's counter lives in its own memory ---
{"pid":3072,"loanCounter":5}
{"pid":3070,"loanCounter":5}
{"pid":3071,"loanCounter":5}
{"pid":3072,"loanCounter":6}
--- a worker is killed, a replacement arrives ---
killed worker pid=3070
worker died pid=3070, forking a replacement
"pid":3071
"pid":3072
"pid":3098
```

The process ids are different on every run. What stays fixed are three observations.

**The distribution is even.** Twelve requests landed four apiece on three workers; the primary
process distributed the connections in turn. Nothing changed on the client side: the request
went to the same address, the same port. How many processes are running is invisible to the
client.

**A replacement arrives for the worker that died.** When a worker is forcibly terminated, the
primary process sees it and forks a new worker; the final list contains none of the old ids, but
a new id in their place. This also shows that a process is a **unit of isolation** as well as a
unit of scaling: one worker crashing does not affect the others, and the service does not go
down.

## Process Memory Is the Scaling Boundary

The most instructive lines in the output are the counter lines. After twelve requests, every
worker's counter reads four; once the next four requests are distributed, the values read `5`,
`5`, `5`, and `6`. No worker knows the total count, because the counter lives in **process
memory**, and processes do not share memory.

In the library application, the counterpart of this is direct. Loan count, session information,
a temporary reservation list, a rate-limit counter — all of them break the moment they are kept
in process memory and scaled. The request in which a member logs in lands on one worker, their
next request lands on another, and the second worker does not recognize the session.

The consequence of this is the binding rule the runtime model places on application design:
**the process must be stateless.** Every value that needs to survive between requests moves out
of the process, into a shared store. The process keeps in its own memory only the values that
live for the duration of that one request. Once this rule is met, increasing the process count
comes down to a configuration decision; when it is not met, starting a second process leads to
errors.

The same rule also follows from the process being mortal. A dead worker's memory goes with it;
the worker that replaces it starts with empty memory. A value kept in process memory is lost not
only during scaling, but on every restart.

## Summary

- Application code runs on a single thread; waiting input/output is cheap, while a computation
  that occupies the processor locks the queue for its entire duration.
- In the measurement, a half-second computation that blocked the event loop made four light
  requests arriving alongside it wait about 580 milliseconds; once the same computation was
  handed to the thread pool, the light requests were answered in under one millisecond.
- The thread pool moves the work; it does not increase capacity. The total processor work a
  process can handle is fixed.
- The unit of scaling is the process: the primary process opens the listening socket, worker
  processes share the same port, twelve requests were distributed four apiece to three workers,
  and a replacement was forked for the worker that died.
- Because processes do not share memory, a counter in process memory diverges across workers;
  state that needs to survive between requests must move out of the process, into a shared
  store.

## Next Step

This lesson measured the outside of the process: how many copies run, which copy a request
lands on, what the copies do not share. Inside the process, however, there is a path a single
request takes, and up to now that path has been treated as a single box. How many separate
stages are there from the moment a connection is accepted to the moment the response's last
byte is written, how long does each stage take, and how does a decision made at one stage
constrain the next? The next lesson breaks the request's life cycle into stages; it timestamps
each stage and prints the order and durations.