Horizontal and Vertical Scaling

The previous two lessons worked with a single process. The proxy forwarded to one target, the process manager started one worker, and the memory limit was that one process’s limit. When load grows, there are two ways to meet it: give the same process more resources — vertical scaling — or open more processes — horizontal scaling.

The speedup side of the two axes, the ceiling the serial fraction sets, and the diminishing return of adding copies were established in the Introduction to System Design lesson; they are taken as given here, not repeated. What this lesson measures is the backend side: how much extra resource sits behind the scenes when two layouts meet the same load, and which setting silently turns wrong when process count changes.

Three Layouts, the Same Load

The setup has three parts: a record store, one or four copies of the loan application, and a process that dispatches between the copies in turn.

• SD1. The store accepts 16 concurrent connections; the seventeenth is closed as soon as it is established. A query takes 20 milliseconds.
• SD2. Each application copy opens as many persistent connections as its pool size on startup (warmup). If none can be established, the pool is empty and the process rejects requests without ever reaching the store.
• SD3. Copies are brought up in sequence; whichever opens its pool first gets the connection first. Memory is averaged per process, ready time is the longest of the copies; both round to 10 and depend on the run. The multipliers do not.

The store genuinely enforces its connection limit.

// topology/store.mjs — record store. Has an upper bound on concurrent connections; a connection
// past it is closed as soon as it is established. Counters are read from a separate port so the
// measurement connection itself is not counted.
import { createServer } from "node:http";

const [port, measurePort, limit, queryMs] = process.argv.slice(2, 6).map(Number);
let open = 0, peak = 0, rejected = 0, queries = 0;

if (Number.isInteger(port) === false) console.log("usage: node topology/store.mjs <port> <measure-port> <connection-limit> <query-ms>");
else {
  const s = createServer((req, res) => {
    queries += 1;
    res.sendDate = false;
    setTimeout(() => res.writeHead(200).end("record"), queryMs);
  });
  s.on("connection", (socket) => {
    open += 1;
    peak = Math.max(peak, open);
    socket.on("close", () => { open -= 1; });
    if (open > limit) { rejected += 1; socket.destroy(); }   // limit crossed: connection closed
  });
  s.listen(port, "127.0.0.1");
  createServer((req, res) => {
    res.sendDate = false;
    res.writeHead(200).end(JSON.stringify({ peak, rejected, queries }));
  }).listen(measurePort, "127.0.0.1");
}

The application copy takes pool size as an argument. This number is the layout’s only setting, and it is chosen correctly for a single process.

// topology/app.mjs — a copy of the loan process. On startup it opens as many persistent
// connections to the store as <pool> (warmup). If none can be established, the pool is empty and
// the process rejects requests without ever reaching the store; otherwise each request is served
// by a connection from the pool.
import { createServer, request, Agent } from "node:http";

const start = Date.now();
const [port, store, poolSize] = process.argv.slice(2, 5).map(Number);
const pool = new Agent({ keepAlive: true, maxSockets: poolSize });
let readyMs = -1, warmupErrors = 0, responded = 0, errors = 0;

const toStore = (path) => new Promise((resolve) => {
  const r = request({ port: store, path, agent: pool }, (y) => { y.resume(); y.on("end", () => resolve(true)); });
  r.on("error", () => resolve(false));
  r.end();
});

if (Number.isInteger(port) === false) console.log("usage: node topology/app.mjs <port> <store> <pool>");
else {
  const results = await Promise.all(Array.from({ length: poolSize }, () => toStore("/warmup")));
  warmupErrors = results.filter((b) => b === false).length;
  readyMs = Date.now() - start;
  createServer(async (req, res) => {
    res.sendDate = false;
    if (req.url === "/measure") {
      const rss = Math.round(process.memoryUsage().rss / (1024 * 1024));
      res.writeHead(200).end(JSON.stringify({ rss, readyMs, warmupErrors, responded, errors }));
      return;
    }
    if (warmupErrors === poolSize) { errors += 1; res.writeHead(503).end("pool empty"); }
    else if (await toStore("/record")) { responded += 1; res.writeHead(200).end("loan"); }
    else { errors += 1; res.writeHead(503).end("store unavailable"); }
  }).listen(port, "127.0.0.1");
}

// topology/dispatcher.mjs — dispatches requests among copies in turn. The copy list is comma-separated.
import { createServer, request } from "node:http";

const [port, listText] = [Number(process.argv[2]), process.argv[3] ?? ""];
const TARGETS = listText.split(",").filter(Boolean).map(Number);
let turn = 0;

if (Number.isInteger(port) === false || TARGETS.length === 0) console.log("usage: node topology/dispatcher.mjs <port> <target,target,...>");
else createServer((clientReq, clientRes) => {
  clientRes.sendDate = false;
  const target = TARGETS[turn++ % TARGETS.length];
  const upstreamReq = request({ port: target, path: clientReq.url, method: clientReq.method }, (upstreamRes) => {
    clientRes.writeHead(upstreamRes.statusCode);
    upstreamRes.pipe(clientRes);
  });
  upstreamReq.on("error", () => clientRes.writeHead(502).end("no copy"));
  clientReq.pipe(upstreamReq);
}).listen(port, "127.0.0.1");

// topology/load.mjs — sends <count> requests with <concurrency> concurrency, then totals the
// copies' and store's counters. Memory and ready time are read per copy and combined.
import { request } from "node:http";

const [port, count, concurrency, storeMeasure, copiesText, label] = [Number(process.argv[2]), Number(process.argv[3]), Number(process.argv[4]), Number(process.argv[5]), process.argv[6] ?? "", process.argv[7] ?? "-"];
const COPIES = copiesText.split(",").filter(Boolean).map(Number);
let answered = 0, rejected = 0;

const one = () => new Promise((resolve) => {
  const r = request({ port, path: "/loan" }, (y) => {
    y.resume();
    y.on("end", () => { if (y.statusCode === 200) answered += 1; else rejected += 1; resolve(); });
  });
  r.on("error", () => { rejected += 1; resolve(); });
  r.end();
});

const read = (p, path) => new Promise((resolve) => {
  request({ port: p, path, agent: false }, (y) => { let b = ""; y.on("data", (d) => { b += d; }); y.on("end", () => resolve(JSON.parse(b))); }).end();
});

if (Number.isInteger(port) === false) console.log("usage: node topology/load.mjs <port> <count> <concurrency> <store-measure> <copies> <label>");
else {
  let sent = 0;
  await Promise.all(Array.from({ length: concurrency }, async () => { while (sent++ < count) await one(); }));
  const k = await Promise.all(COPIES.map((p) => read(p, "/measure")));
  const d = await read(storeMeasure, "/measure");
  const total = (field) => k.reduce((t, x) => t + x[field], 0);
  const P = COPIES.length;
  // Per-process memory average and the longest ready time round to 10, written as a product.
  const memory = Math.round(total("rss") / P / 10) * 10;
  const ready = Math.round(Math.max(...k.map((x) => x.readyMs)) / 10) * 10;
  console.log(`${label} answered ${String(answered).padStart(2)}/${count}  ` +
    `store peak ${String(d.peak).padStart(2)} connections, ${String(d.rejected).padStart(2)} rejected  ` +
    `warmup errors ${String(total("warmupErrors")).padStart(2)}  ` +
    `memory ${P}x${memory}=${String(P * memory).padStart(3)} MB  ready ${P}x${ready}=${String(P * ready).padStart(3)} ms`);
}

Three layouts meet the same sixty requests at the same concurrency. The first is one process, pool eight. The second is four processes and the same pool setting — process count grew, the pool was never touched. The third is four processes, pool two.

# measure.sh — same load with three layouts. The store accepts 16 concurrent connections, query 20 ms.
# Copies come up in sequence: whichever opens its pool first gets the connection first.
setup() {                                 # setup <pool> <copy-count>
  node topology/store.mjs 8950 8959 16 20 & D=$!
  P=""; N=""
  for i in $(seq 1 "$2"); do
    node topology/app.mjs $((8950 + i)) 8950 "$1" & P="$P $!"
    N="$N,$((8950 + i))"
    sleep 0.25
  done
  node topology/dispatcher.mjs 8960 "${N#,}" & G=$!
  sleep 1
}
teardown() { kill $D $G $P 2>/dev/null; wait $D $G $P 2>/dev/null; sleep 0.3; }

setup 8 1; node topology/load.mjs 8960 60 32 8959 "8951" "one process, pool 8    "; teardown
setup 8 4; node topology/load.mjs 8960 60 32 8959 "8951,8952,8953,8954" "four processes, pool 8 "; teardown
setup 2 4; node topology/load.mjs 8960 60 32 8959 "8951,8952,8953,8954" "four processes, pool 2 "; teardown

one process, pool 8     answered 60/60  store peak  8 connections,  0 rejected  warmup errors  0  memory 1x60= 60 MB  ready 1x30= 30 ms
four processes, pool 8  answered 30/60  store peak 17 connections, 16 rejected  warmup errors 16  memory 4x60=240 MB  ready 4x30=120 ms
four processes, pool 2  answered 60/60  store peak  8 connections,  0 rejected  warmup errors  0  memory 4x60=240 MB  ready 4x30=120 ms

The memory and ready-time columns depend on the run; process count, pool size, the store limit, request count, and rejected-connection count do not.

How Many Times the Baseline Is Paid

The first and third rows do the same job: sixty of sixty requests answered. Memory total rose from 60 MB to 240 MB.

The difference does not come from the request itself. Same sixty requests, same twenty-millisecond query, same response. What changes is how many times the baseline is paid: the interpreter, the loaded code, the event loop, and the per-process heap hold 60 MB per process, and four processes pay that four times. The output writes a product rather than a sum for this reason — the number worth reading is the multiplier.

The two axes’ bills diverge here. Vertical scaling pays the baseline once: the process grows, the baseline stays fixed, and the added memory goes to work. Horizontal scaling pays the baseline once per process: moving to four copies spends 180 MB before any work is done, even with no requests arriving. This line item grows linearly with copy count, and does no work.

This does not say horizontal scaling is wrong; K01’s copy table already established why the horizontal axis gets chosen. What it says is that the comparison has to run on total resources: four small processes have less working memory than a single large process using the same total.

Ready Time

A copy spends 30 milliseconds opening its pool and becoming ready for requests. This number is the same in all three layouts; copy count does not change it.

What changes is how many times that time is paid. When four copies start in parallel, wall-clock time stays close to a single copy’s. When copies come up in sequence — exactly what a release replacing copies one at a time does — the times add up, and wall clock reads 120 milliseconds. The same 30 milliseconds is paid four times across four copies, growing linearly with copy count.

The number itself is small; the multiplier is not. In an application where warmup means loading a directory instead of a store connection, establishing a set of connections, or warming a cache, the same multiplier applies to seconds instead of milliseconds; a sequential release’s duration ties itself to copy count.

The Pool Multiplier

The second row is this lesson’s real measurement. The pool setting did not change — eight, correct for one process, stays eight for four. The only thing that changed is process count.

Each process opens its own pool. The connection count landing on the store is not pool per process, it is process count times pool:

Layout	Processes	Pool per process	Reaching the store	Store limit	Result
one process	1	8	8	16	60/60 answered
four processes, pool 8	4	8	32	16	30/60 answered, 16 connections rejected
four processes, pool 2	4	2	8	16	60/60 answered

In the second row the store saw a peak of 17 connections — one above the limit, because a connection crossing the limit is counted and then closed. Sixteen of the thirty-two connection attempts were closed; every closed one was a warmup connection from the two copies that came up later.

The result is that these two copies stay up with an empty pool. Their processes run, they listen on their ports, the process manager counts four live processes, and the dispatcher sees four targets. Thirty of sixty requests go to these two copies and get a 503 without ever reaching the store. The gap between being up and being able to do work opens the instant the layout scales.

This measurement’s silent side is in where the configuration is not. Pool size is a single number in application code, and it is correct. Process count is another number in the process manager, and it is also correct. The total connection count landing on the store is written nowhere; it is the product of two settings, and no file holds that product. A change raising copy count from two to four never touches the pool line, the pool line never appears in a review of it, and the result surfaces not as an error message but as connections closing on the store’s side.

The third row shows the fix: the product is preserved. Four processes, pool two, eight reaching the store. The same sixty requests are answered again. The fix is shrinking the pool, and it has a cost — per-process concurrency drops to a quarter. This is horizontal scaling’s real constraint on the store side: copy count divides the connection budget.

Where the Setting Lives

Setting	Location	How many places	Silent result of a wrong value
Copy count	process manager	1	memory baseline is paid once per copy
Pool per process	application code	1 (same in every copy)	the product exceeds the store limit, connections close silently
Store connection limit	store	1	too low rejects good requests, too high brings the store down
Warmup connection	application code	1	a copy whose pool fails to establish stays up and rejects its share of requests

Two of the four rows can each be correct alone and wrong together. Copy count and pool size sit in separate files, change at separate times, and nothing checks the product between them.

Summary

• The same sixty requests held 60 MB with one process and 240 MB with four; what grows is not working memory, it is the process baseline paid four times.
• The 30-millisecond per-copy ready time multiplies by copy count; bringing up four copies in sequence writes 120 milliseconds to the wall clock.
• Raising process count from one to four without touching the pool setting pushed connections reaching the store from 8 to 32, the store closed 16 connections, 16 warmup requests failed, and only thirty of sixty requests were answered.
• The two copies whose pools failed to establish stayed up: the process manager counted four live processes and the dispatcher four targets, while these two copies rejected their share of thirty requests without ever reaching the store.
• Shrinking the pool to two kept the product at eight, and all sixty of sixty requests were answered again; the cost is per-process concurrency dropping to a quarter.
• The total connection count reaching the store is the product of two settings and is written in no file; because the two settings sit in separate places, a change to one is invisible in a review of the other.

Next Step

This lesson treated copy count as a setting: a number written to the process manager, four processes coming up, the dispatcher splitting requests between them in turn. Across all three layouts, every one of the sixty requests could go to any copy, and the result did not change. This is scaling’s invisible precondition, and it is not automatic: copies can only be multiplied if it does not matter which one a request reaches. A session in memory, a file on local disk, an in-process cache, or a timer set up in application code all break the equivalence between copies. The next lesson counts where these ties sit in the code: how many call sites are pinned to a node, and how many requests get the wrong result once a second process is added.