Load Testing

The previous three lessons collected the numbers the system produces on its own, formatted them, and tied them to a threshold. Every collected number describes what has already happened. It says what happened at today’s load; it does not say what will happen at tomorrow’s load, which metric will cross its threshold first when traffic triples, or whether today’s number will stay the same after the system runs at the same load for hours. These three questions have not been measured, because they have never happened.

This lesson produces load instead of waiting for it. Three separate tests are set up, and the lesson’s claim is this: the three are not the long and short forms of the same test — they ask three separate questions and surface three separate symptoms.

Three Questions

The load test asks whether thresholds are met at the targeted load. Its duration is short, its load is fixed, and its answer is binary: met or not met.

The soak test applies the same load for a long duration and asks whether anything degrades over time. Keeping the load fixed matters specifically: the only thing that changes is duration, so any degradation found is the result not of the load but of something accumulating in the system — memory held and never released, a growing array, a ledger never cleared.

The spike test multiplies the load for a short time and asks two things: what happens during the spike, and how long the system takes to recover after the spike ends. The second question matters more than the first, because a spike is transient, but requests that have nothing to do with the spike also pay a cost throughout the recovery period.

The rig below writes its own load generator. The generator is open-loop: it sends requests at the target rate and does not wait for the server to keep up. A closed-loop generator — one that waits for each response before sending the next — reduces its load on its own once the server slows down, and can never produce a spike at all. On the server side, the tracking endpoint includes a maintenance pass that scans the accumulated records on every hundredth request; this pass is the source of the accumulation the soak test will look for. The measured durations are tied to this machine; the lesson’s claims rest on ratios and counters.

// load/server.mjs — the tracking endpoint: every request spends a wait and a processor share,
// every hundredth request runs a maintenance pass that scans the accumulated records. The
// /metrics endpoint returns the server's counters.
import { createServer } from "node:http";

const WAIT = 3;              // rig parameter: wait standing in for the record lookup (ms)
const CPU = 0.85;            // rig parameter: processor share per request (ms)
const MAINT_INTERVAL = 100;  // rig parameter: how many requests between maintenance passes
const MAINT_FACTOR = 600;    // rig parameter: how many times a maintenance pass scans the accumulated record

const records = [];
let requests = 0, maint = 0, maintSteps = 0;

function work(ms) {
  const end = performance.now() + ms;
  let t = 0;
  while (performance.now() < end) t += 1;
  return t;
}

const port = Number(process.argv[2]);
if (Number.isInteger(port) === false) console.log("usage: node load/server.mjs <port>");
else createServer(async (req, res) => {
  const url = new URL(req.url, "http://local");
  if (url.pathname === "/metrics") {
    res.writeHead(200, { "content-type": "application/json" });
    res.end(JSON.stringify({ requests, records: records.length, maint, maintSteps, factor: MAINT_FACTOR }));
    return;
  }
  requests += 1;
  const seq = requests;                        // its sequence is fixed on entry, not on exit
  records.push({ no: url.searchParams.get("no") ?? "-", step: seq % 7 });
  await new Promise((c) => setTimeout(c, WAIT));
  work(CPU);
  if (seq % MAINT_INTERVAL === 0) {            // maintenance pass: the work grows with the accumulated records
    maint += 1;
    let t = 0;
    for (let k = 0; k < MAINT_FACTOR; k += 1) for (const record of records) t += record.step;
    maintSteps += MAINT_FACTOR * records.length;
  }
  res.writeHead(200, { "content-type": "application/json" });
  res.end(JSON.stringify({ state: "at transfer hub", step: seq % 7 }));
}).listen(port);

// load/generator.mjs — open-loop load generator: sends requests at the target rate, does not
// wait for the server to keep up. All three tests run with the same generator, each against a
// freshly started server.
import { Agent, request } from "node:http";

const port = Number(process.argv[2]);
const test = process.argv[3];
const BASE = 514;          // target rate (requests/s) — K01: peak edge 513.89 requests/s
const SPIKE = 3;           // K01: peak factor 3
const THRESHOLD = 25;      // YT1 (assumption): tracking response's performance threshold (ms)
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });

const pct = (d, p) => {
  const s = [...d].sort((a, b) => a - b);
  return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))];
};
const fix2 = (x) => x.toFixed(2);
const pad = (x, n) => String(x).padStart(n);
const slice = (k, a, b) => k.filter((r) => r.sn >= a && r.sn < b).map((r) => r.ms);

function oneRequest(no, records, sn) {
  const start = performance.now();
  return new Promise((resolve) => {
    const r = request({ port, path: `/tracking?no=${no}`, agent }, (y) => {
      y.resume();
      y.on("end", () => { records.push({ ms: performance.now() - start, sn }); resolve(); });
    });
    r.on("error", () => { records.push({ ms: performance.now() - start, sn, error: 1 }); resolve(); });
    r.end();
  });
}

const metrics = () => new Promise((resolve) => {
  request({ port, path: "/metrics", agent }, (y) => {
    let body = "";
    y.on("data", (p) => { body += p; });
    y.on("end", () => resolve(JSON.parse(body)));
  }).end();
});

async function run(duration, rate) {           // rate(t): target rate as a function of elapsed seconds
  const records = [], inFlight = [];
  let debt = 0, i = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate(t) * (t - lastT);             // based on real elapsed time: absorbs timer drift
    lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(`G${i++}`, records, Math.floor(t))); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  return { records, elapsed: (performance.now() - start) / 1000 };
}

if (Number.isInteger(port) === false || ["load", "soak", "spike"].includes(test) === false) {
  console.log("usage: node load/generator.mjs <port> load|soak|spike");
} else if (test === "load") {
  await oneRequest("warmup", [], 0);
  const r = await run(3, () => BASE);
  const m = r.records.map((k) => k.ms);
  const over = m.filter((x) => x > THRESHOLD).length;
  console.log(`LOAD TEST: target ${BASE} requests/s, 3 s, threshold ${THRESHOLD} ms`);
  console.log(`  completed ${m.length}, reached rate ${fix2(m.length / r.elapsed)} requests/s ` +
    `(${fix2((100 * m.length) / (BASE * r.elapsed))}% of target)`);
  console.log(`  median ${fix2(pct(m, 50))} ms, p95 ${fix2(pct(m, 95))} ms, p99 ${fix2(pct(m, 99))} ms`);
  console.log(`  over threshold ${over} requests (${fix2((100 * over) / m.length)}%)`);
} else if (test === "soak") {
  await oneRequest("warmup", [], 0);
  const o1 = await metrics();
  const r = await run(12, () => BASE);
  const o2 = await metrics();
  console.log(`SOAK TEST: same ${BASE} requests/s, 12 s`);
  console.log(`  quarter   requests   median(ms)   p95(ms)   p99(ms)   over threshold`);
  const c = [];
  for (let i = 0; i < 4; i += 1) {
    const d = slice(r.records, i * 3, (i + 1) * 3);
    c.push(d);
    console.log(`  ${pad(i + 1, 7)}${pad(d.length, 11)}${pad(fix2(pct(d, 50)), 13)}` +
      `${pad(fix2(pct(d, 95)), 10)}${pad(fix2(pct(d, 99)), 10)}${pad(d.filter((x) => x > THRESHOLD).length, 17)}`);
  }
  console.log(`  last/first quarter: median ${fix2(pct(c[3], 50) / pct(c[0], 50))}x, ` +
    `p95 ${fix2(pct(c[3], 95) / pct(c[0], 95))}x, p99 ${fix2(pct(c[3], 99) / pct(c[0], 99))}x`);
  const passes = o2.maint - o1.maint;
  console.log(`  server counters: accumulated records ${o1.records} -> ${o2.records}, maintenance passes ${passes}`);
  console.log(`  steps per maintenance pass: first ${(o1.factor * 100).toLocaleString("en-US")}, ` +
    `last ${(o2.factor * 100 * passes).toLocaleString("en-US")} (${passes}x)`);
} else {
  await oneRequest("warmup", [], 0);
  const r = await run(9, (t) => (t >= 3 && t < 4.5 ? BASE * SPIKE : BASE));
  console.log(`SPIKE TEST: baseline ${BASE} requests/s, ${BASE * SPIKE} requests/s between 3.0-4.5 s, 9 s`);
  console.log(`  second   completed   median(ms)   p95(ms)   over threshold`);
  const baseline = [];
  for (let sn = 0; sn < 9; sn += 1) {
    const d = slice(r.records, sn, sn + 1);
    if (sn < 3) baseline.push(pct(d, 95));
    console.log(`  ${pad(sn, 6)}${pad(d.length, 12)}${pad(fix2(pct(d, 50)), 13)}` +
      `${pad(fix2(pct(d, 95)), 10)}${pad(d.filter((x) => x > THRESHOLD).length, 17)}`);
  }
  let recovery = null;
  for (let sn = 5; sn < 9; sn += 1)
    if (recovery === null && pct(slice(r.records, sn, sn + 1), 95) <= THRESHOLD) recovery = sn - 4.5;
  const extra = Math.round(BASE * (SPIKE - 1) * 1.5);
  const over = r.records.filter((k) => k.ms > THRESHOLD).length;
  console.log(`  baseline p95 ${fix2(pct(baseline, 100))} ms; extra requests sent during spike ${extra}, ` +
    `ended in error ${r.records.filter((k) => k.error).length}`);
  console.log(`  p95 return below threshold: ${recovery === null ? "did not return" : fix2(recovery) + " s after spike ended"}`);
  console.log(`  total requests over threshold ${over} = ${fix2(over / extra)}x the extra requests`);
}
agent.destroy();

# measure.sh — three tests, each run against a freshly started server
for T in load soak spike; do
  node load/server.mjs 8851 & SERVER=$!
  sleep 1
  node load/generator.mjs 8851 $T
  kill $SERVER
  wait $SERVER 2>/dev/null
  echo
done

LOAD TEST: target 514 requests/s, 3 s, threshold 25 ms
  completed 1539, reached rate 512.72 requests/s (99.75% of target)
  median 7.24 ms, p95 9.92 ms, p99 13.05 ms
  over threshold 0 requests (0.00%)

SOAK TEST: same 514 requests/s, 12 s
  quarter   requests   median(ms)   p95(ms)   p99(ms)   over threshold
        1       1538         7.06      9.04      9.65                0
        2       1544         6.97      9.00     10.12                0
        3       1542         7.30     15.95     22.37                4
        4       1541         7.33     20.04     25.99               22
  last/first quarter: median 1.04x, p95 2.22x, p99 2.69x
  server counters: accumulated records 1 -> 6166, maintenance passes 61
  steps per maintenance pass: first 60,000, last 3,660,000 (61x)

SPIKE TEST: baseline 514 requests/s, 1542 requests/s between 3.0-4.5 s, 9 s
  second   completed   median(ms)   p95(ms)   over threshold
       0         512         7.22     10.01                0
       1         514         7.35     12.71                0
       2         514         7.21     11.43                0
       3        1545       101.74   1508.92             1400
       4        1024       111.74   1028.35              976
       5         516        14.07    108.66              224
       6         514         7.31     16.52                4
       7         512         7.61     20.21               12
       8         516         7.32     22.93               15
  baseline p95 12.71 ms; extra requests sent during spike 1542, ended in error 0
  p95 return below threshold: 1.50 s after spike ended
  total requests over threshold 2631 = 1.71x the extra requests

The output’s duration columns — median, p95, p99, and the multiples derived from them — are tied to this machine and its load at that moment; they come out differently on another machine, and differently again if repeated on the same machine. The counter columns, by contrast, do not depend on the machine’s speed: the number of requests sent, the accumulated records, the number of maintenance passes, and the steps per maintenance pass are all determined by the product of the target rate and the duration. The claims in the next three sections rest on this second set and on the direction of the duration numbers; values read from a single run are marked in this run.

The Load Test Passed

The first test reached its target: the generator stayed within one percent of the requested rate (in this run, 512.72 requests/s, 99.75% of target). Reporting the reached rate looks unnecessary, and it is not — if the generator had not reached its target, the measured latency would be the latency of a different load, and the table could not be read. The percentiles stayed clearly below the threshold, and in this run no request exceeded the 25-millisecond threshold. The answer is binary, and it is positive.

This test’s guarantee, on its own, is narrow. What it says is this: a system in this state meets the threshold at this load, for this duration. Two more tests remove two of those three conditions.

Soak Test: The Median Lies

The second test applied the same load for four times the duration, and the table splits in two down the middle. The median stays put across all four quarters (in this run, the last quarter is 1.04x the first). Looking only at that number, someone would say the system has not changed. Yet over the same quarters p95 and p99 have grown by close to two times (2.22x and 2.69x in this run), and threshold-exceeding requests go from zero in the first quarter to dozens in the last (22 in this run). At the same load, in the same code, a threshold violation was born purely because the duration got longer.

The multiples’ exact values move from run to run; their direction does not. And the number carrying the claim is not those multiples anyway — it is the server’s own counters. The cause of the degradation is not a guess; it is written down: accumulated records climbed to 6,166, sixty-one maintenance passes ran, and a maintenance pass’s step count went from 60,000 to 3,660,000 — exactly 61x. This ratio is independent of the run: no matter how fast the machine is, the sixty-first pass does sixty-one times the work of the first, because the scanned array grows by one hundred records every hundred requests. The median’s not budging comes from the same number: a maintenance pass slows down only one request out of a hundred — one percent of requests. The median is the fiftieth-percentile request, and nothing happens there. The symptom of accumulation shows up not in the median, but in tail latency.

Spike Test: The Cost Is Bigger Than the Spike

The third test multiplied the load by three for a second and a half — the peak factor from the Introduction to System Design course was also three. In the spike second, p95 grew by two orders of magnitude (from 12.71 milliseconds to 1,508.92 milliseconds in this run). After the spike ended, p95 took a second and a half to return below the threshold; the measurement has second-level resolution, and the work that had accumulated during that time was worked off.

The test’s real number is in the last line. During the spike window, 1,542 extra requests were sent — this number is a computed value, it follows from the generator’s parameters, and it does not depend on the run. The number of threshold-exceeding requests is larger than that (2,631 in this run, that is, 1.71x). The ratio’s exact value moves from run to run, but its being greater than one does not, because its cause is structural: the excess work entering the queue also makes the ordinary requests behind it wait, so the set of requests whose latency degrades is wider than the set of requests that spiked. A spike’s cost is measured not by the number of requests that spiked, but by the total number of requests whose latency degraded.

One column also stays empty: the number of requests that ended in error is zero. The system did not reject the excess load; it queued it. The back-pressure and load-shedding patterns built in the Resilience and Reliability course change exactly this behavior; neither is present here, so the capacity shortfall was paid for in latency. This is the symptom the spike test surfaces, and neither the load test nor the soak test can see it.

Summary

• The three tests ask three separate questions: the load test measures today’s threshold, the soak test measures accumulation over time, and the spike test measures the degradation during a spike and the recovery afterward.
• The load generator must be open-loop: a generator that waits for each response reduces its own load once the server slows down and can never produce a spike at all. The reached rate is reported on every run (99.75% of target in this run).
• The load test met the threshold: within one percent of the target rate, p99 stayed well below the threshold and no request exceeded it (13.05 ms and 0 requests in this run).
• The soak test found degradation at the same load: the median stayed put while p95 and p99 grew by close to two times, and threshold-exceeding requests went from zero to dozens (1.04x / 2.22x / 2.69x, 22 requests in this run).
• The source of the degradation is in the server’s counters: a maintenance pass’s step count grew 61x; because the pass slows down only one request in a hundred, the symptom shows up in tail latency, not the median.
• In the spike test, 1,542 extra requests caused more requests than that to exceed the threshold (2,631, 1.71x, in this run); the ratio being greater than one is structural. p95 took a second and a half to return below the threshold, and no request was rejected — the excess was paid for in latency.

Next Step

The three tests showed how far a single process can carry a system’s load today, but none of them asked “how many are needed.” The number the load test gives is not a limit but a confirmation: 514 requests/s is met. The limit itself was measured in the Introduction to System Design course — a process’s saturation throughput — and that course also showed that latency does not grow linearly as utilization rises. The next lesson takes these two inputs and turns them into a forecast: it derives how many nodes the measured utilization requires, counts the capacity that remains after a node is lost, and computes how the accumulation the soak test showed changes that node count, and how often a growing demand raises it. The one decision in between is a ratio, and it has a name: headroom.