Performance Testing Types

The closing of the Integration, Contract and End-to-End Testing course left three questions open: how the system behaves under load, what it yields under attack, and how it degrades under failure were never tested in any lesson there — every test in that course asked is the function correct. This course starts with the first of those and asks the first decision: which test runs. Testing under load takes four forms, and what separates them is not duration or severity but the question each one asks.

Four Questions, Four Decision Windows

Load testing asks whether the threshold is met at the targeted load, soak testing asks whether the same load produces degradation over a long duration, spike testing asks how long recovery takes after a surge. All three were set up and measured in the Performance Anti-Patterns and Monitoring course, where the result fed into a capacity decision. Here the test itself is designed, and a fourth type is added: stress testing raises the load until the threshold breaks and looks for the breaking point.

All four read a single metric — the p95 latency of the loan request — but each reads it in a different decision window: load testing reads the whole run, stress testing reads the step at twice the target rate, soak testing reads the last quarter, spike testing reads the first second after the surge.

Threshold and the Threshold’s Source

A performance test’s output is a distribution; saying “passed” means choosing a threshold. This course’s rule: a threshold is written with its source.

NF1 (assumption + calculation) — the circulation-desk flow’s server-side budget is 60 ms; the flow makes three service calls and the budget is split evenly. The loan service’s share: p95 ≤ 20 ms. The budget is an assumption, the share is a calculation; the threshold was never read off a measurement. NF2 (assumption) — the circulation-desk transaction rate at peak hour is 1,000 req/s.

The decision’s owner is written too: a red result stops the release; the breaking point that stress testing produces does not stop it — it feeds the capacity plan.

Harness

The system under test is the library’s loan service: the member’s open loan count is validated, then records are written. A circulation-desk transaction gives a member at most five books — the loan limit carried over from the Unit Testing and Test-Driven Development course — and each book adds one row. Three structures run: an unindexed catalog, an indexed catalog, and a third structure that is indexed but computes the loan count incorrectly.

// library/server.mjs — loan service: validates the member's open loan count in a
// node:sqlite catalog and writes new records. Arguments: <port> <structure 0|1|2>
// 0 = unindexed (full scan), 1 = indexed, 2 = indexed but the count is wrong.
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 18000, BOOK = 5;  // 36 records per member; a desk transaction writes 5 books
const port = Number(process.argv[2]), structure = process.argv[3];

if (Number.isInteger(port) === false || ["0", "1", "2"].includes(structure) === false) {
  console.log("usage: node library/server.mjs <port> <structure 0|1|2>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  if (structure !== "0") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const count = db.prepare(structure === "2"
    ? "SELECT COUNT(*) AS n FROM loan WHERE member_no = ? AND day = 0"   // flawed count
    : "SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const insert = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  let requests = 0, records = START, scan = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ requests, records, scan })); return; }
    requests += 1;
    scan += structure === "0" ? records : Math.ceil(Math.log2(records));   // full scan in the unindexed structure
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n;
    for (let j = 0; j < BOOK; j += 1) insert.run(member, `K-${(requests + j) % 9000}`, requests % 400);
    records += BOOK;
    res.end(JSON.stringify({ member, loan: n + 1 }));
  }).listen(port);
}

The load generator is open-loop: it sends requests at the target rate and does not wait for the server to keep up. A test oracle also runs during the load-test run — the source of the expected result, in the term from the Quality and Testing Fundamentals course. Its rule: since every request writes five records for the member, the returned loan count must be greater than the previous response’s.

// library/types.mjs — open-loop generator and the four test types; all four read the same
// metric (p95) in a different window. Arguments: <port> <structure 0|1|2> <type>
import { Agent, request } from "node:http";

const TARGET = 1000, THRESHOLD = 20;    // NF2: target rate (req/s), NF1: p95 threshold (ms)
const STEP = [0.5, 1, 2, 3];            // stress ladder: multiples of the target rate, one step per second
const STRUCTURE = { 0: "unindexed", 1: "indexed", 2: "fast-wrong" };
const [port, structure, type] = [Number(process.argv[2]), process.argv[3], process.argv[4]];
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
const last = new Map();
let broken = 0, checks = 0;

const percentile = (d, p) => { const s = [...d].sort((a, b) => a - b);
  return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))]; };
const window = (k, a, b) => k.filter((r) => r.sec >= a && r.sec < b).map((r) => r.ms);
const fmt1 = (x) => x.toFixed(1);
const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let g = "";
    y.on("data", (p) => { g += p; }); y.on("end", () => done(JSON.parse(g))); }).end();
});

function oneRequest(i, samples, sec, oracle) {
  const start = performance.now(), member = `U-${i % 500}`;
  return new Promise((done) => {
    const r = request({ port, path: `/loan?member=${member}&book=K-${i % 900}`, agent }, (y) => {
      let g = ""; y.on("data", (p) => { g += p; });
      y.on("end", () => {
        if (oracle) {                   // every request writes five records for the member: count must rise
          const n = JSON.parse(g).loan;
          if (last.has(member)) { checks += 1; if (n <= last.get(member)) broken += 1; }
          last.set(member, n);
        }
        samples.push({ ms: performance.now() - start, sec }); done();
      });
    });
    r.on("error", () => { samples.push({ ms: performance.now() - start, sec, error: 1 }); done(); });
    r.end();
  });
}

async function run(duration, rate, oracle = false) {   // rate(t): target rate at second t
  const samples = [], inFlight = [];
  let debt = 0, i = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate(t) * (t - lastT);     // scaled by real elapsed time: absorbs timer drift
    lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(i++, samples, t, oracle)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

if (Number.isInteger(port) === false || ["load", "stress", "soak", "spike"].includes(type) === false) {
  console.log("usage: node library/types.mjs <port> <structure 0|1|2> load|stress|soak|spike");
} else {
  await oneRequest(0, [], 0, false);                // warm-up request
  let r, decision, extra;
  if (type === "load") {
    r = await run(2, () => TARGET, true);           // the oracle is meaningful only in an unsaturated run
    decision = percentile(r.samples.map((k) => k.ms), 95);
    extra = `oracle ${broken}/${checks} broken`;
  } else if (type === "stress") {
    r = await run(4, (t) => TARGET * STEP[Math.min(3, Math.floor(t))]);
    const p = STEP.map((_, i) => percentile(window(r.samples, i, i + 1), 95));
    const k = p.findIndex((x) => x > THRESHOLD);
    decision = p[2];                                // p95 of the 2x step
    extra = `break ${k < 0 ? "> 3000" : TARGET * STEP[k]} req/s`;
  } else if (type === "soak") {
    r = await run(10, () => TARGET);
    const c = [0, 1, 2, 3].map((i) => percentile(window(r.samples, i * 2.5, i * 2.5 + 2.5), 95));
    decision = c[3];                                // p95 of the last quarter
    extra = `quarter p95 ${c.map(fmt1).join(" / ")}`;
  } else {
    r = await run(5, (t) => (t >= 1.5 && t < 3 ? TARGET * 3 : TARGET));
    decision = percentile(window(r.samples, 3, 4), 95);   // first second after the spike
    const d = [3, 4].find((sec) => percentile(window(r.samples, sec, sec + 1), 95) <= THRESHOLD);
    extra = `recovery ${d === undefined ? "2.0+" : fmt1(d - 2)} s`;
  }
  const o = await metrics();
  console.log(JSON.stringify({ structure: STRUCTURE[structure], type, decision, extra,
    rate: r.samples.length / r.elapsed, scan: o.scan }));
}
agent.destroy();

The report reads the nine runs; the scan moves the threshold away from the 20 milliseconds NF1 gives and counts, at each value, how many types make the wrong decision.

// library/report.mjs — turns nine runs into a table and scans the p95 threshold. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 20, TYPE = ["load", "stress", "soak", "spike"];
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (y, t) => s.find((r) => r.structure === y && r.type === t);
const fmt2 = (x) => x.toFixed(2);
const pad = (x, n) => String(x).padStart(n);
const verdict = (x) => (x <= THRESHOLD ? "pass" : "fail");

console.log(`threshold p95 <= ${THRESHOLD} ms (NF1), target rate 1000 req/s (NF2)`);
console.log("type    structure     reached/s  decision p95  verdict  scan steps    type-specific figure");
for (const t of TYPE) for (const y of ["unindexed", "indexed", "fast-wrong"]) {
  const r = find(y, t);
  if (r !== undefined) console.log(t.padEnd(8) + y.padEnd(14) + pad(fmt2(r.rate), 9) +
    pad(fmt2(r.decision), 14) + pad(verdict(r.decision), 9) + pad(r.scan.toLocaleString("en-US"), 13) + "  " + r.extra);
}

console.log("\nthreshold scan: how many of the four types give a wrong verdict (unindexed flawed, indexed sound)");
console.log("p95 threshold (ms)  false pass /4  false fail /4  distinguishing type /4  distinguishing types");
for (const e of [1, 3, 5, 20, 50]) {
  const pass = TYPE.filter((t) => find("unindexed", t).decision <= e);
  const fail = TYPE.filter((t) => find("indexed", t).decision > e);
  const dist = TYPE.filter((t) => find("indexed", t).decision <= e && find("unindexed", t).decision > e);
  console.log(String(e).padEnd(20) + pad(pass.length, 14) + pad(fail.length, 15) + pad(dist.length, 22) +
    "  " + (dist.join(", ") || "-"));
}

# measure.sh — nine runs: four types x two structures, plus a load test for the third structure.
# Each run works with a freshly started server; the catalog always starts at the same size.
rm -f result.jsonl
for S in 0 1 2; do
  for T in load stress soak spike; do
    if [ "$S" = 2 ] && [ "$T" != load ]; then continue; fi
    node library/server.mjs 8851 $S & SERVER=$!
    sleep 0.7
    node library/types.mjs 8851 $S $T >> result.jsonl
    kill $SERVER
    wait $SERVER 2>/dev/null
  done
done
node library/report.mjs result.jsonl

threshold p95 <= 20 ms (NF1), target rate 1000 req/s (NF2)
type    structure     reached/s  decision p95  verdict  scan steps    type-specific figure
load    unindexed        996.70          3.18     pass   45,883,050  oracle 0/1495 broken
load    indexed          997.03          0.91     pass       30,000  oracle 0/1499 broken
load    fast-wrong       996.38          1.44     pass       29,955  oracle 1493/1496 broken
stress  unindexed       1107.94       1486.37     fail  222,457,280  break 2000 req/s
stress  indexed         1622.47          1.33     pass      100,918  break > 3000 req/s
soak    unindexed        940.88        991.11     fail  429,907,005  quarter p95 3.2 / 4.0 / 9.3 / 991.1
soak    indexed          999.33          1.10     pass      157,504  quarter p95 1.1 / 1.0 / 0.9 / 1.1
spike   unindexed        903.12         39.91     fail  304,212,030  recovery 2.0 s
spike   indexed         1598.71          1.08     pass      125,030  recovery 1.0 s

threshold scan: how many of the four types give a wrong verdict (unindexed flawed, indexed sound)
p95 threshold (ms)  false pass /4  false fail /4  distinguishing type /4  distinguishing types
1                                0              3                     1  load
3                                0              0                     4  load, stress, soak, spike
5                                1              0                     3  stress, soak, spike
20                               1              0                     3  stress, soak, spike
50                               2              0                     2  stress, soak

The duration columns depend on this machine; the scan step does not: a full scan’s step count is the record count at that moment, and an indexed lookup’s is its binary logarithm. The requests sent are arithmetic too — the target rate times the duration, 55,000 across nine runs. The claims rest on this set; values read from a single run are marked in this run.

Four Types, Four Decisions

The unindexed structure passed one of the four types and failed three. Load testing found p95 at the target rate at 3.18 ms in this run — about a sixth of the threshold. The same structure broke at twice the target rate under stress testing, climbed to 991.1 ms in the last quarter under soak testing, and under spike testing could not drop back below the threshold within two seconds of the surge. Same metric, same threshold, four separate decisions; the difference is the window.

The soak test’s quarter sequence also gives the reason. The load is constant, the code is the same; the only thing that changes is the number of records in the catalog. A full scan touches the entire table on every request, the table grows by five rows on every request, service time rises, and saturation catches up somewhere along the way. The scan step shows this independently of the run: the unindexed structure spent 429,907,005 row steps while the indexed structure did the same work in 157,504 steps — about 2,730 times. The name of the defect class this catches is degradation tied to data growth; it is the input to the anti-pattern diagnoses in the Performance Anti-Patterns and Monitoring course.

The number stress testing gives is of a different kind. The ladder is kept short; a longer one would mix data growth into the rate. The unindexed structure broke at 2,000 req/s; the indexed structure was still under the threshold at 3,000 req/s. This is not a pass–fail but a distance: how many multiples of the target rate the structure holds up to.

Threshold Scan

A threshold can only be defended with two numbers: how many real defects it misses and how many sound runs it turns red. The scan counts this across the threshold’s range.

At the 20 milliseconds NF1 gives, false pass is 1, false fail is 0. The escaping type is load testing: in a two-second run the table only grows by 10,000 rows, so the flawed structure stays well under the threshold. Lowering the threshold to 5 ms does not change this; at 3 ms every type classifies correctly in this run. Tightening to 1 ms overcorrects: the sound structure now fails three of the four types, its own p95 crossing 1 ms under stress, soak and spike. At 50 ms false pass rises to 2.

Two conclusions follow. Load testing cannot see this defect class at NF1’s threshold, because its window gives the defect no time to grow. And a value like 3 ms is derived from no budget — it is fitted to this run’s numbers, and two milliseconds tighter the same fitted value turns three sound results into false fails. Decoupling the threshold from its source empties the decision without fixing the scan.

The Class It Misses and the Cost

The third structure is indexed, fast and wrong: it computes the loan count from only a single day’s records. Load testing’s p95 is 1.44 ms in this run — indistinguishable from the sound structure’s 0.91 ms — and all four types would let this structure pass, since none of them read the response body. Running the oracle shows 1,493 of 1,496 checks come back broken. The name of the defect class this misses is a response-correctness defect: performance tests read duration, not the body. What keeps them blind is not the window, it is the metric.

The run-dependent side of the cost depends on the environment: nine runs finished in under a minute on this machine. The run-independent side is countable — nine server processes, 55,000 requests, roughly 1.00 billion row steps in the unindexed structure. The most expensive type is soak testing: close to five times the requests of load testing, and about 9.4 times its scan steps.

Summary

• Four types read the same metric (p95) in a different decision window: the whole run, the 2x step, the last quarter, the first second after the surge.
• A threshold is written with its source: NF1’s 20 ms comes from splitting a 60 ms budget assumption evenly across three services, never read off a measurement.
• The same defect got four separate decisions: load testing passed, stress testing broke at 2,000 req/s, soak testing found 991.1 ms in the last quarter in this run, spike testing only recovered a window later.
• The threshold scan gives false pass 1, false fail 0 at NF1; at 3 ms every type classifies correctly in this run, but two milliseconds tighter the same fitted value turns three sound results into false fails — a threshold fitted to one run is fragile.
• The class it catches is degradation tied to data growth (scan steps about 2,730 times apart); the class it misses is a response-correctness defect: a fast, wrong structure passes all four types.

Next Step

Every run in this lesson sent requests to a single endpoint, at a fixed rate, without a pause. A real circulation-desk day is not like that: the clerk searches, looks at the result, checks a book out, and pauses between transactions. The next lesson takes on the scenario — how pauses and transaction ratios produce a different result at the same target rate, how an unrealistic scenario lets a real defect pass, and at what share of the mix the defect becomes visible.