Benchmarking Discipline

The previous lesson’s shares were divided by a single baseline run, and one share crossed

1. The cause was not a defect: the same harness gives two different numbers when run twice. The harness also excluded the first second from measurement — the existence of a warm-up effect was accepted, its size never measured. This lesson puts the measurement itself to the test.

The question is a release decision. The loan service’s candidate version adds the member’s overdue book count to the circulation-desk screen, and this runs one more query per request. The release gate has to ask: when the candidate version’s measurement comes back high, does the difference come from the version, or from the measurement?

When a Difference Counts as Meaningful

Two runs of the same harness not giving the same number is called noise; numbers spreading from run to run is called run-to-run variance. A version difference can only be read to the extent it can be told apart from that spread.

NF9 (assumption) — if the candidate version’s measurement exceeds the baseline’s by more than five percent, it counts as a regression and the release stops. Its source is not a requirement or a measurement but a number chosen for gate sensitivity; it is scanned below. The decision’s owner is the release gate: when it turns red, the candidate version does not ship.

To count the threshold’s two error types, the harness runs three arms. The baseline and control arms run the same code; the candidate arm runs the new version. The control arm’s real difference is zero; if the gate turns it red, that is a false fail. The candidate arm’s difference is real; if the gate lets it pass, that is a false pass. Running two identical arms is the cheapest way to read the harness’s own error.

Harness

The measurement runs ten rounds; each round is three runs, and each run starts with its own server process. Running the three arms back to back keeps a shift in machine load from loading onto a single arm.

// library/server.mjs — loan endpoint. The candidate version (b) also adds the overdue
// book count to the response, meaning it runs one more query per request. Arguments: <port> <version a|b>
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 80000, RECENT = 40, OVERDUE = 30;   // catalog size, list length, overdue-day cutoff
const port = Number(process.argv[2]), version = process.argv[3];

if (Number.isInteger(port) === false || ["a", "b"].includes(version) === false) {
  console.log("usage: node library/server.mjs <port> <version a|b>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const recentStmt = db.prepare("SELECT id, book_no, day FROM loan WHERE member_no = ? LIMIT ?");
  const overdueStmt = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ? AND day > ?");
  let requests = 0, queries = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ requests, queries })); return; }
    requests += 1;
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n, list = recentStmt.all(member, RECENT); queries += 2;
    let overdue = -1;
    if (version === "b") { overdue = overdueStmt.get(member, OVERDUE).n; queries += 1; }   // the candidate's extra field
    res.end(JSON.stringify({ member, loan: n, overdue, list }));
  }).listen(port);
}

The generator carries over from the previous lessons, with two differences: it does not exclude warm-up itself, and it counts failed requests. The second is required — a request whose connection is refused looks like it finished very fast, so if it is not counted, a dead server looks like a good measurement.

// library/measure.mjs — open-loop generator. Records latencies in send order and does not
// exclude warm-up. Arguments: <arm name> <port> <round> <target rate> <duration s>
import { Agent, request } from "node:http";

const [arm, port, round, rate, duration] = process.argv.slice(2).map((x, i) => (i === 0 ? x : Number(x)));
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
let errors = 0;

function oneRequest(ms, i) {
  const start = performance.now();
  return new Promise((done) => {
    const finish = () => { ms[i] = Math.round((performance.now() - start) * 1000) / 1000; done(); };
    const r = request({ port, path: `/loan?member=U-${i % 500}`, agent }, (y) => { y.resume(); y.on("end", finish); });
    r.on("error", () => { errors += 1; finish(); });   // a failed request looks fast, so it is counted
    r.end();
  });
}

const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let body = "";
    y.on("data", (p) => { body += p; }); y.on("end", () => done(JSON.parse(body))); }).end();
});

if (Number.isInteger(port) === false) {
  console.log("usage: node library/measure.mjs <arm> <port> <round> <target rate> <duration>");
} else {
  const ms = [], inFlight = [];
  let debt = 0, lastT = 0, seq = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(ms, seq++)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  const m = await metrics(), elapsed = (performance.now() - start) / 1000;
  console.log(JSON.stringify({ arm, round, reached: ms.length / elapsed, errors, queries: m.queries / m.requests, ms }));
}
agent.destroy();

The scan picks random sets from the ten rounds and re-makes the decision; the generator is a linear congruential one and its seed is visible.

// library/compare.mjs — warm-up, variability and decision scan. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 0.05, SEED = 20260731, TRIALS = 500, WARMUP = 100, BLOCK = 100;   // NF9, visible seed
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const fmt = (x, n = 2) => x.toFixed(n);
const pad = (x, n) => String(x).padStart(n);
const percentile = (d, p) => { const t = [...d].sort((a, b) => a - b);
  return t[Math.min(t.length - 1, Math.floor((p / 100) * t.length))]; };
const ROUND = [...new Set(s.map((r) => r.round))];
const find = (t, k) => s.find((r) => r.round === t && r.arm === k);
const pool = (k, start, end) => s.filter((r) => r.arm === k).flatMap((r) => r.ms.slice(start, end));
const P = (k) => percentile(pool(k, WARMUP, 1e9), 95);

console.log(`${ROUND.length} rounds, ${Math.round(pool("baseline", 0, 1e9).length / ROUND.length)} ` +
  `requests per arm per round, ${s.reduce((a, r) => a + r.errors, 0)} failed requests. Pooled p95: baseline ${fmt(P("baseline"))} ms, control ` +
  `${fmt(P("control"))}, candidate ${fmt(P("candidate"))}; candidate/baseline ${fmt(P("candidate") / P("baseline"))}x. Queries/request: ` +
  `baseline ${find(ROUND[0], "baseline").queries}, candidate ${find(ROUND[0], "candidate").queries}.`);

console.log(`\nwarm-up: every round's matching request blocks pooled, block size ${BLOCK} requests`);
console.log("request block  baseline median  baseline p95(ms)  candidate p95(ms)");
for (let i = 0; i < 5; i += 1) {
  const x = i * BLOCK, y = i < 4 ? x + BLOCK : 1e9, d = pool("baseline", x, y);
  console.log((i < 4 ? `${x}-${x + BLOCK - 1}` : `${x}+`).padEnd(13) + pad(fmt(percentile(d, 50), 3), 17) +
    pad(fmt(percentile(d, 95)), 18) + pad(fmt(percentile(pool("candidate", x, y), 95)), 19));
}

const p95 = (t, k) => percentile(find(t, k).ms.slice(WARMUP), 95);
console.log("\nper-round reading: one number per round, first 100 requests dropped");
console.log("measure           min      median    max       spread factor");
for (const name of ["baseline p95(ms)", "control/baseline", "candidate/baseline"]) {
  const d = ROUND.map((t) => (name.includes("/") ? p95(t, name.split("/")[0]) / p95(t, "baseline") : p95(t, "baseline")))
    .sort((m, n) => m - n);
  console.log(name.padEnd(18) + pad(fmt(d[0]), 8) + pad(fmt(percentile(d, 50)), 9) + pad(fmt(d[d.length - 1]), 10) +
    pad(fmt(d[d.length - 1] / d[0]), 13));
}

let t = SEED;
const rand = () => { t = (1103515245 * t + 12345) % 2147483648; return t / 2147483648; };
const shuffle = (n) => { const a = [...Array(n).keys()];
  for (let i = n - 1; i > 0; i -= 1) { const j = Math.floor(rand() * (i + 1)); [a[i], a[j]] = [a[j], a[i]]; }
  return a; };
const median = (d, ix) => percentile(ix.map((i) => d[i]), 50);
const scan = (n, threshold, x) => {
  const series = (k) => ROUND.map((tt) => percentile(find(tt, k).ms.slice(x), 95));
  const BASE = series("baseline"), CTRL = series("control"), CAND = series("candidate");
  let falseFail = 0, falsePass = 0;
  for (let i = 0; i < TRIALS; i += 1) {
    const ix = shuffle(ROUND.length).slice(0, n), gate = median(BASE, ix) * (1 + threshold);
    if (median(CTRL, ix) > gate) falseFail += 1;      // control runs the same code as baseline: no regression
    if (median(CAND, ix) <= gate) falsePass += 1;     // the candidate runs one more query per request
  }
  return `${pad(falseFail, 3)}/${pad(falsePass, 3)}`;
};

const THRESHOLDS = [0.02, THRESHOLD, 0.10, 0.20];
console.log(`\ndecision scan: ${TRIALS} trials, seed ${SEED}; cell = false fail / false pass`);
console.log("rounds/side  window                " + THRESHOLDS.map((e) => pad(`threshold ${Math.round(e * 100)}%`, 15)).join(""));
for (const [n, x] of [[1, WARMUP], [2, WARMUP], [3, WARMUP], [5, WARMUP], [8, WARMUP], [1, 0], [8, 0]]) {
  console.log(pad(n, 7) + "  " + (x === 0 ? "with warm-up" : "first 100 dropped").padEnd(20) +
    THRESHOLDS.map((e) => pad(scan(n, e, x), 15)).join(""));
}

# measure.sh — ten rounds; each round runs three arms back to back: baseline, control
# (same code as baseline) and candidate. Each run starts its own server process.
rm -f result.jsonl
for R in $(seq 1 10); do
  for ARM in "baseline a 8855" "control a 8856" "candidate b 8857"; do
    set -- $ARM
    node library/server.mjs $3 $2 & SERVER=$!
    sleep 0.8
    node library/measure.mjs $1 $3 $R 500 1.4 >> result.jsonl
    kill $SERVER
    wait $SERVER 2>/dev/null
  done
done
node library/compare.mjs result.jsonl

10 rounds, 698 requests per arm per round, 0 failed requests. Pooled p95: baseline 2.12 ms, control 2.13, candidate 2.27; candidate/baseline 1.07x. Queries/request: baseline 2, candidate 3.

warm-up: every round's matching request blocks pooled, block size 100 requests
request block  baseline median  baseline p95(ms)  candidate p95(ms)
0-99                     0.496              1.65               1.84
100-199                  0.784              2.24               2.54
200-299                  1.022              2.27               2.45
300-399                  0.947              1.94               2.17
400+                     0.882              2.08               2.17

per-round reading: one number per round, first 100 requests dropped
measure           min      median    max       spread factor
baseline p95(ms)      1.69     2.16      2.27         1.34
control/baseline      0.92     1.00      1.31         1.43
candidate/baseline    0.71     1.10      1.43         2.03

decision scan: 500 trials, seed 20260731; cell = false fail / false pass
rounds/side  window                   threshold 2%   threshold 5%  threshold 10%  threshold 20%
      1  first 100 dropped           179/155        180/188        125/277         82/375
      2  first 100 dropped           212/ 49        168/144         90/314          0/424
      3  first 100 dropped           140/ 80         58/120          0/340          0/497
      5  first 100 dropped            79/ 20          6/ 56          0/415          0/500
      8  first 100 dropped             0/  0          0/  0          0/500          0/500
      1  with warm-up                181/150        125/191        124/279         81/376
      8  with warm-up                  0/  0          0/  0          0/500          0/500

The Warm-Up Window

Warm-up is a process’s first requests being served at a different cost than the settled state. The first table reads the first hundred requests’ median at 0.496 ms and the requests past the four-hundredth at 0.882 ms: the first block is about 1.78 times optimistic on the median. The same block’s p95 does not follow: at 1.65 ms it is close to the last block’s 2.08 ms, and the two middle blocks (2.24 and 2.27 ms) read highest — in the tail the effect is real but has no single direction. Warm-up does two things at once: compilation and cache fill make a handful of requests very slow, and because the generator’s concurrency has not yet built up, the remaining requests find an empty queue.

The last column matters most. In the first block the candidate/baseline ratio is 1.84/1.65, or 1.12; in the settled window it is 2.17/2.08, or 1.04. In this run warm-up does not flip the sign of the regression, but it does inflate it, and the two middle blocks inflate it further (1.13 and 1.08). The window is read off the measurement for exactly this reason — the harness drops the first hundred requests, because the table only settles from the second block on.

Noise and the Single-Round Decision

The second table reads the same number ten times. The baseline’s p95 spreads 1.34 times, from 1.69 to 2.27 ms. Nothing changed in the code; the only thing that changed is the moment the round ran.

The second row gives the harness’s own error. The control arm runs the same code as the baseline; the ratio is expected to be 1.00 in every round, but it ranges from 0.92 to 1.31. The median is 1.00, so there is no systematic bias; but the spread is 1.43 times, meaning two identical versions can show what looks like a 31-percent “difference” in a single round.

The third row carries the real difference: the rounds’ candidate/baseline median is 1.10, spread 0.71–1.43. The pooled ratio is 1.07; the rounds’ median treats every round equally, the pool weights the slower rounds more. The smallest value is below 1: in one round, the candidate version — despite running one more query per request — measured about 29 percent faster than the baseline. A decision that looks at a single round does not just miss the regression there, it reads it backwards.

How Many Rounds, Which Threshold

The third table scans two axes together: the number of rounds pooled per side, and the threshold. Each cell counts the control arm’s false fails and the candidate arm’s false passes.

The two-percent threshold needs all eight rounds to clear: 179 false fails at one round, zero only at eight. The reason is spread, not median: the control arm’s readings climbing as high as 1.31 push even a pooled median over the gate. A threshold cannot go below the spread the same code produces against itself.

NF9’s five percent sits above this lower bound and is defensible — but not at a single round. At one round there are 180 false fails and 188 false passes; at three rounds 58/120, at five rounds 6/56, at eight rounds 0/0. This is the number the lesson is after: the five-percent gate carries no decision until eight rounds are pooled per side.

The upper bound comes from the last column. At twenty percent, false fail is zero, but false pass is 500 out of 500 even with eight rounds pooled: since the real difference is only about 1.07 to 1.24 times, the gate sits well above it and the regression always ships. The threshold has to stay clearly below the size of the difference it is meant to catch.

The warm-up rows give the table a second time. At eight rounds and five percent both windows read 0/0; at a single round, taking warm-up in raises the false pass from 188 to 191. The warm-up decision only turns the outcome while the round count is low, and it always turns it the same way: it hides the regression.

Cost and the Class It Misses

The run-dependent side of the cost depends on the environment: thirty runs finished in under seventy seconds on this machine. The run-independent side is countable — thirty server processes, an 80,000-row fill per process (2.4 million rows total), 20,940 requests sent, zero failed requests, and 28,000 re-made decisions in the scan table. The version difference is also tied to a run-independent quantity: two queries per request against three.

This measurement misses two classes. First, the scan counts random noise, not systematic bias. The control arm’s median came out at 1.00, so no bias is visible here; if it had appeared, adding rounds would not have shrunk it, because averaging only removes the random component. What would tell the two apart is not this harness but rotating the arm order between rounds.

Second is the threshold’s shape. A proportional gate lets large absolute regressions through on an endpoint whose baseline is slow: five percent here is about 0.11 ms, but if the baseline were 200 ms it would be 10 ms. The gate therefore has to carry the first lesson’s absolute threshold too, not just a ratio.

Summary

• Noise is measured, not assumed: the baseline’s p95 spread 1.34 times across ten rounds, and two identical arms’ ratio ranged from 0.92 to 1.31.
• The warm-up window is read with request blocks: the first hundred requests are about 1.78 times optimistic on the median; its net effect on the candidate/baseline ratio is to read 1.12 in the first block against 1.04 once settled.
• A decision that looks at a single round can mislead: in one of ten rounds candidate/baseline measured 0.71, and a version running one extra query per request looked faster than the baseline.
• The threshold’s lower bound is the harness’s own spread: the two-percent gate needs all eight rounds pooled to reach zero false fails; NF9’s five percent also only reaches 0/0 at eight rounds.
• The threshold’s upper bound is the real difference: the twenty-percent gate lets the roughly 1.07–1.24 times regression through in every trial, even with eight rounds pooled.

Next Step

Performance testing’s five lessons built a threshold, modeled the scenario, chose the metric, attributed the result to a layer, and put the comparison under discipline. The same assumption sits under all five: the requests were valid, the intent was good. The load generator did, quickly, what a real reader could do; no run ever asked what a client that deliberately abuses the system would find. The next lesson takes on that question where it is cheapest to ask, in the program’s source text, and again measures the test itself: how many defects are found, how many warnings turn out empty, which defect never shows up at any threshold.