Metric Selection

The previous two lessons tied every decision to a single number: p95. That choice was never discussed. If the average had been taken, the same runs would look completely different; if p99 had been chosen, the decision would change again, and how many samples a three-second run’s p99 rests on was never asked. This lesson treats the metric itself as a decision.

A Metric Is a Choice

What a run produces is not a single number but thousands of latency values. A metric is the function that draws a single number out of that distribution, and the choice changes what the threshold means. The mean sums every sample with equal weight; one very slow request moves it a little, a thousand moderately slow requests move it a lot. Percentile latency instead reads one position in the sorted sample: p99 is the value below which 99 percent of requests fall, and it says nothing about how much worse the remaining one percent is.

The threshold carries over from the previous lessons unchanged: NF1 (assumption + calculation) — p95 ≤ 20 ms. This lesson adds a sample-size condition to the threshold.

NF5 (calculation) — for a percentile to carry a decision, the run needs at least $1/(1-p)$ samples: 20 for p95, 100 for p99, 1,000 for p99.9. Rationale: if the share the percentile excludes does not reach at least one sample, the value read belongs not to that share but to a lower point in the distribution. This number is not an assumption but a lower bound derived from the sample count, and it is tested below with a scan.

The decision’s owner is the choice of metric itself: a threshold tied to p99 also places a lower bound on the run’s duration. Choosing a metric is also choosing how long the test has to run.

Harness

The service carries over from the previous lesson unchanged: /search is indexed in both structures, /loan only in the sound one. This time the generator accumulates every latency and writes it to a file; the analysis runs without a server.

// library/server.mjs — circulation-desk service: /search looks up a book, /loan
// validates the member's open loan count and writes five records. Arguments:
// <port> <structure 0|1> — 0 = no member index on the loan path (full scan), 1 = indexed.
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 110000, BOOK = 5;  // 220 records per member; a desk transaction writes 5 books
const port = Number(process.argv[2]), structure = process.argv[3];

if (Number.isInteger(port) === false || ["0", "1"].includes(structure) === false) {
  console.log("usage: node library/server.mjs <port> <structure 0|1>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  db.exec("CREATE INDEX loan_book ON loan(book_no)");      // the search path is indexed in both structures
  if (structure === "1") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const search = db.prepare("SELECT id, member_no, day FROM loan WHERE book_no = ? LIMIT 5");
  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const insert = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  let searches = 0, loans = 0, records = START, scan = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ searches, loans, records, scan })); return; }
    if (u.pathname === "/search") {
      searches += 1;
      res.end(JSON.stringify({ found: search.all(u.searchParams.get("book") ?? "K-1").length }));
      return;
    }
    loans += 1;
    scan += structure === "0" ? records : Math.ceil(Math.log2(records));   // full scan in the unindexed structure
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n;
    for (let j = 0; j < BOOK; j += 1) insert.run(member, `K-${(loans + j) % 9000}`, loans % 400);
    records += BOOK;
    res.end(JSON.stringify({ member, loan: n + 1 }));
  }).listen(port);
}

// library/measure.mjs — open-loop generator; accumulates a run's full set of latencies
// and prints a JSON line. Arguments: <port> <structure 0|1> <target rate> <loan share> <duration s>
import { Agent, request } from "node:http";

const [port, structure, rate, share, duration] = process.argv.slice(2).map((x, i) => (i === 1 ? x : Number(x)));
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
let seq = 0;

function oneRequest(samples) {
  const i = seq++;
  const path = (i * 9973) % 1000 < share * 1000 ? `/loan?member=U-${i % 500}` : `/search?book=K-${i % 9000}`;
  const start = performance.now();
  return new Promise((done) => {
    const r = request({ port, path, agent }, (y) => {
      y.resume(); y.on("end", () => { samples.push(performance.now() - start); done(); });
    });
    r.on("error", () => { samples.push(performance.now() - start); done(); });
    r.end();
  });
}

if (Number.isInteger(port) === false || ["0", "1"].includes(structure) === false) {
  console.log("usage: node library/measure.mjs <port> <structure 0|1> <target rate> <loan share> <duration>");
} else {
  await oneRequest([]);                             // warm-up request
  const samples = [], inFlight = [];
  let debt = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(samples)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  const elapsed = (performance.now() - start) / 1000;
  console.log(JSON.stringify({ structure: structure === "0" ? "unindexed" : "indexed", rate, share,
    reached: samples.length / elapsed, ms: samples.map((x) => Math.round(x * 1000) / 1000) }));
}
agent.destroy();

The subsample scan draws samples with replacement from the recorded latencies; the generator is a linear congruential one and its seed is visible.

// library/analysis.mjs — metric selection, sample-size scan and a throughput-latency
// reading over the recorded latencies. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 20, SEED = 20260731, TRIALS = 200;      // NF1 threshold, visible seed, subsample trial count
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (y, h, p) => s.find((r) => r.structure === y && r.rate === h && r.share === p);
const fmt = (x, n = 2) => x.toFixed(n);
const pad = (x, n) => String(x).padStart(n);
const mean = (d) => d.reduce((a, b) => a + b, 0) / d.length;
const percentile = (d, p) => { const t = [...d].sort((a, b) => a - b);
  return t[Math.min(t.length - 1, Math.floor((p / 100) * t.length))]; };

const A = find("unindexed", 1700, 0.25), B = find("indexed", 1700, 0.25);
const METRIC = [["mean", (d) => mean(d)], ["p50", (d) => percentile(d, 50)], ["p90", (d) => percentile(d, 90)],
  ["p95", (d) => percentile(d, 95)], ["p99", (d) => percentile(d, 99)], ["p99.9", (d) => percentile(d, 99.9)],
  ["max", (d) => Math.max(...d)]];
console.log(`same two runs, seven metrics; threshold ${THRESHOLD} ms (NF1), samples ${A.ms.length} / ${B.ms.length}`);
console.log("metric    unindexed(ms) verdict indexed(ms)  verdict unindexed/indexed");
for (const [name, f] of METRIC) {
  const a = f(A.ms), b = f(B.ms);
  console.log(name.padEnd(10) + pad(fmt(a), 12) + pad(a <= THRESHOLD ? "pass" : "fail", 7) +
    pad(fmt(b), 13) + pad(b <= THRESHOLD ? "pass" : "fail", 7) + pad(fmt(a / b), 18));
}

let t = SEED;
const rand = () => { t = (1103515245 * t + 12345) % 2147483648; return t / 2147483648; };
const subsample = (d, n) => Array.from({ length: n }, () => d[Math.floor(rand() * d.length)]);
console.log(`\nsample-size scan: ${TRIALS} subsamples, seed ${SEED}; full-sample p99 ` +
  `unindexed ${fmt(percentile(A.ms, 99))} ms, indexed ${fmt(percentile(B.ms, 99))} ms`);
console.log("samples  p99 min  p99 max  spread factor  false pass/200  false fail/200");
for (const n of [20, 50, 100, 500, 2000]) {
  const a = Array.from({ length: TRIALS }, () => percentile(subsample(A.ms, n), 99));
  const b = Array.from({ length: TRIALS }, () => percentile(subsample(B.ms, n), 99));
  console.log(pad(n, 5) + pad(fmt(Math.min(...a)), 13) + pad(fmt(Math.max(...a)), 14) +
    pad(fmt(Math.max(...a) / Math.min(...a)), 13) + pad(a.filter((x) => x <= THRESHOLD).length, 18) +
    pad(b.filter((x) => x > THRESHOLD).length, 18));
}

console.log("\nthroughput-latency: unindexed structure, same mix, four target rates");
console.log("target rate  reached/s  reached/target  p95(ms)  verdict");
for (const h of [900, 1300, 1700, 2100]) {
  const r = find("unindexed", h, 0.25);
  console.log(pad(h, 9) + pad(fmt(r.reached), 12) + pad(fmt(r.reached / h, 3), 16) +
    pad(fmt(percentile(r.ms, 95)), 9) + pad(percentile(r.ms, 95) <= THRESHOLD ? "pass" : "fail", 7));
}

# measure.sh — five runs: the same scenario on two structures (5 s), plus three extra target rates on the unindexed structure (2 s).
rm -f result.jsonl
for T in 0 1; do
  node library/server.mjs 8853 $T & SERVER=$!
  sleep 1.2
  node library/measure.mjs 8853 $T 1700 0.25 5 >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
for H in 900 1300 2100; do
  node library/server.mjs 8853 0 & SERVER=$!
  sleep 1.2
  node library/measure.mjs 8853 0 $H 0.25 2 >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
node library/analysis.mjs result.jsonl

same two runs, seven metrics; threshold 20 ms (NF1), samples 8496 / 8494
metric    unindexed(ms) verdict indexed(ms)  verdict unindexed/indexed
mean             10.79   pass         0.79   pass             13.60
p50               9.40   pass         0.64   pass             14.80
p90              19.97   pass         1.47   pass             13.58
p95              24.80   fail         1.74   pass             14.29
p99              37.59   fail         2.42   pass             15.53
p99.9            96.32   fail         6.41   pass             15.02
max             151.32   fail         8.53   pass             17.75

sample-size scan: 200 subsamples, seed 20260731; full-sample p99 unindexed 37.59 ms, indexed 2.42 ms
samples  p99 min  p99 max  spread factor  false pass/200  false fail/200
   20        18.85         45.63         2.42                22                 0
   50        28.60         45.63         1.60                 0                 0
  100        35.80         45.63         1.27                 0                 0
  500        40.07         45.62         1.14                 0                 0
 2000        40.07         40.07         1.00                 0                 0

throughput-latency: unindexed structure, same mix, four target rates
target rate  reached/s  reached/target  p95(ms)  verdict
      900      896.57           0.996    12.97   pass
     1300     1290.84           0.993    15.14   pass
     1700     1694.54           0.997    24.80   fail
     2100     1404.40           0.669  1388.52   fail

What the Average Hides

The first table draws seven numbers from a single run, and all seven give the same structure two separate decisions. In this run the mean is 10.79, the median 9.40 and p90 19.97 — all under the threshold; p95 is 24.80, p99 37.59, p99.9 96.32 and the max 151.32 — all over it. The code is the same, the run is the same, the sample is the same; the only thing that changes is which point of the distribution gets read. The numbers belong to the measurement class and depend on this machine; which pair of metrics the cutoff falls between shifts from run to run — in this run it is p90 and p95 that split, in another it could just as well be p95 and p99. What stays independent of the run is the ordering itself: moving toward the tail, the value read grows, and so does the unindexed structure’s ratio to the sound one.

Why the average passes shows up in the last column. In this run a quarter of requests go to the loan path and do a full scan; the remaining three-quarters are cheap searches. The average blends the two populations and cuts the expensive one’s weight to a quarter. Tail percentiles do not blend: p99 by definition sits only in the slowest one percent, and there the unindexed structure is 15.53 times the sound one in this run, and 17.75 times at the max. The exact multiple depends on the run; what stays independent of it is that the ratio in the tail is larger than at the mean.

The name of the defect class this catches is tail latency. The sound structure passes all seven metrics, so it is also visible in this run that tightening the metric toward p99 produces no false fail: the sound structure never turns red.

How Many Samples a Percentile Rests On

The second table draws subsamples from the same latencies and recomputes p99 at each size. At the full sample p99 is 37.59 ms; in twenty-sample subsamples the same figure ranges from 18.85 to 45.63, a spread of about 2.42 times. This is the swing that breaks the decision: 22 of two hundred subsamples put p99 under the threshold and the run reports a false pass. At fifty samples this number already drops to zero.

That the false-pass count reaches zero this early does not overturn NF5’s bound, which is a worst-case guarantee, not a promise that every smaller sample must fail. The share p99 excludes is one percent, so under a hundred samples that share can go unrepresented in the draw, and the value read then comes from a lower point in the distribution; this run’s zero at fifty samples is a case where the draws still happened to land near the true tail. The spread keeps narrowing regardless: 1.27 at a hundred samples, 1.14 at five hundred, 1.00 at two thousand. The false-fail column stays at zero across every size: the sound structure’s p99 sits so far from the threshold that subsampling cannot push it back over. Sample count only distorts the false-pass side — too few samples always bias toward an optimistic result.

The rule that follows looks at run duration: at a target rate of 1,700 req/s, the hundred samples p99 needs are gathered in 0.06 seconds, but in an environment running at 10 req/s outside peak hour the same count needs 10 seconds. For p99.9 that duration grows a thousandfold.

Latency Cannot Be Read Without Throughput

The third table runs the same structure at four target rates. In the first three rows the reached rate is between 0.993 and 0.997 of the target; p95 is 12.97, 15.14 and 24.80 ms. These three numbers are comparable, because all three were measured at essentially the requested load — the last of them already crosses the threshold, but that is a real result, not an artifact of an unmet rate.

The fourth row is not comparable. The target is 2,100 req/s, the reached rate is 1,404.40 — 0.669 of the target. p95 shows as 1,388.52 ms, but that number is not the p95 of 2,100 req/s; it is the p95 of the 1,404 req/s the system could actually accept. Roughly 696 requests per second were never served. A report that carries only the latency column reads this row as “slow,” but what it says is something else: the system never reached the requested load. Latency and throughput are read as a single pair; a latency figure with no reached rate reported alongside it does not say which load it belongs to.

Cost and the Class It Misses

The run-dependent side of the cost depends on the environment: five runs finished in under half a minute on this machine. The run-independent side is countable — the two analysis runs accumulated 8,496 and 8,494 latency values, and the subsample scan ran two hundred trials at each of five sizes for both structures, drawing 1,068,000 samples in total. The class it misses was also measured: the mean and median let this defect pass through three of the seven metrics in this run (mean, median, p90), meaning a metric set’s own limits are written in its false-pass count too.

Summary

• A metric is a choice: in the same run the mean (10.79) and median (9.40) pass the threshold while p95 (24.80) and p99 (37.59) already fail.
• The average blends expensive and cheap requests; a tail percentile does not, and it separates the two structures by 15.53 times at p99 and 17.75 times at the max.
• A percentile is sensitive to sample count: at twenty samples the p99 estimate spreads about 2.42 times and 22 of 200 subsamples give a false pass; this already drops to zero by fifty samples.
• NF5 ties this to a lower bound: a percentile needs at least $1/(1-p)$ samples, meaning 100 for p99 and 1,000 for p99.9 — a worst-case guarantee, so an early zero does not overturn it.
• Latency is read together with throughput: at a 2,100 req/s target the reached rate drops to 0.669, so a p95 of 1,388.52 ms is the figure for the accepted load, not the requested one.

Next Step

The metric is chosen and the threshold has turned red. The report now says: p99 37.59 ms, threshold 20, fail. That sentence reports a defect but does not say where it is. The same number could come from a computation in the application layer, a scan in the database, or the size of the response, and the fix for each of the three is different. The next lesson takes on this attribution: which measurement ties a load-test result to a layer, how two layers producing the same symptom are told apart, and when attribution is wrong.