Test Scenario Modeling

Every run in the previous lesson sent requests to a single endpoint, at a fixed rate, without a pause. A real circulation-desk day is not like that: the clerk searches first, then checks a book out, and pauses between transactions. This lesson takes on the shape of the load rather than its size, and measures this: the same target rate produces a different decision on the same system once the scenario changes.

The Scenario’s Two Parameters

Workload mix is how the total load is distributed across request types. A system’s endpoints do not cost the same; the mix determines which path the load falls on, and so which defect class comes close to saturation.

Think time is the pause a virtual user takes between two transactions, and it only has meaning in a closed-loop generator. In a closed loop there are $N$ virtual users; each waits for the response, pauses for $Z$, then sends the next request; the rate reached is $N / (Z + R)$. Here $R$ is the system’s own response time — in other words, the load is a function of the system under test. An open-loop generator carries no such feedback.

Threshold and Assumptions

The threshold carries over from the previous lesson unchanged. NF1 (assumption + calculation) — the circulation-desk flow’s 60 ms server-side budget split evenly across three services: p95 ≤ 20 ms. NF2 (assumption) — the peak-hour target is 1,000 req/s. This lesson writes two more assumptions.

NF3 (assumption) — the circulation desk’s peak-hour workload mix: half the requests are loans, half are searches. Rationale: the person who comes to the desk comes to check a book out; catalog searches also happen through other endpoints but never exceed half of desk traffic. NF4 (assumption) — in the closed-loop model, the virtual-user count is derived from the target rate: $N = 1000 \cdot (Z + R_0)$, $R_0 \approx 2$ ms. Two pairs are tested: 22 users / 20 ms and 202 users / 200 ms. Both have a nominal rate of 1,000 req/s.

The decision’s owner in this lesson is not the release, it is the test itself: the choice of scenario determines which defect class gets measured, and a poorly chosen scenario reports green.

Harness

The service has two endpoints. /search looks a book up by number in the catalog and is indexed in both structures; /loan validates the member’s open loan count and writes five records, and the index exists only in the sound structure. The catalog starts with 110,000 records, so a full scan is six times more expensive than in the previous lesson.

// library/server.mjs — circulation-desk service: /search looks up a book, /loan
// validates the member's open loan count and writes five records. Arguments:
// <port> <structure 0|1> — 0 = no member index on the loan path (full scan), 1 = indexed.
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";

const MEMBER = 500, START = 110000, BOOK = 5;  // 220 records per member; a desk transaction writes 5 books
const port = Number(process.argv[2]), structure = process.argv[3];

if (Number.isInteger(port) === false || ["0", "1"].includes(structure) === false) {
  console.log("usage: node library/server.mjs <port> <structure 0|1>");
} else {
  const db = new DatabaseSync(":memory:");
  db.exec("CREATE TABLE loan(id INTEGER PRIMARY KEY, member_no TEXT, book_no TEXT, day INT)");
  db.exec("CREATE INDEX loan_book ON loan(book_no)");      // the search path is indexed in both structures
  if (structure === "1") db.exec("CREATE INDEX loan_member ON loan(member_no)");
  db.exec("BEGIN");
  const fill = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  for (let i = 0; i < START; i += 1) fill.run(`U-${i % MEMBER}`, `K-${i % 9000}`, i % 400);
  db.exec("COMMIT");

  const search = db.prepare("SELECT id, member_no, day FROM loan WHERE book_no = ? LIMIT 5");
  const count = db.prepare("SELECT COUNT(*) AS n FROM loan WHERE member_no = ?");
  const insert = db.prepare("INSERT INTO loan(member_no, book_no, day) VALUES(?,?,?)");
  let searches = 0, loans = 0, records = START, scan = 0;

  createServer((req, res) => {
    const u = new URL(req.url, "http://local");
    res.writeHead(200, { "content-type": "application/json" });
    if (u.pathname === "/metrics") { res.end(JSON.stringify({ searches, loans, records, scan })); return; }
    if (u.pathname === "/search") {
      searches += 1;
      res.end(JSON.stringify({ found: search.all(u.searchParams.get("book") ?? "K-1").length }));
      return;
    }
    loans += 1;
    scan += structure === "0" ? records : Math.ceil(Math.log2(records));   // full scan in the unindexed structure
    const member = u.searchParams.get("member") ?? "U-1";
    const n = count.get(member).n;
    for (let j = 0; j < BOOK; j += 1) insert.run(member, `K-${(loans + j) % 9000}`, loans % 400);
    records += BOOK;
    res.end(JSON.stringify({ member, loan: n + 1 }));
  }).listen(port);
}

The generator carries both loop styles. A scenario is just three numbers: the loan share, the virtual-user count, and the think time.

// library/scenario.mjs — open- and closed-loop generator; the scenario sets the request
// mix and the think time. Arguments: <port> <structure 0|1> <scenario>
import { Agent, request } from "node:http";

const TARGET = 1000, THRESHOLD = 20, DURATION = 3;  // NF2: target rate, NF1: p95 threshold (ms), run length
const SCENARIO = {                          // share: loan-request ratio, n/z: closed loop
  easy: { share: 0.10 }, "share-25": { share: 0.25 }, realistic: { share: 0.50 }, "share-100": { share: 1.00 },
  "closed-22": { share: 0.50, n: 22, z: 20 }, "closed-202": { share: 0.50, n: 202, z: 200 },
};
const [port, structure, name] = [Number(process.argv[2]), process.argv[3], process.argv[4]];
const agent = new Agent({ keepAlive: true, maxSockets: 8192 });
const percentile = (d, p) => { const s = [...d].sort((a, b) => a - b);
  return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))]; };
let seq = 0;

function oneRequest(samples, share) {       // a `share` fraction of requests goes to the loan path
  const i = seq++;
  const path = (i * 9973) % 1000 < share * 1000 ? `/loan?member=U-${i % 500}` : `/search?book=K-${i % 9000}`;
  const start = performance.now();
  return new Promise((done) => {
    const r = request({ port, path, agent }, (y) => {
      y.resume(); y.on("end", () => { samples.push(performance.now() - start); done(); });
    });
    r.on("error", () => { samples.push(performance.now() - start); done(); });
    r.end();
  });
}

async function open(duration, rate, share) {   // sends at the target rate, does not wait for the response
  const samples = [], inFlight = [];
  let debt = 0, lastT = 0;
  const start = performance.now();
  while (performance.now() - start < duration * 1000) {
    const t = (performance.now() - start) / 1000;
    debt += rate * (t - lastT); lastT = t;
    while (debt >= 1) { debt -= 1; inFlight.push(oneRequest(samples, share)); }
    await new Promise((c) => setTimeout(c, 5));
  }
  await Promise.all(inFlight);
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

async function closed(duration, n, z, share) {   // n virtual users; each pauses z ms after a request
  const samples = [];
  const start = performance.now();
  const worker = async () => {
    while (performance.now() - start < duration * 1000) {
      await oneRequest(samples, share);
      await new Promise((c) => setTimeout(c, z));
    }
  };
  await Promise.all(Array.from({ length: n }, worker));
  return { samples, elapsed: (performance.now() - start) / 1000 };
}

const metrics = () => new Promise((done) => {
  request({ port, path: "/metrics", agent }, (y) => { let g = "";
    y.on("data", (p) => { g += p; }); y.on("end", () => done(JSON.parse(g))); }).end();
});

const sc = SCENARIO[name];
if (Number.isInteger(port) === false || sc === undefined) {
  console.log(`usage: node library/scenario.mjs <port> <structure 0|1> ${Object.keys(SCENARIO).join("|")}`);
} else {
  await oneRequest([], 1);                 // warm-up request
  const r = sc.n === undefined ? await open(DURATION, TARGET, sc.share) : await closed(DURATION, sc.n, sc.z, sc.share);
  const o = await metrics();
  console.log(JSON.stringify({ name, structure: structure === "0" ? "unindexed" : "indexed", share: sc.share,
    loop: sc.n === undefined ? "open" : `closed N=${sc.n} Z=${sc.z}`, p95: percentile(r.samples, 95),
    rate: r.samples.length / r.elapsed, loans: o.loans, scan: o.scan }));
}
agent.destroy();

// library/report.mjs — turns ten runs into a table and scans the p95 threshold. Argument: <jsonl>
import { readFileSync } from "node:fs";

const THRESHOLD = 20, SCENARIO = ["easy", "realistic", "closed-22", "closed-202"];
const s = readFileSync(process.argv[2], "utf8").trim().split("\n").map((x) => JSON.parse(x));
const find = (y, a) => s.find((r) => r.structure === y && r.name === a);
const fmt2 = (x) => x.toFixed(2);
const pad = (x, n) => String(x).padStart(n);
const verdict = (x) => (x <= THRESHOLD ? "pass" : "fail");

console.log(`threshold p95 <= ${THRESHOLD} ms (NF1), target rate 1000 req/s (NF2), run 3 s`);
console.log("scenario     loop               loan share  structure   reached/s  p95(ms)  verdict");
for (const a of SCENARIO) for (const y of ["unindexed", "indexed"]) {
  const r = find(y, a);
  console.log(a.padEnd(13) + r.loop.padEnd(19) + pad(fmt2(r.share), 10) +
    "  " + y.padEnd(11) + pad(fmt2(r.rate), 10) + pad(fmt2(r.p95), 9) + pad(verdict(r.p95), 9));
}

console.log("\nloan-share scan (unindexed structure, open loop, same 1000 req/s target)");
console.log("loan share      loan req/s  p95(ms)  verdict   scan steps");
for (const a of ["easy", "share-25", "realistic", "share-100"]) {
  const r = find("unindexed", a);
  console.log(pad(fmt2(r.share), 10) + pad(fmt2(r.loans / 3), 16) + pad(fmt2(r.p95), 9) +
    pad(verdict(r.p95), 9) + pad(r.scan.toLocaleString("en-US"), 13));
}

console.log("\nthreshold scan: how many of the four scenarios give a wrong verdict");
console.log("p95 threshold (ms)  false pass /4  false fail /4  distinguishing scenario /4  distinguishing ones");
for (const e of [2, 5, 15, 20, 30, 100]) {
  const pass = SCENARIO.filter((a) => find("unindexed", a).p95 <= e);
  const fail = SCENARIO.filter((a) => find("indexed", a).p95 > e);
  const dist = SCENARIO.filter((a) => find("indexed", a).p95 <= e && find("unindexed", a).p95 > e);
  console.log(String(e).padEnd(20) + pad(pass.length, 14) + pad(fail.length, 15) + pad(dist.length, 27) +
    "  " + (dist.join(", ") || "-"));
}

# measure.sh — ten runs: four scenarios x two structures, plus two extra loan shares.
# Each run works with a freshly started server; the catalog always starts with 110,000 records.
rm -f result.jsonl
for T in 0 1; do
  for SC in easy realistic closed-22 closed-202; do
    node library/server.mjs 8852 $T & SERVER=$!
    sleep 1.2
    node library/scenario.mjs 8852 $T $SC >> result.jsonl
    kill $SERVER
    wait $SERVER 2>/dev/null
  done
done
for SC in share-25 share-100; do
  node library/server.mjs 8852 0 & SERVER=$!
  sleep 1.2
  node library/scenario.mjs 8852 0 $SC >> result.jsonl
  kill $SERVER
  wait $SERVER 2>/dev/null
done
node library/report.mjs result.jsonl

threshold p95 <= 20 ms (NF1), target rate 1000 req/s (NF2), run 3 s
scenario     loop               loan share  structure   reached/s  p95(ms)  verdict
easy         open                     0.10  unindexed      997.48     8.21     pass
easy         open                     0.10  indexed        997.52     1.69     pass
realistic    open                     0.50  unindexed      916.07   760.33     fail
realistic    open                     0.50  indexed        997.51     1.83     pass
closed-22    closed N=22 Z=20         0.50  unindexed      692.28    29.06     fail
closed-22    closed N=22 Z=20         0.50  indexed        970.94     2.55     pass
closed-202   closed N=202 Z=200       0.50  unindexed      855.57    77.97     fail
closed-202   closed N=202 Z=200       0.50  indexed        985.30    11.75     pass

loan-share scan (unindexed structure, open loop, same 1000 req/s target)
loan share      loan req/s  p95(ms)  verdict   scan steps
      0.10           99.67     8.21     pass   33,112,755
      0.25          248.67    13.61     pass   83,449,425
      0.50          500.00   760.33     fail  170,621,250
      1.00          998.67  6106.87     fail  351,992,550

threshold scan: how many of the four scenarios give a wrong verdict
p95 threshold (ms)  false pass /4  false fail /4  distinguishing scenario /4  distinguishing ones
2                                0              2                          2  easy, realistic
5                                0              1                          3  easy, realistic, closed-22
15                               1              0                          3  realistic, closed-22, closed-202
20                               1              0                          3  realistic, closed-22, closed-202
30                               2              0                          2  realistic, closed-202
100                              3              0                          1  realistic

The Mix Changes the Decision

The target rate is 1,000 req/s in all four runs. The only thing that changes is how that rate is distributed across the endpoints. At a loan share of 0.10 the flawed structure stays at 8.21 ms and passes; at 0.25 it is still passing at 13.61 ms; at 0.50 it jumps to 760.33 ms. The difference is not in the system, it is in the scenario.

The reason is in the second column: the loan request rate climbs from 99.67 to 500.00. In the flawed structure this endpoint does a full scan and its own ceiling is around 400 req/s; the share is under that ceiling at 0.25 and over it at 0.50. The scan step shows this independently of the run: from 33,112,755 to 170,621,250, about 5.2 times. The name of the defect class this catches is saturation of a single path: while the system as a whole meets the target rate, one endpoint hits its ceiling and corrupts the latency of every request.

The easy scenario is where this escapes. A search request is the cheapest one to write, so scenarios tend to skew toward search-heavy. If NF3 writes the realistic mix as 0.50, a test run at 0.10 produces a false pass, and the name of the class it misses is already known.

The Closed Loop Throttles Its Own Load

The bottom two scenarios run the same mix (0.50) through the closed loop, and both have a nominal rate of 1,000 req/s. In the flawed structure the rate reached is 692.28 and 855.57 req/s — that is, 69.2% and 85.6% of the target. The generator throttled its own load because the system under test slowed down. The load NF2 calls for was never applied; the p95 measured is the p95 of a different rate.

The result is an understatement. The open loop reported 760.33 ms at the same mix; the closed loop reported 29.06 and 77.97. Same system, same defect, a number about 26 times and 10 times smaller. In this run all three still cross the 20 ms threshold so the decision stays the same; if the threshold were 30 ms, closed-22 would pass while the open loop would still fail.

The sound-structure column shows think time’s own effect separately. Even with no defect at all, the same nominal rate gives three separate p95 values: 1.83 ms in the open loop, 2.55 ms with 22 users / 20 ms, 11.75 ms with 202 users / 200 ms — a spread of about 6.4 times. The number belongs to the generator’s parameters, not the system.

Threshold Scan and Cost

The scan gives a two-sided result. At NF1’s 20 ms, false pass is 1 (the easy scenario), false fail is 0 — and this still holds at 15 ms in this run. Tightening further to 5 ms introduces one false fail — the sound structure’s closed-202 scenario, whose own think-time noise alone crosses 5 ms — and tightening to 2 ms doubles that to two, adding closed-22. At 100 ms false pass climbs to three; only the open-loop realistic scenario still distinguishes.

The class it misses can also be counted: since the two closed-loop scenarios never reach the rate NF2 calls for, the saturation that appears at the target rate was never measured in them; in the flawed structure the missing load is about 307 and 144 req/s. The run-dependent side of the cost depends on the environment — ten runs finished in under a minute on this machine. The run-independent side: every open-loop run sends 3,000 requests and the loan-request count comes directly from the share (300, 750, 1,500, 3,000); the scan step in the unindexed structure ranges from 33,112,755 to 351,992,550, about 10.7 times.

Summary

• The scenario has two parameters: workload mix decides which endpoint the load falls on, think time decides the rate reached in a closed loop.
• The same 1,000 req/s target passed at a loan share of 0.10 (8.21 ms), passed at 0.25 (13.61 ms), and failed at 0.50 (760.33 ms) in this run; the defect became visible through the scenario, not the system.
• The class it catches is saturation of a single path: as the loan request rate climbs from about 100 to 500 req/s, the endpoint crosses its own ceiling and the scan step rises about 5.2 times.
• The closed loop never reached the target rate (about 69% and 86% of it) and understated the degradation by roughly 26 and 10 times; the p95 measured is not the p95 of the requested load.
• Even in the sound structure, the generator’s own parameters carry p95 from 1.83 to 11.75 ms; tightening the threshold to 5 ms is where this noise starts producing a false fail.

Next Step

Up to this point every decision was made by looking at a single number: p95. That choice was never discussed. If the average had been taken, the 760.33-millisecond run would look completely different; if p99 had been chosen, the decision would change again, and how many samples a three-second run’s p99 rests on was never asked. That throughput and latency need to be read together was only implied. The next lesson takes on metric choice: what an average hides, how many samples a percentile needs to be meaningful, and why latency read apart from throughput is misleading.